Images as signals

A picture is a grid of numbers, and its simplest pattern is a set of stripes, with a frequency and a direction. Watch the stripes below turn with the dot beside them, and close in as it moves away from the centre.

A frequency with a direction

Look closely at a phone screen and you see tiny squares of light, in rows and columns. Each square is a pixel, and it shows one brightness. So a picture on a screen is a grid of numbers.

Where do the numbers come from? A camera’s sensor is a grid of small light meters, so a photograph is the scene sampled at a grid of points. That is the sampling of Sampling & aliasing (10.1), with dots in two directions instead of one.

I write the brightness of one pixel as $x[n_1,n_2]$, from 0 for black to 1 for white. The first index, $n_1$, is the column, counted across from the image’s left edge. The second, $n_2$, is the row, counted down from the top, the way we read text. An image is this signal of two indices.

Stripes

The simplest pictures are stripes. Take one row and make it a sinusoid that repeats every 16 pixels, as in Sinusoids (3.2), “Height and repeat time”: $0.5+0.5\cos(2\pi n_1/16)$. The 0.5 in front and the height 0.5 keep the brightness between 0 and 1.

Give every row the same values, and you get upright stripes, 16 pixels apart. Along a row the wave goes through 1/16 of a cycle per pixel. Now let both indices take part:

Here $u$ is the number of cycles per pixel as you go across, and $v$ the number as you go down. They are plain numbers, set once for the whole picture. A picture like this is a grating, and the pair $(u,v)$ is its spatial frequency: cycles per pixel of distance, not per second.

In radians per pixel the two parts are $\Omega_1=2\pi u$ and $\Omega_2=2\pi v$, the $\Omega$ of 3.2 counted per pixel instead of per sample. On this page I quote them in cycles per pixel, as $u$ and $v$.

Size and direction

Where are the stripes? The brightness depends only on the sum $un_1+vn_2$. Walk in a direction that keeps that sum the same, and the brightness stays the same: that path is a stripe. The sum stays the same when you walk at right angles to the arrow from $(0,0)$ to $(u,v)$.

So the stripes run at right angles to $(u,v)$. Now walk a distance $s$ along that arrow instead. The sum grows by $s\sqrt{u^2+v^2}$, and one cycle has passed when it has grown by 1. The stripes are therefore

pixels apart. The size of the spatial frequency, $\sqrt{u^2+v^2}$, sets the spacing. Its direction is the angle of the arrow, measured from the across axis and turning toward down. Think of a ploughed field: how close the furrows are and which way they run are two separate facts.

To draw a spatial frequency, I use a map with $u$ across and $v$ down, each from −0.5 to 0.5. I call it the frequency plane, and one grating is a point on it. Why it stops at half a cycle per pixel comes in the next section.

A grating shows up on the plane as two dots, not one. In Complex exponentials & phasors (3.4) a cosine was two arrows spinning opposite ways. Here the two arrows are $e^{\pm j2\pi(un_1+vn_2)}$, one at $(u,v)$ and one at $(-u,-v)$, mirror images through the centre.

The picture at the top of the page draws both: the grating, and its dot and mirror on the plane.

Notice the middle frame. After the turn, $u$ and $v$ have both fallen to 0.0442, yet the spacing is still 16.0 pixels. Along one row, the turned stripes now cross every $1/0.0442=22.6$ pixels. The spacing is measured across the stripes, the shortest way: 1 over the size of the frequency, 0.0625.

When the clip ends, the dot is yours. Pull it toward the centre and watch the stripes spread apart; turn it and watch them turn.

Half a cycle per pixel, in each direction

Follow one row of a grating. There $vn_2$ is fixed, so the row is a sinusoid with $u$ cycles per pixel. By 10.1’s “Where a tone lands”, $u$ and $u+1$ give the same samples:

The extra $2\pi n_1$ is a whole number of turns. A cosine is even as well, so $u$ and $-u$ look the same along a row. That leaves $u$ from −0.5 to 0.5, and the same argument down a column leaves $v$ from −0.5 to 0.5.

So a picture holds spatial frequencies up to half a cycle per pixel in each part, and the frequency plane is that square. Each part has its own limit. The grating $(0.4,0.4)$ is fine, though its size, 0.566, is more than 0.5.

A finer pattern comes back as a coarser, false one somewhere else in the square. That is aliasing, and the next instrument shows it happening.

Skip pixels, and false rings appear

To see where aliasing happens in a picture, I want every spatial frequency at once, each at its own place. A zone plate does that. On 256 × 256 pixels, with $n_1$ and $n_2$ from 0 to 255, it is

where $d_1=n_1-128$ and $d_2=n_2-128$ are the pixel’s offsets from the centre, across and down. Its brightness makes rings around the centre, and they close in as you move out.

How many cycles per pixel do the rings have near a given pixel? Step one pixel across, from $d_1$ to $d_1+1$. The angle inside the cosine grows by

which is $(d_1+\tfrac12)/256$ of a cycle. So near offset $d_1$ the rings have about $d_1/256$ cycles per pixel across, and about $d_2/256$ down.

At the centre that is 0. At the edges it reaches about half a cycle per pixel: the step from $d_1=-128$ to $-127$ is 0.498 of a cycle. Every spatial frequency in the square appears once, the frequency $(d_1/256,\,d_2/256)$ at offset $(d_1,d_2)$.

Keep every M-th pixel

Now shrink the zone plate the crudest way: keep every $M$-th pixel in each direction, $x[Mi,Mj]$. This is “Keep every M-th sample: the spectrum stretches and folds” of Downsampling and decimation (22.1), done across and down.

There every frequency was multiplied by $M$, and here too. A pattern with $f$ cycles per original pixel has $Mf$ cycles per kept pixel, and it folds when $Mf$ passes ½. Near offset $d_1$, $f=d_1/256$, so the rings fold where

The same holds down, with $d_2$. So everything farther than $128/M$ pixels from the centre, across or down, aliases. The limit is $1/(2M)$ cycle per original pixel, and it halves each time $M$ doubles.

The false coarse pattern that fine detail turns into is called moiré. You have seen it on television: a presenter’s finely striped shirt shimmers with broad patterns that the shirt does not have.

Watch the corners and the middle of each edge as fewer pixels are kept: false rings grow there, outside the dashed square.

I measure the rings’ contrast as the RMS of the brightness about 0.5, the RMS of How big is a signal (1.3), “Four everyday sizes”. For a cosine of height 0.5 it is $0.5/\sqrt2=0.354$.

Notice the end frame: with every 4th pixel kept, 92.94 % of the kept pixels lie where the rings alias. The false rings there are as strong as the true ones, 0.353 against the 0.354 of the whole zone plate.

Why 92.94 % and not the 93.75 % of the area outside the square? The kept pixels on the dashed square itself count as inside: 17 of the 64 kept columns lie within 32 pixels of the centre. For $M=2$ it is 65 of 128, which gives 74.21 % against 75 %.

Where the false centres come from

Take the middle of the left edge, $d_1=-128$. There the rings have half a cycle per original pixel. Keep every 2nd pixel and that becomes 1 cycle per kept pixel, which gives the same samples as 0 cycles, by 10.1’s rule.

Zero frequency is what the true centre has. So a false centre appears at the middle of each edge, and at each corner, where both parts are half a cycle. The kept picture repeats every 64 kept pixels, so each false centre is a copy of the true one.

With every 4th pixel, the frequency per kept pixel is a whole number at every multiple of 64 pixels from the centre: $4\cdot64/256=1$. That puts false centres on a grid, 64 pixels apart, as in the end frame.

Blur first

When the clip ends, press “blur first” under the zone plate. It averages each $M\times M$ block of pixels before keeping one value per block. I call this area averaging: each kept value is the mean brightness over the area its block covers.

The block average is the $M$-point moving average of Frequency response of discrete-time systems (12.4), “A moving average, probed”, run across and then down. So area averaging is 22.1’s “Filter first, then keep every M-th”, with a simple filter. With $f$ in cycles per pixel, its gain in each direction is

The gain is 0 at $f=1/M$, $2/M$ and so on, the frequencies that land on false centres. At the middle of the left edge, where the false centre sat, area averaging leaves plain grey: 0.500 at $M=2$, 0.501 at $M=4$.

Outside the dashed square the false rings fade from 0.353 to 0.117 at $M=2$, and to 0.056 at $M=4$. Each block’s average is placed at its block’s middle when I sort it inside or outside the square.

The price is that the true rings fade as well. Inside the square they keep 0.289 at $M=2$ and 0.278 at $M=4$, down from 0.354. At the limit, $f=1/(2M)$, the gain in each direction is 0.707 for $M=2$ and 0.653 for $M=4$.

So a block average is a rough low-pass filter: it fades what would alias, and some of what would not. The decimation filters of 22.1 were designed to cut more sharply, and Image filtering (28.2) builds blurs of other shapes.

Jaggies

A sharp edge in a picture is the square wave of 10.1’s “A square wave aliases at any note”. The brightness jumps from 0 to 1, and a jump holds frequencies that never end. However many pixels you have, some of those frequencies lie past half a cycle per pixel.

On a tilted edge, that aliasing shows as a staircase. The steps are called jaggies. Area averaging softens them: an edge pixel gets the fraction of its area that the shape covers, a grey between 0 and 1.

Worked example

1. Spacing from frequency. Take $(u,v)=(0.03,0.04)$. Its size is $\sqrt{0.03^2+0.04^2}=\sqrt{0.0025}=0.05$, so the stripes are $1/0.05=20$ pixels apart. The direction is the angle whose tangent is $0.04/0.03$, which is 53.1°.

In radians per pixel, $\Omega_1=0.06\pi$ and $\Omega_2=0.08\pi$, and $2\pi/(0.1\pi)$ gives the same 20 pixels. Along one row the stripes cross every $1/0.03=33.3$ pixels, farther apart than 20, as in the turned frame of the first instrument.

2. The alias limit. Keep every 2nd pixel of the zone plate. The limit is $1/(2\cdot2)=1/4$ cycle per original pixel. The rings reach that where $d_1/256=1/4$, which is $d_1=64$ pixels from the centre, where the dashed square sits.

3. A grating that turns. Take $(u,v)=(0.1,0.2)$ and keep every 4th pixel. The limit is $1/8=0.125$, so $u$ is fine and $v$ is past it. On the kept pixels the frequencies are multiplied by 4, to $(0.4,0.8)$, and $0.8$ gives the same samples as $0.8-1=-0.2$.

So the kept pixels show the grating $(0.4,-0.2)$. Its stripes are $1/\sqrt{0.2}=2.24$ kept pixels apart, which is 8.94 original pixels, twice the true 4.47. Its direction is −26.6° instead of 63.4°. The false stripes are coarser, and they have turned through a right angle.

Where you’ll meet this

Every image resizer and thumbnail maker meets this problem. Keeping every $M$-th pixel is fast but makes moiré. OpenCV’s area resampling, INTER_AREA, averages each block, and its documentation suggests it for shrinking images, as it gives results free of moiré.

Many cameras put a thin optical low-pass filter in front of the sensor. It blurs the light slightly before the pixels sample it, the job of the anti-alias filter of Anti-aliasing and practical converters (10.4). Some cameras leave it out for sharper pictures, and then fine fabrics can show moiré.

Fonts on screens are drawn with grey edge pixels, each as dark as the share of it that the letter covers, as in the figure; this is called anti-aliasing. Scanners meet moiré when they scan printed pictures, which are made of fine dots, and many offer a “descreen” setting that blurs first.

The page Image filtering (28.2) slides small grids of numbers over an image, which is how blurs are made. The 2-D DFT (28.3) computes the frequency plane of any image, not only of a grating. For more, see Gonzalez and Woods, Digital Image Processing (4th ed., 2018), chapters 2 and 4, and J. S. Lim, Two-Dimensional Signal and Image Processing (1990), chapter 1.

Reference card