Quantization & noise

Here a slow ramp from $-1$ to $1$ goes through a converter that may keep only 8 values, and each sample is rounded to the nearest one. Watch the error, the gap between output and input, as the ramp climbs.

Rounding to the nearest level

Kinds of signals (1.2) described a digital signal as a ruler with only so many marks: every value has to be one of a fixed list of levels. Chapter 10 chose when we look at a signal. This page chooses which values we may keep.

A converter with $B$ bits can store $2^B$ different levels. I take the signal’s range to be $-1$ to $1$ and spread the levels evenly across it. The gap between neighbouring levels is the step, written $\Delta$:

The quantizer, written $Q\{\cdot\}$, replaces each sample with the nearest level, so the stored signal is $x_q[n]=Q\{x[n]\}$. What it gets wrong is the error $e[n]=x_q[n]-x[n]$.

Here the levels sit at $\pm\Delta/2$, $\pm3\Delta/2$, and so on, with no level at 0. This placement is called mid-rise. With $B=3$ there are 8 levels, $\Delta=0.25$, and the levels are $\pm0.125$, $\pm0.375$, $\pm0.625$ and $\pm0.875$.

Do not read this as “more bits make a drawn curve smoother”. The staircase in Reconstruction (10.3) was a different one: it held each value for a whole sample period, so it changed the time. This staircase changes the value. More bits do not mainly smooth a curve; they make the error quieter.

The error of a slow ramp

The picture at the top of the page put the simplest input, a slow ramp from $-1$ to $1$, through a 3-bit quantizer.

Look at the sawtooth. Each level covers one stretch of the ramp. Inside that stretch the output stays put while the input climbs past it, so the error runs from $+\Delta/2$ down to $-\Delta/2$. The same tooth then repeats on every step, and the error never leaves the dashed lines.

That is a general fact about rounding: the nearest level is never more than half a step away. Prices rounded to the nearest 5 cents are never more than 2.5 cents off, whatever the price.

The readout “mean square of error ÷ Δ²” keeps a running mean square of the error over the stretch drawn so far. Watch it settle at 0.083.

To see where 0.083 comes from, write the error as a fraction of a step, $u=e/\Delta$. On the ramp, $u$ spreads evenly from $-\tfrac12$ to $\tfrac12$ along each tooth. How big is a signal (1.3) found the average of $u^2$ over 0 to 1 to be $\tfrac13$. The same recipe over $-\tfrac12$ to $\tfrac12$ gives:

The mean square of the error is therefore $\Delta^2/12$, which is $0.083\,\Delta^2$ to three decimals. Its square root, $\Delta/\sqrt{12}$, is the RMS error.

A busy signal: same values, scrambled

A ramp is too tidy to be a real signal. Now try one that is busy: two tones for 20 ms, into the same 3-bit quantizer. The signal is $0.6\sin(2\pi\cdot997t)+0.35\sin(2\pi\cdot2311t+1.0)$, sampled at 48 kHz, so 960 samples. Watch the error, and its mean square.

The error no longer traces teeth. It jumps about, because the signal keeps landing at different places inside different steps. It still stays inside $\pm\Delta/2$, and it still takes the same range of values, only in a scrambled order. So its mean square comes out close to $\Delta^2/12$ again: the readout reads 0.082 at 10 ms and 0.083 at 20 ms.

This is the noise model. It treats the error as a steady hiss of mean square $\Delta^2/12$ added to the signal, $x_q[n]=x[n]+e[n]$. The model is an approximation. It works when there are many levels and the signal is busy, and the last two sections show where it fails.

Where the levels sit

Rounding to the nearest level is one choice. Two others show up in practice.

In the figure, each plot has input along the horizontal axis and quantized output up the vertical axis, and the dashed diagonal is a perfect, unquantized output.

Mid-tread puts a level at 0, so the levels are $0$, $\pm\Delta$, $\pm2\Delta$ and so on. A quiet input then comes out as exactly 0. This is the version that matters for the last section. Truncation always rounds down, so the error runs from 0 to $-\Delta$ instead of from $-\Delta/2$ to $\Delta/2$. Over that range, $u=e/\Delta$ is spread evenly from $-1$ to $0$: its mean is $-\tfrac12$ and its mean square is $\int_{-1}^{0}u^2\,du=\tfrac13$. So the error has mean $-\Delta/2$ and mean square $\Delta^2/3$, which is four times the mean square of rounding.

Each bit is worth 6 dB

Now the quantity this whole page has been heading for: the signal-to-noise ratio. How big is a signal (1.3) defined it as the ratio, in dB, of signal power to noise power. I use the loudest sine that fits, a full-scale sine of amplitude 1.

Its RMS is $1/\sqrt2$, so its mean square is $(1/\sqrt2)^2=\tfrac12$. By the noise model, the noise has mean square $\Delta^2/12$. Put in $\Delta=2/2^B$:

The ratio is of powers, so SNR in dB uses $10\log_{10}$. The log of a product is a sum of logs, as in Frequency response and Bode plots (8.4), and $\log_{10}4^B=B\log_{10}4$:

Each extra bit multiplies the number of levels by 2, halves the step, and so cuts the noise power by 4. A factor of 4 in power is $10\log_{10}4=6.02$ dB. That is the rule of thumb: about 6 dB per bit.

The formula rests on the noise model, so I measured it. The instrument below quantizes a full-scale 997 Hz sine, 1 second at 48 kHz, to $B$ bits for $B=1$ to 16, and compares the measured SNR with the formula. Watch the measured dots against the formula’s line as $B$ grows.

Look at the last readout, “formula − measured”. At $B=1$ it reads 1.34: the formula says 7.78 dB, the measurement is 6.44 dB. At $B=8$ it reads 0.13, and at $B=16$ it reads 0.00, with both values at 98.08 dB.

From 8 bits upward the formula and the measurement agree within 0.13 dB. At 1 to 4 bits the formula is between 0.53 and 1.34 dB too optimistic. With only a few levels, the error is no longer a steady hiss: it follows the shape of the signal instead of scrambling it.

The same rule gives the dynamic range of a converter, the span from its noise to the loudest signal it can hold, in dB. A 12-bit converter gets $6.02\cdot12+1.76=74.00$ dB, and 16-bit audio gets 98.08 dB. At 24 bits the rounded formula gives 146.24 dB, and unrounded constants give 146.26 dB.

A quiet sine

The noise model also fails in the other direction, when the signal is quiet and simple. Take a sine whose peak is only 1.4 steps, quantized with mid-tread levels, with 32 samples in each period.

The output takes only the values $-1$, $0$ and $1$ steps, and it is the same staircase in every period. So the error repeats with the signal: it is a copy of the same pattern, period after period. A wave that repeats is a sum of harmonics, sines at whole-number multiples of the fundamental frequency, as in Signals as sums of sinusoids (7.1). The rounded sine therefore holds the original fundamental plus extra harmonics that were never in the input.

The third harmonic has amplitude 0.273 of a step against the fundamental’s 1.220. That is distortion: tones in the output that follow the signal, not hiss spread across all frequencies. Dither (11.2) fixes this.

Worked example

A 12-bit converter on 0 to 3.3 V. The range is 3.3 V wide, so the step is $\Delta=3.3/2^{12}=3.3/4096=0.806$ mV. (On $\pm1$ the range is 2 wide, which gives $\Delta=2/2^B$.) The RMS error is $\Delta/\sqrt{12}=0.806/3.464=0.233$ mV.

Quiet music. Take a 16-bit recording of a sine 20 dB below full scale. The noise stays where it was, so the signal loses 20 dB against it, and the SNR is $98.08-20=78.08$ dB. Each dB of level you give up costs a dB of SNR.

Rounding against truncation. For the same step $\Delta$, rounding gives an error with mean 0 and mean square $\Delta^2/12$. Truncation gives mean $-\Delta/2$ and mean square $\Delta^2/3$, which is $4\times$ as much.

Where you’ll meet this

Audio CDs store 16 bits per sample, a dynamic range of about 98 dB, and studios record at 24 bits to leave room to adjust levels. A microcontroller’s built-in converter is usually 10 or 12 bits, so its step sets the smallest voltage change it can see. A photo saved with too few shades shows bands in smooth skies, the same rounding on brightness. Oversampling and noise shaping (11.3) shows how to push the noise out of the band you care about.

Reference card