Random variables for signals

Noise has no formula, but draw many of its values, sort them into bins, and they pile up in the same shape every time. Watch the bars settle onto the curve as the draws grow from 10 to 10 000.

Draws pile up into a density

In “Predictable, or not” of Kinds of signals (1.2), a steady tone sat beside noise. The tone has a formula and the noise has none. Yet noise is not lawless: draw many of its values, and they pile up in a shape that is the same every time. This page is about that shape.

A random variable is a number you get fresh each time you look. Each look is a draw. I write the random variable as $x$, and its draws, numbered in the order they came, as $x[0]$, $x[1]$ and so on. A noise trace on this site is such a row of draws.

When the draws of a signal depend on each other, the signal is a random process, the subject of Random processes (24.2). Here every draw is made afresh, on its own. This page assumes no statistics: I define each term before I use it, and leave the proofs to statistics.

The probability of an event is the share of draws in which it happens, in the long run. I write it $\Pr\{\cdot\}$. If $x$ is the roll of a fair die, $\Pr\{x=6\}=1/6$: one roll in six, over many rolls.

To see the shape of many draws, sort them into bins of width 0.5 and count each bin. Then divide each count by $N\times0.5$, where $N$ is the number of draws. A bar’s area is its height times 0.5, which is its count divided by $N$: the share of the draws that landed in it. All the bars together have area 1.

As $N$ grows, the bars settle on a curve, the density $p_x(\alpha)$. I use $\alpha$ for a value on the axis, to keep it apart from the random variable $x$. The probability that a draw lands between $\alpha_1$ and $\alpha_2$ is the area under the density there:

Every draw lands somewhere, so the total area is 1. A density is never negative, but it can be larger than 1: it is a probability per unit of width, not a probability.

The density met most often in signal work is the Gaussian, a bell. With mean 0 and variance 1 (both defined just below) it is

which peaks at $\alpha=0$ at a height of $1/\sqrt{2\pi}=0.3989$. The site’s generator, gaussian in lib/random.ts, makes such draws from pairs of seeded uniform numbers, so every visit sees the same draws.

The picture at the top of the page has two readouts that follow the draws. The mean is their plain average, the $\bar x$ of “Four everyday sizes” in How big is a signal (1.3). The variance is their average squared distance from that mean, a measure of how widely they spread.

Watch the variance readout: 1.255, 0.898, 0.993, 1.027. Ten draws can be far off, but ten thousand land within 3 % of the density’s 1. The tallest bar tells the same story. With ten draws it is 0.600, three draws in one bin; at 10 000 it is 0.380.

Even then the tallest bar stays a little below the curve’s peak, 0.3989. A bar shows the curve’s average over its width, and next to 0 that average is 0.3829.

Mean and variance

The expected value $\mathbb{E}\{\cdot\}$ of anything that depends on $x$ is its average over the density: each value weighted by how likely it is. The mean $\mu_x$ and the variance $\sigma_x^2$ are two expected values, with the integrals running over every value:

The variance is the average power of 1.3, measured about the mean. For noise with mean 0 it is simply the average power $\mathbb{E}\{x^2\}$. Its square root $\sigma_x$ is the standard deviation, and for mean-0 noise that is the RMS. So the 0.2 kg RMS noise of “A longer average: less noise, more delay” in Simple smoothing filters (18.2) has a standard deviation of 0.2 kg.

The density is never known from a finite set of draws, only estimated. A hat means “estimated from draws”:

The first is 1.3’s mean $\bar x$ of the draws. The estimates change with every new set of draws, while $\mu_x$ and $\sigma_x^2$ do not, and the first picture’s readouts are these two estimates. NumPy’s mean and var compute them; var divides by $N$ unless told otherwise.

A Gaussian can have any mean and any standard deviation. It is the bell above, moved to $\mu_x$ and widened by $\sigma_x$:

Its areas are worth knowing by heart. Within 1, 2 and 3 standard deviations of the mean lie 68.27 %, 95.45 % and 99.73 % of all draws. The 10 000 draws give 67.57 %, 95.44 % and 99.68 %.

A Gaussian has no edge, though. About 3 draws in 1000 land further out than 3 standard deviations, so noise has no fixed peak value.

The uniform density

The uniform density is flat. Every value in a width $\Delta$ is equally likely, and no value outside it ever comes. Its height is $1/\Delta$, so its area is 1, and its mean is the middle of the width.

“The error of a slow ramp” in Quantization & noise (11.1) found its variance: a value spread evenly over a width $\Delta$ has mean square $\Delta^2/12$ about its middle. That is the rounding error of 11.1’s noise model, a uniform draw over one step.

A width of $2\sqrt3=3.4641$ gives variance 1 and height $0.2887$. The generator’s uniform numbers $u$ run from 0 to 1, so $\sqrt3\,(2u-1)$ runs from $-1.732$ to $1.732$.

After the clip, press “uniform” under the first picture. It draws 10 000 values as $\sqrt3\,(2u-1)$, with $u$ from the same seed, and draws the flat density at 0.2887 between ±1.73. The mean and variance read −0.022 and 0.993: the width $2\sqrt3$ was chosen for variance 1.

Look at the ends, though. The inner bars read 0.272 to 0.301, but the two end bars only 0.137 and 0.135. Their bins run from 1.5 to 2 and from −2 to −1.5, and the density stops at ±1.732, so less than half of each bin can catch draws.

Sums of draws

Add two draws that are made separately, with nothing linking them, and their variances add. This is “the powers add” of 18.2, now said of spread about the mean. The next section names this kind of pair: independent.

11.2’s triangular dither is the sum of two separate uniform values, each of width $\Delta$. So its variance is $2\times\Delta^2/12=\Delta^2/6$, twice that of rectangular dither. Add the rounding’s own $\Delta^2/12$, and you have the $\Delta^2/4$ that “Rectangular or triangular dither” in Dither (11.2) found.

The sum of two uniform values has a triangular density, which is where the name comes from. Add more of them and the shape rounds off further. Add many independent values, all with one density of finite variance, and the sum looks Gaussian, whatever that density was.

This is the central limit theorem, and statistics proves it. It is why thermal noise is Gaussian: it is the sum of countless tiny motions of electrons.

A cloud that leans: correlation

Now take two values per draw, a pair $(x,y)$: say a sensor’s error and its neighbour’s. Pairs have a joint density, spread over a plane instead of along a line. The probability that a pair lands in a region is the volume under the joint density there.

Two random variables are independent when knowing one tells you nothing about the other. Then the joint density is the product of the two densities, and averages of products split: $\mathbb{E}\{xy\}=\mathbb{E}\{x\}\,\mathbb{E}\{y\}$.

Pairs that are not independent can hang together in many ways. The simplest measure of how they do is their correlation coefficient:

It always lies between −1 and 1. It is 1 when the pairs lie on a rising straight line, −1 on a falling one, and 0 when there is no straight-line trend. Its estimate $\hat\rho_{xy}$ puts estimates from the $N$ pairs in place of each expected value and each $\sigma$, and NumPy’s corrcoef computes it.

If $x$ and $y$ are independent, the top of that fraction splits into $\mathbb{E}\{x-\mu_x\}\,\mathbb{E}\{y-\mu_y\}$, which is $0\times0$. So independent values are uncorrelated: $\rho_{xy}=0$. A figure after the instrument shows that the reverse fails.

To make pairs with a chosen correlation $\rho$, take two independent Gaussian draws $x$ and $z$, each with mean 0 and variance 1, and set

The two parts of $y$ are independent, so their variances add: $\rho^2+(1-\rho^2)=1$. And $\mathbb{E}\{xy\}=\rho\,\mathbb{E}\{x^2\}+\sqrt{1-\rho^2}\,\mathbb{E}\{xz\}=\rho$, since $\mathbb{E}\{xz\}=0$. With both variances 1, that makes $\rho_{xy}=\rho$.

Watch the cloud lean toward the dashed line as ρ grows. That line marks the straight-line trend: for these pairs, $y$ is $\rho x$ on average.

Why does $\rho=0$ measure −0.080? The 500 draws of $x$ and the 500 of $z$ are not perfectly unrelated over so few pairs: their own measured correlation is −0.080. Their measured means are 0.057 and 0.070, and their variances 0.960 and 1.025, not exactly 0 and 1.

After the clip, use the slider to set ρ yourself. Try 0.3 and 0.8: the pairs measure 0.219 and 0.772. Over the whole slider, 500 pairs miss the set value by at most 0.084, at $\rho=0.16$. Near ±1 the miss shrinks: 0.006 at 0.95.

Uncorrelated, but not independent

Correlation measures only a straight-line relation. Here is a pair that is tied together completely and still has a correlation of 0.

Draw $\theta$ uniformly from 0 to $2\pi$, with the generator’s uniform numbers (seed 2410), and set $x=\cos\theta$ and $y=\sin\theta$. Every point lies on a ring of radius 1, so once $x$ is known, $y$ is $\pm\sqrt{1-x^2}$: only its sign is left to chance.

The ring is symmetric: a point above the $x$-axis is as likely as its mirror image below it. So in the long run the products cancel: $\mathbb{E}\{\cos\theta\sin\theta\}=\tfrac12\mathbb{E}\{\sin2\theta\}=0$. Both means are 0 too, so $\rho_{xy}=0$ exactly. The 500 drawn points measure −0.039.

So uncorrelated does not mean independent. For the instrument’s pairs the two do coincide: at $\rho=0$ the recipe gives $y=z$, which is independent of $x$. Statistics shows that this holds for every pair of jointly Gaussian values.

Worked example

1. Gaussian areas. Take the kitchen scale of 18.2: noise of standard deviation 0.2 kg on readings at 50 per second. Within ±0.2, ±0.4 and ±0.6 kg of the true weight lie 68.27 %, 95.45 % and 99.73 % of the readings.

So 0.27 % land further out than 0.6 kg, one reading in 370. At 50 readings a second, that happens about once every 7.4 s.

2. Uniform. In 11.1’s 3-bit quantizer the step is $\Delta=0.25$, so the rounding error is uniform from −0.125 to 0.125. Its density is $1/0.25=4$, its variance $0.25^2/12=0.005208$, and its standard deviation $0.25/\sqrt{12}=0.0722$.

A width of $2\sqrt3=3.4641$ instead gives variance $12/12=1$ and density $1/3.4641=0.2887$.

3. Estimates from the seeded draws. With seed 241 the Gaussian draws estimate the variance as 1.255 from 10 draws and 1.027 from 10 000. The means are −0.005 and −0.002.

4. Making a correlation. For $\rho=0.6$, $\sqrt{1-0.36}=0.8$, so $y=0.6x+0.8z$. The 500 seeded pairs measure 0.543.

Where you’ll meet this

Every receiver and measuring instrument has a noise budget. Its noise sources are usually independent, so their variances add, as here. Two sensors whose errors have correlation $\rho_{xy}$ and equal variance $\sigma_x^2$ average to a variance of $\sigma_x^2(1+\rho_{xy})/2$: half when the errors are uncorrelated, no gain at all when $\rho_{xy}=1$.

Quantization noise is modelled as uniform, as in 11.1, and the thermal noise of sensors and radios as Gaussian. How often Gaussian noise crosses a threshold, the area beyond a few standard deviations, is the question Matched filters and detection (26.1) answers.

Next, Random processes (24.2) lets draws depend on their neighbours, and Correlation (24.3) measures the correlation of a signal with a shifted copy of itself or of another signal.

NumPy computes this page’s estimates with histogram (its density=True scales the bars as here), mean, var and corrcoef. For more, see appendix A, “Random signals”, of A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing (3rd ed., 2010). See also A. Papoulis and S. U. Pillai, Probability, Random Variables and Stochastic Processes (4th ed., 2002), chapters 4 to 6.

Reference card