Transforming random variables

Drop dots evenly down the side of a box, bounce them off a curve, and let them fall. Watch them pile into a new shape.

Even dots in, any shape out

Computers make random numbers that are spread evenly between 0 and 1: a uniform variable $U\sim\mathrm{U}(0,1)$, from Continuous random variables. A simulation usually needs something else: waiting times, noise, heights. The picture shows how to get any shape from even dots.

Each dot starts at a height $u$ and moves across to the curve $F_X$, then drops to the $x$ where $F_X(x)=u$. That $x$ is written $F_X^{-1}(u)$, the inverse of the distribution function: it answers “at which value has the curve climbed to $u$?”. So the method is

Why does the pile take the shape of $F_X$? Because $F_X$ climbs, $F_X^{-1}(U)\le x$ happens exactly when $U\le F_X(x)$. And a uniform $U$ lands below any level with chance equal to that level:

The new variable has distribution function $F_X$, which is what I wanted. This is the inverse distribution function method, often called the inverse CDF method.

Exponential waits. For $\mathrm{Exp}(\lambda)$, $F_X(x)=1-e^{-\lambda x}$. Solve $1-e^{-\lambda x}=u$ for $x$:

With $\lambda=1$, the level $u=0.3$ gives $-\ln0.7=0.3567$.

Steep and flat

Look at where the dots land. A band of starting heights from $u$ to $u+\Delta u$ catches the same share of dots, $\Delta u$, wherever it is. Where the curve is steep, that band meets the curve over a narrow stretch of $x$, so its dots land close together and the pile is tall. Where the curve is flat, the same share of dots is spread over a wide stretch of $x$, and the pile is low.

That is the density again: the slope of $F_X$ is $f_X$. Steep means a high density, flat means a low one.

The density of a function

The same idea answers a general question. If $X$ has a known density and $Y=g(X)$ for some function $g$, what is the density of $Y$?

Take $g$ increasing, so that it keeps order. Then $Y\le y$ exactly when $X\le g^{-1}(y)$, and

Take the slope of both sides, by the chain rule. For $g$ decreasing the same steps give a minus sign, and the absolute value covers both:

The last factor is the stretch factor. Where $g$ stretches a stretch of $x$ into a wider stretch of $y$, the same probability is spread thinner, and the density drops by that factor.

A small case: $X\sim\mathrm{U}(0,1)$ and $Y=2X+1$. Then $g^{-1}(y)=(y-1)/2$, its slope is $\tfrac12$, and $Y$ has density $1\times\tfrac12=\tfrac12$ on $(1,3)$: the same flat shape, twice as wide and half as tall.

Three shapes made by transforming

Rayleigh. Take a point whose two coordinates are independent and each $\mathcal{N}(0,\sigma^2)$, and let $R$ be its distance from the origin. $R$ follows the Rayleigh law, with $F_R(r)=1-e^{-r^2/(2\sigma^2)}$ for $r\ge0$. Its inverse gives the recipe

and its mean is $\sigma\sqrt{\pi/2}=1.2533\,\sigma$.

Log-normal. If $Z$ is standard normal, $Y=e^Z$ is log-normal: always positive, with a long tail to the right. An increasing map keeps the middle value in the middle, so its median is $e^0=1$. Its mean is larger, $e^{1/2}=1.6487$, pulled up by the tail.

Chi-square with one degree of freedom. $Y=Z^2$ is the square of a standard normal. Its mean is $\mathbb{E}\{Z^2\}=1$, the variance of $Z$. The square is not increasing over all of $z$, since $z$ and $-z$ give the same $y$; its density adds the two halves, one from each side.

Worked example

1. Squaring a uniform. Let $X\sim\mathrm{U}(0,1)$ and $Y=X^2$, which also lies between 0 and 1. Squaring is increasing there, so

The density is tall near 0 and low near 1: squaring pushes small values closer to 0. Half of the values of $Y$ lie below 0.25, since $F_Y(0.25)=\sqrt{0.25}=0.5$.

Its mean is $\mathbb{E}\{X^2\}=\int_0^1x^2\,dx=\tfrac13$, not $\left(\tfrac12\right)^2=\tfrac14$, as Expected value warned.

2. Reading levels off three curves. Here is what the slider gives at a few levels $u$, for the exponential with rate $\lambda=1$, the Rayleigh with $\sigma=1$ and the standard normal:

The standard normal at 0.975 gives 1.96, the edge of the 95 % band of The normal distribution: 2.5 % lies above it and 2.5 % below $-1.96$.

Try it yourself. An exponential draw with rate 1 from the level $u=0.3$: $-\ln0.7=0.3567$.

Where you’ll meet this

Simulation. Simulators of queues, networks and markets need random draws of every shape. Most start from uniform numbers and use this page’s method, or a cousin of it, for each law.

The envelope of noise. The strength of narrow-band radio noise, its envelope, is the distance from the origin of two independent bell-shaped parts: a Rayleigh variable. Fading radio channels are described the same way.

Decibels and growth. Quantities that grow by multiplying, such as incomes or signal strengths through many layers of fading, are often log-normal: their logarithm, in decibels, is a bell.

Squared errors. The square of a standard bell-shaped value is a chi-square variable; a bell-shaped error with mean 0, squared and divided by its variance, is one, and so is the power of a bell-shaped noise sample scaled the same way. Chi-square tests are built on sums of them.

Reference card