Time series and autocorrelation

Plot a series against itself one step later, then two, then ten. Watch the cloud loosen as the lag grows.

Lag plots

Before the clip you guessed. At lag 1 the cloud is a slanted band: a high value tends to be followed by a high one. As the lag grows the band loosens, and by lag 10 it is nearly a round blob: the series has almost forgotten. The clip’s series measured 0.8124, 0.6236, 0.4729, 0.3520 and 0.2864 at lags 1 to 5, against 0.8, 0.64, 0.512, 0.410 and 0.328 in theory.

A time series is a sequence of values in time order: daily temperatures, monthly sales, a sensor read every second. I write the observed values $x_k$, $k=1,\dots,n$, and the random series behind them $X_k$. Signal processing pages write the same thing $x[n]$, with $n$ the time; here $n$ is the number of values.

The correlation of the lag plot at lag $\ell$ measures how much the series remembers $\ell$ steps back. Taken lag by lag, it is the series’ autocorrelation function.

Trend, season and noise

Many series are three things added together: a slow trend, a season that repeats with a fixed period, such as 12 months, and noise:

A series is stationary if its behaviour does not drift over time: the mean stays the same, and so does the covariance between values a fixed number of steps apart. Trends and seasons break stationarity. They are usually removed first, as Smoothing and seasonality shows, and what is left is studied for its memory.

The autocorrelation function

For a stationary series with mean $\mu_X$ and variance $\sigma_X^2$, the autocovariance at lag $\ell$ is

the same for every $k$, and the autocorrelation is $\rho_X[\ell]=C_X[\ell]/C_X[0]$. It is 1 at lag 0. The lag goes in square brackets; the log-likelihood $\ell(\theta)$ of Maximum likelihood is a different use of the letter.

From data, use the sample mean $\bar x$ and divide every lag by the same total:

White noise. A series of independent values with mean 0 has $\rho_X[\ell]=0$ at every lag except 0. Its sample autocorrelations are not exactly 0; for about 95 % of lags they fall within $\pm1.96/\sqrt n$. For 200 values that band is $\pm0.139$, the band in the picture.

Two extremes

A series with fading memory. The picture’s series is an AR(1) series, short for autoregressive of order 1:

where $V_k$ is white noise with variance $\sigma_V^2$ and $\lvert\phi\rvert<1$. Its autocorrelation decays geometrically, $\rho_X[\ell]=\phi^\ell$, and its variance is $\sigma_X^2=\sigma_V^2/(1-\phi^2)$. Here $\phi$ is a coefficient; signal processing pages use the letter for a phase.

A series with no forgetting. With $\phi=1$ each step adds fresh noise to the last value: a random walk. Its variance after $k$ steps is $k\sigma_V^2$, growing without limit, so it is not stationary, and its sample autocorrelation barely decays. Take its differences, $x_k-x_{k-1}$, and you get the white noise back.

Trends that look related

Two series that both wander can look related when they are not.

In the clip, the first pair of walks has a correlation of 0.4736, and 377 of the 1000 independent pairs, 37.7 %, land beyond ±0.5.

Cross-correlation

The same idea compares two series. If $y$ is a delayed copy of $x$ plus noise, correlate $y_k$ with $x_{k-\ell}$ for each lag $\ell$. The correlation peaks at the lag equal to the delay. This is the cross-correlation $\rho_{YX}[\ell]$, written with the delayed series first, as in Correlation of the signal processing course.

Worked example

1. A short series. The series $1,2,4,5,4,3,2,3$ has mean 3 and deviations $-2,-1,1,2,1,0,-1,0$, whose squares add to 12. The sums of lagged products are 5 at lag 1, $-4$ at lag 2 and $-7$ at lag 3, so

With only 8 values the white-noise band is $\pm1.96/\sqrt8=\pm0.693$, so none of these stands out: eight points cannot tell a pattern from noise. The plain correlation of the seven lag-1 pairs is $0.5296$, not $0.4167$, because it uses each set’s own mean and a different divisor; for long series the two agree closely.

2. The AR(1) series. With $\phi=0.8$ and $\sigma_V=1$, $\sigma_X^2=1/0.36=2.7778$ and $\sigma_X=1.667$. The autocorrelations are $0.8,\ 0.64,\ 0.512,\ 0.4096,\ 0.32768$ at lags 1 to 5. For 100 values the band is $\pm0.196$, and the theory drops below it at lag 8, where $0.8^8=0.168$.

3. Many lags of white noise. Check 20 lags against a 95 % band: on average one falls outside, and the chance that at least one does is $1-0.95^{20}=0.6415$, the many-tests problem of p-values.

4. A delay. If $y$ is $x$ delayed by 3 samples plus noise, $\rho_{YX}[\ell]$ peaks at $\ell=3$.

5. Trend and season. Monthly data with a rising trend and a yearly cycle have an autocorrelation that stays high at every lag and swings with period 12, peaking at lags 12, 24 and so on.

Try it yourself. For $\phi=0.8$, $\rho_X[2]=0.64$. For 400 values the band is $\pm0.098$. A random walk of unit steps has variance 100 after 100 steps.

Where you’ll meet this

Is it noise? Checking that a signal, or the residuals of a fitted model, has no autocorrelation beyond the band is a standard check that nothing systematic is left. Linear regression assumes exactly that.

Delays and echoes. The delay between two microphones, two sensors or a sound and its echo is the lag of a cross-correlation peak.

Hidden periods. A peak in the autocorrelation at lag $P$ reveals a cycle of period $P$: a machine’s vibration, a heartbeat, a weekly pattern in web traffic.

Reference card