Autoregressive models and forecasting

A series ends on a high value. Forecast the next steps, and watch the forecast lean back while its interval opens like a fan.

A forecast and its interval

Before the clip you guessed. The fan opens fast at first and then stops: its half-width creeps up towards 3.27 and never passes it. Each step ahead adds new noise to the uncertainty, but the series also forgets, and the two balance. In the clip, 95.7 % of the points of 1000 simulated futures fell inside the fan, against the nominal 95 %.

The forecast itself leans back to the mean. The series is high now, and it remembers that, but its memory fades by a factor of 0.8 a step, so the forecast fades too.

After the clip, drag the coefficient to 1. The series becomes a random walk, which never forgets: the forecast stays at 5, and the fan keeps opening, wider and wider, as the square root of the horizon.

The autoregressive model

The picture’s series is the AR(1) series of Time series and autocorrelation. Allowing more of the past gives the autoregressive model of order $p$, AR($p$):

with $V_k$ white noise of variance $\sigma_V^2$ and coefficients $\phi_1,\dots,\phi_p$. Today’s value is a weighted sum of recent values plus a fresh surprise.

The model is stationary when its memory fades rather than builds up. For AR(1) that means $\lvert\phi\rvert<1$. For AR($p$) it means every root of $1-\phi_1z-\dots-\phi_pz^p$ lies outside the unit circle. A coefficient of 1.05 makes AR(1) explode, not settle.

Fitting the model

By regression. An AR($p$) model is a regression of each value on its $p$ predecessors, so Linear regression fits it: least squares of $x_k$ on $x_{k-1},\dots,x_{k-p}$.

By the autocorrelation. Multiply the model by $X_{k-\ell}$ and take means. For AR(2), with $\rho_\ell$ short for $\rho_X[\ell]$, lags 1 and 2 give the Yule–Walker equations:

Put in the sample autocorrelations and solve for the coefficients. The matrix is symmetric with constant diagonals, the structure of Positive definite matrices.

Partial autocorrelation

How do you pick the order $p$? The autocorrelation of an AR(1) series does not stop at lag 1; it fades like $\phi^\ell$. A better tool is the partial autocorrelation at lag $\ell$: the last coefficient of the best AR($\ell$) fit. It measures what lag $\ell$ adds once the lags in between are accounted for. For an AR($p$) series it is 0 beyond lag $p$: it cuts off, and the cut-off shows the order. Criteria such as the AIC of Overfitting and model choice make the choice a rule.

Forecasts and their uncertainty

For AR(1) about a mean of 0, the forecast $h$ steps ahead of the last value $x_n$ is

which decays to the mean. Its error collects the noise of the $h$ steps not yet seen, each faded by the steps after it, so its variance is

rising towards the series’ own variance $\sigma_X^2=\sigma_V^2/(1-\phi^2)$: far ahead, the forecast knows no more than the mean. For a random walk, $\phi=1$, the variance is $h\sigma_V^2$ and grows without limit.

The same model as a filter

An AR(1) series is white noise passed through a one-pole filter. Its spectrum shows the same memory in frequency.

Slow swings dominate: the series’ memory, $0.8^\ell$ in the lag domain, is the same fact as its power at low frequencies.

Moving-average models, in one line

A different model mixes recent noise instead of recent values: the moving-average model MA(1), $X_k=V_k+\theta V_{k-1}$, has $\rho_X[1]=\theta/(1+\theta^2)$ and no autocorrelation beyond lag 1. Combining both kinds gives ARMA models.

Worked example

1. Fitting the short series. For the series $1,2,4,5,4,3,2,3$ of Time series and autocorrelation, with deviations $d_k$ from the mean 3, the regression of each deviation on the one before gives

the same as $\hat\rho_X[1]$, because the last deviation, $d_8$, is 0.

2. Forecasting. With $\phi=0.8$, $\sigma_V=1$ and $x_n=5$:

At 12 steps ahead the half-width is 3.26, nearly at its limit. With $\phi=0.5$ the limit is lower, 2.26. A random walk has a half-width of $1.96\times2=3.92$ at 4 steps and $6.79$ at 12, still growing.

3. Yule–Walker for AR(2). Take $\phi_1=0.5$ and $\phi_2=0.3$, a stationary pair. Its autocorrelations are $\rho_1=\phi_1/(1-\phi_2)=0.7143$ and $\rho_2=\phi_1\rho_1+\phi_2=0.6571$. Solving the two equations with these values returns $(0.5,\ 0.3)$: the determinant is $1-\rho_1^2=0.4898$, $\phi_1=\rho_1(1-\rho_2)/0.4898=0.5$ and $\phi_2=(\rho_2-\rho_1^2)/0.4898=0.3$. The partial autocorrelation is 0.3 at lag 2 and 0 at lag 3.

4. A moving average. MA(1) with $\theta=0.5$ has $\rho_X[1]=0.5/1.25=0.4$; no MA(1) gets above 0.5, reached at $\theta=1$.

Try it yourself. Two steps ahead from $x_n=5$ with $\phi=0.8$, the forecast is 3.2. A coefficient of 1.05 is not stationary. The partial autocorrelation of an AR(2) series is 0 beyond lag 2.

Where you’ll meet this

Forecasting. Demand, electricity load and temperature are forecast with autoregressive models and their fans; planners use the fan, not just the line.

Speech coding. Linear prediction fits an AR model to each short frame of speech, and sends the coefficients and the small prediction error instead of the samples.

Tracking. The Kalman filter often models the motion it tracks as an autoregressive process, predicting one step ahead and correcting with each measurement.

Reference card