Functions as vectors

A curve is a list of heights, one at every point. Watch the best constant, line, parabola and cubic close in on $e^x$.

A curve is a long list

A vector in three dimensions is a list of three numbers. A sampled signal of 1000 samples is a list of 1000 numbers, a vector in 1000 dimensions. Now let the samples get closer and closer together. In the end there is a height at every point of an interval, and the list has become a curve.

Curves can be added and scaled like vectors: $f+g$ adds the heights at each point, and $2f$ doubles them. So a curve is a vector with a height at every point. The question is how to measure lengths and angles between them.

The inner product

For two lists, the dot product multiplies matching entries and adds them up. For two curves on an interval from $a$ to $b$, multiply the heights at each point and add them up with an integral:

This is the inner product of the two functions. The angle brackets replace the dot. The length follows as before, $\lVert f\rVert^2=\langle f,f\rangle$. On the interval from $-1$ to $1$, the constant function 1 has $\lVert1\rVert^2=\int_{-1}^1 1\,\mathrm{d}x=2$, so its length is $\sqrt2$.

Two functions are orthogonal when their inner product is zero. That does not mean their curves cross at a right angle. It means that where $f\,g$ is positive and where it is negative, the areas cancel.

So $\sin x$ and $\sin 2x$ are orthogonal on $[-\pi,\pi]$, while $\langle\sin x,\sin x\rangle=\pi$. On $[-1,1]$, the functions 1 and $x$ are orthogonal: $\int_{-1}^1x\,\mathrm{d}x=0$, because the area left of 0 cancels the area right of it. The same reason makes $\sin x$ and $\cos x$ orthogonal on $[-\pi,\pi]$: their product is odd.

Orthogonal polynomials

The powers $1,x,x^2,x^3$ are not orthogonal on $[-1,1]$: for example $\langle1,x^2\rangle=\tfrac23$. Run Gram–Schmidt on them, with the integral as the dot product, and you get the Legendre polynomials:

with $\langle P_k,P_k\rangle=\dfrac{2}{2k+1}$. They are written with an argument, $P_k(x)$, so they are not confused with a projection matrix. This page keeps their usual scale, which is not length 1.

Projecting a curve

The best fit to $f$ by a combination of orthogonal functions $u_k$ is the projection of Projections, with integrals in place of dot products:

“Best” means that $\lVert f-p\rVert$, the length of the gap, is as small as any combination can make it. Each coefficient is a separate shadow, so adding a new $u_k$ never changes the earlier coefficients. That is the second readout in the picture at the top of the page.

Here is $f(x)=e^x$ on $[-1,1]$:

• On $P_0$: $\langle e^x,1\rangle=e-e^{-1}=2.3504$, divided by 2: $1.1752$. The best constant is the average height of the curve.
• On $P_1$: $\langle e^x,x\rangle=2/e=0.7358$, divided by $\tfrac23$: $1.1036$.
• On $P_2$: $\langle e^x,P_2\rangle=e-7/e=0.14313$, divided by $\tfrac25$: $0.3578$.
• On $P_3$: the coefficient is $0.0705$.

Written in powers of $x$, the best parabola is $0.9963+1.1036x+0.5367x^2$. The coefficients in powers of $x$ do change from fit to fit, because the powers are not orthogonal; only the coefficients on the $P_k$ stay fixed.

The gap shrinks fast. Its squared length is $1-e^{-2}=0.8647$ after the constant, $1-7e^{-2}=0.05265$ after the line, and $0.00144$ after the parabola. Each step removes the square of the new shadow’s length, the right-angle rule again. The lengths in the readout are the square roots: 0.930, 0.229, 0.038, and 0.0047 after the cubic.

Compare a Taylor parabola, $1+x+x^2/2$, which copies $e^x$ perfectly near $x=0$. At $x=1$ it gives $2.5$, while the projection gives $2.637$ and the truth is $2.718$. Over the whole interval, the largest miss of the Taylor parabola is $0.218$, and of the projection only $0.082$. Taylor is best at one point; the projection is best across the interval.

Worked example

1. Orthogonal. On $[-1,1]$, $\langle1,x\rangle=0$, and $\lVert1\rVert=\sqrt2$.
2. Sines. On $[-\pi,\pi]$, $\langle\sin x,\sin2x\rangle=0$ and $\langle\sin x,\sin x\rangle=\pi$; $\sin x$ and $\cos x$ are orthogonal, odd times even.
3. The best constant. For $e^x$ on $[-1,1]$, the best constant is its average, $1.1752$.
4. The best parabola. $0.9963+1.1036x+0.5367x^2$, with a gap of length $0.038$.
5. A straight line by sines. Project $f(x)=x$ on $[-\pi,\pi]$ onto $\sin x$, $\sin2x$, $\sin3x$. Each coefficient is $\langle x,\sin kx\rangle/\pi=2(-1)^{k+1}/k$: $2$, $-1$ and $\tfrac23$. The squared length of $x$ is $2\pi^3/3=20.671$. After one, two and three sines, the squared gap is $8.104$, $4.963$ and $3.567$: each sine removes $\pi$ times its coefficient squared.

Where you’ll meet this

A maths library computes $\sin x$ or $e^x$ by first moving $x$ into a short interval, then evaluating a polynomial chosen to be close to best there. A Fourier series is a projection onto sines and cosines, and keeping more of them shrinks the gap, as item 5 does for a straight line.

Reference card