Partial derivatives and the gradient

You stand on a hillside. Which way is steepest? Below, an arrow answers at every point of a map. Before it plays: will the arrow point straight away from the bottom?

Surfaces and contours

A loss with two parameters is a surface: over each point $\boldsymbol\theta=(\theta_1,\theta_2)$ of the floor, the height $J(\boldsymbol\theta)$. The two numbers $\theta_1$ and $\theta_2$ are things a model could adjust, its parameters; on these pages $\theta$ is never an angle. This surface is a bowl with its bottom at $(2,1)$, steeper along $\theta_2$ than along $\theta_1$ because of the 5.

A map draws the surface from above, as a walker’s map draws hills. Each ellipse is a contour: every point where the loss has one value. Walking along a contour, you neither climb nor descend. Here the levels are 4 apart, so the gap between neighbouring ellipses says how far you must walk to climb 4: a short walk where they crowd, a long one where they spread.

Slopes with one input frozen

The slope of a surface depends on the direction you face. Start with the two easiest: along $\theta_1$, holding $\theta_2$ still, and along $\theta_2$, holding $\theta_1$ still.

Freeze $\theta_2$ at 3. Then $J=(\theta_1-2)^2+5\cdot2^2=(\theta_1-2)^2+20$ is a curve in one variable, $\theta_1$, and The derivative as a slope gives its slope, $2(\theta_1-2)$. That slope is the partial derivative of $J$ with respect to $\theta_1$, written $\partial J/\partial\theta_1$. It is the ordinary derivative, taken with every other input treated as a constant. Doing the same for each input,

The curly $\partial$ only says that other inputs were held still.

The gradient

Put the two slopes together as one arrow, the gradient:

It is a vector, not a number: it has a direction and a length at every point. At $(-2,3)$ it is $(-8,20)$.

The slope in any direction. Face a direction given by a unit vector $\mathbf{u}=(u_1,u_2)$ and take a tiny step of length $\delta$ along it. The step moves $\theta_1$ by $\delta u_1$ and $\theta_2$ by $\delta u_2$, so, over so small a step, the loss changes by about $\delta\,u_1\,\partial J/\partial\theta_1+\delta\,u_2\,\partial J/\partial\theta_2$. Per unit of distance, the slope along $\mathbf{u}$ is a dot product:

which is $\lVert\nabla J\rVert$ times the cosine of the angle between $\mathbf{u}$ and the gradient. Read off the three cases:

• facing along the gradient, cosine 1: the slope is $\lVert\nabla J\rVert$, the largest there is. The gradient points straight uphill, and its length is the steepest slope.
• facing the opposite way: the slope is $-\lVert\nabla J\rVert$, steepest downhill.
• facing at right angles to it: the slope is 0. That is the direction of the contour, where the loss does not change. So the gradient crosses its contour at a right angle, never along it.

Now the answer to the question. The arrow points the steepest way up from where it stands, square to the contour through that point. On this stretched bowl that is straight away from the bottom only at the four ends of the ellipse’s axes; everywhere else on the ellipse it leans towards the steep direction, $\theta_2$.

Length and crowding. To climb the 4 between neighbouring contours you walk about $4/\lVert\nabla J\rVert$ along the arrow. A long arrow means a short walk, so the contours crowd; a short arrow means they spread. On the ellipse where $J=16$ the arrow is 8 long at the ends of the long axis and $10\sqrt{3.2}=17.9$ at the ends of the short one.

At the bottom

At $(2,1)$ both partial derivatives are zero, so $\nabla J=(0,0)$: no direction is uphill. The bottom of a smooth bowl is where the gradient vanishes, as the bottom of a valley in one variable is where the slope does. The bowl itself is a quadratic form about its bottom, the kind of surface Positive definite matrices draws.

Worked example

The gradient at the start. At $(-2,3)$, $\nabla J=(2(-4),\ 10(2))=(-8,20)$. Its length is $\sqrt{64+400}=\sqrt{464}=21.54$.

Slopes in four directions.

• along $(1,0)$: $\nabla J\cdot(1,0)=-8$;
• along $(0,1)$: $20$;
• along the contour, $(20,8)/\sqrt{464}$: $(-8\cdot20+20\cdot8)/\sqrt{464}=0$;
• along $\nabla J/\lVert\nabla J\rVert$: $464/\sqrt{464}=\sqrt{464}=21.54$, the steepest.

Downhill is not towards the bottom. Downhill from $(-2,3)$ is $(8,-20)$, but the bottom lies along $(2,1)-(-2,3)=(4,-2)$. The cosine of the angle between them is

so the steepest way down misses the bottom by 41.6° (0.727 rad). Gradient descent starts from this point and pays for that miss with a zig-zag.

More points. At $(2,1)$, $\nabla J=(0,0)$. At $(0,1)$, $\nabla J=(-4,0)$: uphill is along $-\theta_1$. At $(2,2)$, $\nabla J=(0,10)$: uphill is along $+\theta_2$.

A second function. Take $g(\theta_1,\theta_2)=\theta_1^2\theta_2+3\theta_2$; I call it $g$ because a bare $f$ is a frequency. Holding $\theta_2$ still, $\theta_1^2\theta_2$ is a constant times $\theta_1^2$, so $\partial g/\partial\theta_1=2\theta_1\theta_2$. Holding $\theta_1$ still, $g$ is $(\theta_1^2+3)$ times $\theta_2$, so $\partial g/\partial\theta_2=\theta_1^2+3$. At $(1,2)$ the gradient is $(4,4)$. A central difference checks the first entry: $\big(g(1.1,2)-g(0.9,2)\big)/0.2=(8.42-7.62)/0.2=4$.

A round bowl. For $\theta_1^2+\theta_2^2$ at $(3,4)$ the gradient is $(6,8)$, of length 10, and it points straight away from the bottom at the origin: on a round bowl, uphill is directly away from the bottom everywhere.

Where you’ll meet this

Edges in images. Edge detectors take the gradient of an image’s brightness: long arrows where the brightness changes fast, which is at an edge.

Water and heat. Water runs straight down a slope, against the gradient of height, crossing the contour lines of a map at right angles. Heat flows against the gradient of temperature.

Fitting models. Every training step in this course moves against a gradient. Gradient descent does it on this same bowl.

Reference card