The derivative as a slope: derivative

How steep is a curve right here, at one point? Below, a line through two points becomes the slope at one. Before it plays, sketch the slope curve: what shape will it be?

The slope at a point

The loss J(w) = 15 − 21w + 7.5w² above, its slope J′(w) below, on one w axis.

The loss curve of the last lesson, J(w) = 15 − 21w + 7.5w², and a point on it at w = 0.5.

point w

0.50

gap h

slope of the line

−6

Point, w0.50

Describe this picture

Two panels stacked on one shared $w$ axis from 0 to 2.5. Above, the loss curve of Models and loss, $J(w)=15-21w+7.5w^2$ , a bowl lowest at $w=1.4$ . Below, the panel of its slope, empty at first. In the clip a point sits on the curve at $w=0.5$ . A second point slides in towards it from $w=1.5$ and pauses at gaps $h$ of 1, 0.5, 0.1 and 0.01, where the line through the two has slope −6, −9.75, −12.75 and −13.425. Then the two points meet, and the line just touches the curve: the tangent, slope −13.5. The tangent goes back to $w=0$ and rides along the curve to $w=2.5$ , while a vertical link line joins the point above to a dot below at the height of the tangent’s slope. The dot traces the slope curve: a straight line rising from −21, negative on the left, crossing zero at $w=1.4$ , where the crossing is marked, and positive on the right. Readouts give the point $w$ , the gap $h$ and the slope of the line. After the clip a slider, “Point, w”, runs from 0 to 2.5 in steps of 0.01 and starts at 2.5; the point can also be dragged across either panel, and the link line follows.

From two points to one

The slope between two points on a curve is rise over run. Take the point at $w$ and a second point a small step $h$ further on. The line through them, a secant, has slope

\frac{J(w+h)-J(w)}{h}.

Here $h$ is a small number, not a filter’s impulse response (that one is written $h[n]$ , with brackets). As $h$ shrinks, the second point slides towards the first and the secant turns, settling onto one line, the tangent. Its slope is the derivative of $J$ at $w$ :

J'(w)=\frac{\mathrm dJ}{\mathrm dw}=\lim_{h\to0}\frac{J(w+h)-J(w)}{h}.

Both ways of writing it say the same thing: the rate at which $J$ changes per unit of $w$ , right at $w$ . The derivative is a slope, so it has the units of $J$ per unit of $w$ .

For the curve in the picture, the secant slopes from $w=0.5$ work out to $-13.5+7.5h$ . At $h=1$ that is −6, at $h=0.5$ it is −9.75, at $h=0.1$ it is −12.75, and as $h$ goes to 0 it tends to −13.5, the slope of the tangent.

Now the answer to the question. The slope curve is a straight line. The curve above is a bowl, but its slope is steeply negative on the far left, rises steadily and passes through zero at the bottom. A graph of slopes need not look like the graph it came from.

Watch out

The slope curve is not a copy of the curve. Read it as “how steep”, never as “how high”.

Rules for slopes

Working out a limit every time would be slow. A few rules cover most of what this course needs. For a power of $w$ , with $a$ a fixed number,

\frac{\mathrm d}{\mathrm dw}\,w^a=a\,w^{a-1},

so the slope of $w^2$ is $2w$ and the slope of $w$ is 1. A constant has slope 0. A constant multiple keeps its multiple, and the slope of a sum is the sum of the slopes, term by term. Two more are worth knowing by their pictures, and I state them without proof:

\frac{\mathrm d}{\mathrm dw}\,e^w=e^w,\qquad \frac{\mathrm d}{\mathrm dw}\,\ln w=\frac1w.

The curve $e^w$ is its own slope everywhere. The curve $\ln w$ (for $w>0$ ) climbs ever more gently, with slope $1/w$ .

Four curves, each above its slope curve on the same w axis: w² over 2w, w³ over 3w², eʷ over itself and ln w over 1/w. At w = 0 the tangent to w³ is flat, a slope of 0, yet it crosses the curve there: a zero slope need not be a bottom.

Describe this picture

Four small panels, each a curve above its slope curve on a shared $w$ axis: $w^2$ , a bowl, over $2w$ , a straight line through 0; $w^3$ over $3w^2$ , a bowl that touches 0 at $w=0$ ; $e^w$ , rising ever faster, over the same curve; and $\ln w$ for $w$ from 0.2 to 4, rising ever more slowly, over $1/w$ , falling towards 0. In the $w^3$ panel the tangent at 0 is drawn: the flat line at height 0, which crosses the curve there. The caption ends “a zero slope need not be a bottom”. There is no control.

For the loss curve, term by term:

J'(w)=\frac{\mathrm d}{\mathrm dw}\big(15-21w+7.5w^2\big)=0-21+15w=-21+15w.

That is the straight line in the lower panel. At $w=0.5$ it gives $-21+7.5=-13.5$ , the tangent’s slope found above.

Finding the bottom

At the bottom of a smooth valley the tangent is flat, so the slope is zero. For the loss curve, $-21+15w=0$ gives $w=1.4$ : the best slope of Models and loss, found there by completing the square, here in one line.

The sign of the slope says which way the bottom lies. Left of 1.4 the slope is negative: the curve falls as $w$ grows, so the bottom is to the right. Right of 1.4 it is positive, so the bottom is to the left. At $w=2$ , $J'(2)=-21+30=9$ : positive, so the bottom is to the left. Gradient descent, later in the course, steps against that sign.

A zero slope does not always mean a bottom. At the top of a hill, $-w^2$ at 0, the slope is zero too. And $w^3$ has zero slope at 0 while it keeps rising on both sides: there the flat tangent crosses the curve instead of resting on it. A tangent can cross its curve; what defines it is the slope, not touching once.

Watch out

Zero slope means a bottom, a top or a flat step. Check which: on a bowl like the loss here, it is the bottom.

Slopes from numbers

Sometimes there is no formula to differentiate, only a way to compute the curve. Then the secant itself is the estimate. The forward difference uses one point ahead:

\frac{J(w+h)-J(w)}{h},

and the central difference one point on each side:

\frac{J(w+h)-J(w-h)}{2h}.

The central difference is much more accurate for the same $h$ . Each side’s error has a part of size $h$ , and those two parts are equal and opposite, so they cancel; what is left is of size $h^2$ . For a parabola the central difference is exact at any $h$ .

In signal processing · the first difference

Amplitude operations in the signal processing course takes the first difference of a signal, each sample minus the one before: a forward difference with a step of one sample, the derivative’s sampled cousin. It is near zero at a peak or a trough and largest where the signal is steepest.

Worked example

Secants. On $J(w)=15-21w+7.5w^2$ : $J(0.5)=15-10.5+1.875=6.375$ , $J(1)=1.5$ and $J(1.5)=15-31.5+16.875=0.375$ . From $w=0.5$ :

$h=1$ : $(0.375-6.375)/1=-6$ ;
$h=0.5$ : $(1.5-6.375)/0.5=-9.75$ ;
in general $-13.5+7.5h$ : $-12.75$ at $h=0.1$ , $-13.425$ at $h=0.01$ , tending to $-13.5$ .

A step backwards works too: $h=-0.5$ gives $-13.5-3.75=-17.25$ .

The rule. $J'(w)=-21+15w$ , so $J'(0.5)=-13.5$ and $J'(w)=0$ at $w=1.4$ .

A second curve. The slope of $3w^2-4w$ is $6w-4$ , zero at $w=2/3$ .

Numerical slopes. For $e^w$ at 0, whose true slope is 1, with $h=0.1$ : the central difference is $(1.105171-0.904837)/0.2=1.00167$ , the forward difference $(1.105171-1)/0.1=1.05171$ . The central one is about thirty times closer. For $\ln w$ at 2, true slope 0.5, the central difference is $(0.741937-0.641854)/0.2=0.50042$ .

Exact for a parabola. For the loss curve at $w=0.5$ with the large step $h=1$ : $J(1.5)=0.375$ and $J(-0.5)=15+10.5+1.875=27.375$ , so the central difference is $(0.375-27.375)/2=-13.5$ , the exact slope.

Where you’ll meet this

Speed. A speedometer shows the derivative of distance travelled: metres per second, right now.

Sensitivity. A sensor’s sensitivity is the slope of its calibration curve: how much the output moves per unit of what it measures.

Fitting models. Every training method in this course follows slopes downhill. Partial derivatives and the gradient takes the slope of a loss with two parameters, and Gradient descent walks down it.

In linear algebra · differentiation as a matrix

The matrix of a linear map in the linear algebra course writes differentiation of a cubic as a 4×4 matrix acting on its coefficients. The rules here, sums term by term and constant multiples kept, are what make differentiation linear.

Reference card

Quantity	Formula	Notes
Derivative	$J'(w)=\lim_{h\to0}\frac{J(w+h)-J(w)}{h}$	the tangent’s slope
Power	$\frac{\mathrm d}{\mathrm dw}w^a=aw^{a-1}$	$a$ a fixed number
Sums, multiples	term by term; constants keep their multiple	a constant’s slope is 0
Exponential, logarithm	$(e^w)'=e^w$ , $(\ln w)'=1/w$
Bottom of a valley	$J'(w)=0$	also a top or a flat step: check
Sign	negative: bottom to the right; positive: to the left
Central difference	$\frac{J(w+h)-J(w-h)}{2h}$	error of size $h^2$ ; exact on a parabola
Forward difference	$\frac{J(w+h)-J(w)}{h}$	error of size $h$

The derivative as a slope

The slope at a point

From two points to one

Rules for slopes

Finding the bottom

Slopes from numbers

Worked example

Where you’ll meet this

Reference card

Keep reading with a free account.

Are you 13 or over?

Where to go next.

Account

Subjects

Site

Look

The derivative as a slope

First, the picture

From two points to one

Rules for slopes

Finding the bottom

Slopes from numbers

Worked example

Where you’ll meet this

Reference card