Speak Maths · University

The gradient: which way is uphill

On a surface with many directions, the gradient is one arrow that tells you everything about the slope where you stand. Machine learning walks against it.

Draft: written by an AI helper and not yet checked by a teacher.

The alphabet

Read each sign aloud, the way it is written under it.

$\frac{\partial f}{\partial x}$ Say: partial f by partial xMeans: how fast f changes when only x moves and everything else stays still
$\nabla f$ Say: grad f (or “nabla f”, “del f”)Means: an arrow pointing the steepest way uphill
$\nabla f = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y})$ Say: grad f is the vector of partial f by partial x and partial f by partial yMeans: the gradient puts all the partial derivatives side by side
$∥ \nabla f ∥$ Say: the norm of grad fMeans: the length of the arrow: how steep the steepest way is
$D_{u} f = \nabla f \cdot u$ Say: the derivative of f in direction u is grad f dot uMeans: the slope if you walk in direction u (a unit vector)
$f (x, y) = c$ Say: f of x, y equals cMeans: a level set or contour: every point at the same height c
$θ \leftarrow θ - η \nabla L (θ)$ Say: theta becomes theta minus eta times grad L of thetaMeans: one step of gradient descent: move a little in the downhill direction

The story: The hiker in the fog

Subtitles:

Tap a line to see its words.

$h (x, y) = x^{2} + 2 y^{2}$ The height of a valley side above its floor depends on where you stand: x metres east and y metres north.
$(x, y) = (3, 1)$ Fog comes down. The hiker is 3 metres east and 1 metre north of the floor’s lowest point, but cannot see it.
$h (3, 1) = 11$ Here the ground is 11 metres above the floor.
$\frac{\partial h}{\partial x} = 2 x, \frac{\partial h}{\partial y} = 4 y$ Feeling with a boot: stepping east, the ground rises about 2x for each metre; stepping north, about 4y.
$\nabla h (3, 1) = (6, 4)$ So right here the steepest way up points 6 east and 4 north.
$- \nabla h (3, 1) = (- 6, - 4)$ The steepest way down is exactly the opposite arrow.
$(x, y) \leftarrow (3, 1) - 0.1 (6, 4) = (2.4, 0.6)$ The hiker takes a small step, one tenth of that arrow, and stands at 2.4 east and 0.6 north.
$h (2.4, 0.6) = 6.48 < 11$ The height has dropped from 11 to 6.48 metres.
$\nabla h (0, 0) = (0, 0)$ Step after step, the arrow shrinks, until at the lowest point there is no uphill left at all.

In words this story is 712 letters long; in symbols, 125.

Read it

What does it say?

$\nabla f (1, 2) = (0, 0)$

$\frac{\partial f}{\partial y} > 0$

$∥ \nabla f (p)∥ = 5$

$D_{u} f = 0$

$\frac{\partial T}{\partial t} = - 2$

$\nabla f (p) = (0, - 3)$

$∥ \nabla f ∥ = 0$

$D_{u} f = - ∥ \nabla f ∥$

$h (x, y) = 100$

$θ \leftarrow θ - 0.01 \nabla L (θ)$

$\nabla f = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z})$

$\frac{\partial f}{\partial x} (2, 5) = 0$

Write it

Which notation says this?

The gradient of f is the vector of its two partial derivatives.

Update θ by a step against the gradient of the loss, of size η times the gradient.

Along a contour line, the height f stays the same number c.

The partial derivative of f with respect to y.

The steepest slope at the point p is 3.

The slope of f in the direction of the unit arrow u, written as a dot product.

The contour of the hill h at height 4.

The gradient of x squared plus y squared is the arrow (2x, 2y).

One step of gradient descent with learning rate 0.1.

The gradient is perpendicular to the contour line.

Walking in direction u, f climbs 2 units of height per unit of distance.

At the lowest point of the valley the gradient is the zero vector.

Build it

Tap the tiles in order. One tile does not belong.

The slope of f in direction u is the gradient of f dotted with u.

The next θ is this θ minus η times the gradient of the loss at this θ.

The gradient of f is the arrow (2x, 2y).

The norm of the gradient of f at p is 5.

A contour line: h of x, y equals c.

Partial f by partial x equals 2 x.

Challenges

Creative puzzles that use every sign of this chapter. Take your time; there is no clock.

1 The foggy hill, mapped

Here f(x, y) = x² + y². The grid shows which contour each point lies on. Judge each sentence: true or false?

	$c = 1$	$c = 4$	$c = 8$
$(1, 0)$	✓
$(0, 2)$		✓
$(2, 2)$			✓
$(1, 1)$

A tick means the point of the row lies on the contour f(x, y) = c of the column.

$\nabla f (1, 0) = (2, 0)$
$∥ \nabla f (2, 2)∥ = 42$
$u = (0, 1) : D_{u} f (1, 0) = 0$
$\frac{\partial f}{\partial y} (0, 2) = 2$
$\nabla f (1, 1) \cdot (1, - 1) = 2$
$\nabla f (x, y) = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y})$

2 The slope dictionary

Six lines for ideas about slopes in two or more variables. Pair each with its meaning.

Tap a line, then its meaning.

3 Which way is down?

This should be one step of gradient descent, going downhill. One tile is wrong. Find it, then fix it.

Tap the tile that is wrong.

4 Slope in any direction

This should give the slope of f along the unit arrow u. One tile is wrong. Find it, then fix it.

Tap the tile that is wrong.

5 Collect the slopes

Which arrow is the gradient at the point (1, 1)?

$f (x, y) = x^{2} + 3 y^{2}, \nabla f (1, 1) = ?$

6 Slope in a chosen direction

With the gradient of xy, find the slope in the direction of the unit arrow u at (2, 1).

$f (x, y) = x y, u = (0.6, 0.8), D_{u} f (2, 1) = ?$

7 Walking along the contour

Along a contour line you neither climb nor descend. Write in symbols: if the gradient is perpendicular to the direction u, then the slope in that direction is zero. Then compare.

Your version

8 Compute and collect

For f(x, y) = x² y, write the gradient as the arrow of its two partial derivatives, and give its value. Then compare.

Your version

9 The saddle

A mountain pass has height f(x, y) = x² − y²: up along x, down along y. Judge each sentence: true or false?

	$\frac{\partial f}{\partial x} > 0$	$\frac{\partial f}{\partial y} < 0$	$\nabla f = 0$
$(1, 0)$	✓
$(0, 1)$		✓
$(0, 0)$			✓
$(1, 1)$	✓	✓

A tick means the column’s claim holds at the row’s point, for f = x² − y², whose gradient is (2x, −2y).

$\nabla f (1, 1) = (2, - 2)$
$\frac{\partial f}{\partial y} (0, 1) = 2$
$u = (0, 1) : D_{u} f (1, 0) = 0$
$\nabla f (0, 0) = (0, 0)$
$f (1, 0) = f (0, 1)$
$∥ \nabla f (1, 1)∥ = 22$

10 Climb, coast or drop

Four things the slope in direction u can be. Pair each with what it means for the walker.

Tap a line, then its meaning.

11 Hold y still

Take both partial derivatives of f = 3xy + y². One tile is wrong. Find it, then fix it.

Tap the tile that is wrong.

12 A step the wrong way

One step of gradient descent with θ = 4, gradient 6 and learning rate 0.5. One tile is wrong. Find it, then fix it.

Tap the tile that is wrong.

13 The steady slope

A plane roof has height 3x + 4y. How steep is its steepest direction?

$f (x, y) = 3 x + 4 y, ∥ \nabla f ∥ = ?$

14 Walking sideways

At p the gradient points along y. What is the slope if you walk along x?

$\nabla f (p) = (0, 2), u = (1, 0), D_{u} f (p) = ?$

15 One step down

The loss is θ², you stand at θ = 3 and the learning rate is 0.1. Where does one step of gradient descent take you?

$L (θ) = θ^{2}, θ = 3, η = 0.1, θ \leftarrow ?$

16 The steepest way

Write in symbols: among all unit directions, the slope is largest along the gradient, and that largest slope is the norm of the gradient. Then compare.

Your version

Tell your own

Write a two-line story of something going downhill — a ball, a price, a mistake shrinking — in notation only: a height, a point, a gradient, a step.

Kept on this device only.

The gradient: which way is uphill

The alphabet

The story: The hiker in the fog

Read it

Write it

Build it

Challenges

1 The foggy hill, mapped

2 The slope dictionary

3 Which way is down?

4 Slope in any direction

5 Collect the slopes

6 Slope in a chosen direction

7 Walking along the contour

8 Compute and collect

9 The saddle

10 Climb, coast or drop

11 Hold y still

12 A step the wrong way

13 The steady slope

14 Walking sideways

15 One step down

16 The steepest way

Tell your own

Download for offline use