Gradient descent | From a Dot to a Chatbot

The loss landscape, seen from above

● start the path, one dot per step★ the true bottomup: start b

Settings

Too slow Zig-zag Explode

It exploded after 17 steps. Each step overshot the valley and landed higher up the other side, so the next slope was steeper, so the next step was bigger. Lower the learning rate.

∞loss after 17 steps

30.9lowest possible

?m, b reached

The best line is m = 5.22, b = 39.8.

Loss after every step

Plotted on a log scale, so each gridline is ten times the one below.

The first steps, in numbers

new m = m − 0.04 × (∂loss/∂m) new b = b − 0.04 × (∂loss/∂b)

Step	m	b	Loss	∂loss/∂m	∂loss/∂b
0	−3.000	10.000	5,812.5	−876.0	−146.4
1	32.040	15.856	18,387.9	1,578.9	235.3
2	−31.117	6.444	59,026.8	−2,833.8	−450.4
3	82.235	24.459	190,336.1	5,097.9	782.5
4	−121.679	−6.840	614,591.6	−9,159.1	−1,433.1
5	244.685	50.485	1,985,326.4	16,467.3	2,549.7
6	−414.007	−51.503	6,414,039.1	−29,595.4	−4,608.9
…
16	−148,981.217	−23,137.428	794,975,968,936.9	−10,419,865.7	−1,619,402.9
17	267,813.413	41,638.688	2,568,486,352,268.1	18,729,398.7	2,910,805.9

A negative slope means "going up in this direction makes the loss smaller", so the step goes up.

What to notice

The path bends. It first rushes sideways to fix the slope m, then crawls up towards the right b. The valley is long and narrow: changing m changes the loss a lot (hours go up to 10), changing b only a little. Gradient descent struggles in narrow valleys.

The learning rate is a gamble. Too small and you need thousands of steps. Too big and you bounce between the valley walls, or fly off completely. Try the three buttons.

This is how LLMs learn. Same loop, but the "landscape" has billions of directions, one per weight. Real training uses smarter versions (momentum and Adam) that remember recent steps to get through narrow valleys faster.

The code that just ran

Everything on this page came from calling this code, which lives at the top of this same file.

Show or hide the source

final class GradientDescent
{
    public function __construct(private array $points) {}

    // Loss (MSE) at knob settings (m, b), and how steeply it rises if you
    // nudge m or b. Those two slopes together are the gradient.
    //   loss = average of (m·x + b − y)²
    //   ∂loss/∂m = average of 2·(m·x + b − y)·x
    //   ∂loss/∂b = average of 2·(m·x + b − y)
    public function lossAndGradient(float $m, float $b): array
    {
        $n = count($this->points);
        $loss = $dm = $db = 0.0;
        foreach ($this->points as $p) {
            $err = ($m * $p['x'] + $b) - $p['y'];
            $loss += $err * $err;
            $dm += 2 * $err * $p['x'];
            $db += 2 * $err;
        }
        return [$loss / $n, $dm / $n, $db / $n];
    }

    // THE TRAINING LOOP: feel the slope, step the other way, repeat.
    public function run(float $m, float $b, float $learningRate, int $steps): array
    {
        $history = [];
        for ($t = 0; $t <= $steps; $t++) {
            [$loss, $dm, $db] = $this->lossAndGradient($m, $b);
            $history[] = ['t' => $t, 'm' => $m, 'b' => $b, 'loss' => $loss, 'dm' => $dm, 'db' => $db];
            if (!is_finite($loss) || $loss > 1e12) break;     // it blew up
            $m -= $learningRate * $dm;                        // downhill in m
            $b -= $learningRate * $db;                        // downhill in b
        }
        return $history;
    }
}