Gradient descent | From a Dot to a Chatbot

The loss landscape, seen from above

● start the path, one dot per step★ the true bottomup: start b

Settings

Too slow Zig-zag Explode

758,146,277,049.0loss after 100 steps

30.9lowest possible

−145,491.06, −22,582.0m, b reached

The best line is m = 5.22, b = 39.8.

Loss after every step

Plotted on a log scale, so each gridline is ten times the one below.

The first steps, in numbers

new m = m − 0.03 × (∂loss/∂m) new b = b − 0.03 × (∂loss/∂b)

Step	m	b	Loss	∂loss/∂m	∂loss/∂b
0	−3.000	10.000	5,812.5	−876.0	−146.4
1	23.280	14.392	6,970.0	965.2	139.9
2	−5.676	10.196	8,366.5	−1,056.7	−174.2
3	26.024	15.423	10,051.2	1,163.5	170.9
4	−8.881	10.296	12,083.2	−1,274.5	−207.9
5	29.354	16.533	14,534.2	1,402.6	208.3
6	−12.725	10.284	17,490.2	−1,537.2	−248.5
…
99	132,505.939	20,622.540	628,734,504,439.7	9,266,566.5	1,440,152.2
100	−145,491.056	−22,582.027	758,146,277,049.0	−10,175,638.2	−1,581,441.8

A negative slope means "going up in this direction makes the loss smaller", so the step goes up.

What to notice

The path bends. It first rushes sideways to fix the slope m, then crawls up towards the right b. The valley is long and narrow: changing m changes the loss a lot (hours go up to 10), changing b only a little. Gradient descent struggles in narrow valleys.

The learning rate is a gamble. Too small and you need thousands of steps. Too big and you bounce between the valley walls, or fly off completely. Try the three buttons.

This is how LLMs learn. Same loop, but the "landscape" has billions of directions, one per weight. Real training uses smarter versions (momentum and Adam) that remember recent steps to get through narrow valleys faster.

The code that just ran

Everything on this page came from calling this code, which lives at the top of this same file.

Show or hide the source

final class GradientDescent
{
    public function __construct(private array $points) {}

    // Loss (MSE) at knob settings (m, b), and how steeply it rises if you
    // nudge m or b. Those two slopes together are the gradient.
    //   loss = average of (m·x + b − y)²
    //   ∂loss/∂m = average of 2·(m·x + b − y)·x
    //   ∂loss/∂b = average of 2·(m·x + b − y)
    public function lossAndGradient(float $m, float $b): array
    {
        $n = count($this->points);
        $loss = $dm = $db = 0.0;
        foreach ($this->points as $p) {
            $err = ($m * $p['x'] + $b) - $p['y'];
            $loss += $err * $err;
            $dm += 2 * $err * $p['x'];
            $db += 2 * $err;
        }
        return [$loss / $n, $dm / $n, $db / $n];
    }

    // THE TRAINING LOOP: feel the slope, step the other way, repeat.
    public function run(float $m, float $b, float $learningRate, int $steps): array
    {
        $history = [];
        for ($t = 0; $t <= $steps; $t++) {
            [$loss, $dm, $db] = $this->lossAndGradient($m, $b);
            $history[] = ['t' => $t, 'm' => $m, 'b' => $b, 'loss' => $loss, 'dm' => $dm, 'db' => $db];
            if (!is_finite($loss) || $loss > 1e12) break;     // it blew up
            $m -= $learningRate * $dm;                        // downhill in m
            $b -= $learningRate * $db;                        // downhill in b
        }
        return $history;
    }
}