Gradient descent | From a Dot to a Chatbot

The loss landscape, seen from above

● start the path, one dot per step★ the true bottomup: start b

Settings

Too slow Zig-zag Explode

163.8loss after 100 steps

30.9lowest possible

9.39, 12.9m, b reached

The best line is m = 5.22, b = 39.8.

Loss after every step

Plotted on a log scale, so each gridline is ten times the one below.

The first steps, in numbers

new m = m − 0.001 × (∂loss/∂m) new b = b − 0.001 × (∂loss/∂b)

Step	m	b	Loss	∂loss/∂m	∂loss/∂b
0	−3.000	10.000	5,812.5	−876.0	−146.4
1	−2.124	10.146	5,051.2	−814.6	−136.8
2	−1.309	10.283	4,392.8	−757.5	−128.0
3	−0.552	10.411	3,823.1	−704.5	−119.7
4	0.153	10.531	3,330.4	−655.1	−112.0
5	0.808	10.643	2,904.1	−609.2	−104.9
6	1.417	10.748	2,535.4	−566.5	−98.3
…
99	9.387	12.922	163.9	0.8	−9.8
100	9.386	12.932	163.8	0.9	−9.8

A negative slope means "going up in this direction makes the loss smaller", so the step goes up.

What to notice

The path bends. It first rushes sideways to fix the slope m, then crawls up towards the right b. The valley is long and narrow: changing m changes the loss a lot (hours go up to 10), changing b only a little. Gradient descent struggles in narrow valleys.

The learning rate is a gamble. Too small and you need thousands of steps. Too big and you bounce between the valley walls, or fly off completely. Try the three buttons.

This is how LLMs learn. Same loop, but the "landscape" has billions of directions, one per weight. Real training uses smarter versions (momentum and Adam) that remember recent steps to get through narrow valleys faster.

The code that just ran

Everything on this page came from calling this code, which lives at the top of this same file.

Show or hide the source

final class GradientDescent
{
    public function __construct(private array $points) {}

    // Loss (MSE) at knob settings (m, b), and how steeply it rises if you
    // nudge m or b. Those two slopes together are the gradient.
    //   loss = average of (m·x + b − y)²
    //   ∂loss/∂m = average of 2·(m·x + b − y)·x
    //   ∂loss/∂b = average of 2·(m·x + b − y)
    public function lossAndGradient(float $m, float $b): array
    {
        $n = count($this->points);
        $loss = $dm = $db = 0.0;
        foreach ($this->points as $p) {
            $err = ($m * $p['x'] + $b) - $p['y'];
            $loss += $err * $err;
            $dm += 2 * $err * $p['x'];
            $db += 2 * $err;
        }
        return [$loss / $n, $dm / $n, $db / $n];
    }

    // THE TRAINING LOOP: feel the slope, step the other way, repeat.
    public function run(float $m, float $b, float $learningRate, int $steps): array
    {
        $history = [];
        for ($t = 0; $t <= $steps; $t++) {
            [$loss, $dm, $db] = $this->lossAndGradient($m, $b);
            $history[] = ['t' => $t, 'm' => $m, 'b' => $b, 'loss' => $loss, 'dm' => $dm, 'db' => $db];
            if (!is_finite($loss) || $loss > 1e12) break;     // it blew up
            $m -= $learningRate * $dm;                        // downhill in m
            $b -= $learningRate * $db;                        // downhill in b
        }
        return $history;
    }
}