What the network believes, everywhere
Settings
Training took 321 ms in PHP. GPT-3 has 175,000,000,000 parameters; this has 25.
Loss during training
What each hidden neuron learned
Each small square is one hidden neuron's output over the whole plane (blue = high, red = low). The number is its weight into the output neuron: positive weights vote "label 1", negative vote "label 0".
What to notice
Each hidden neuron is just lesson 5 again. Its picture is split by one straight line. The curve in the big picture comes from adding those splits together with different weights, then squashing once more.
More neurons, more bends. XOR needs at least 2 hidden neurons (try 1: it fails). Moons and rings need more. Set hidden neurons to 16 on "rings" and it draws a closed loop.
Different seeds, different solutions. Change the seed: the network may find a different set of lines that works just as well. There is no single "right" set of weights.
From here to an LLM
An LLM is built from the same parts: layers of weighted sums with an activation in between, trained by gradient descent. The differences are size and shape. Instead of 2 inputs it takes thousands of numbers per word, instead of 1 hidden layer it has dozens of blocks (lesson 13), and instead of one yes/no output it outputs a probability for every possible next word (lesson 11).
The line in the code marked BACKPROPAGATION is the key idea. The next lesson slows it down to a single example, so you can check every number by hand.
The code that just ran
Everything on this page came from calling this code, which lives at the top of this same file.
Show or hide the source
final class MLP
{
public array $W1 = []; // W1[k] = [weight from x, weight from y] for hidden neuron k
public array $b1 = []; // bias of hidden neuron k
public array $W2 = []; // weight from hidden neuron k to the output
public float $b2 = 0.0; // output bias
private array $v = []; // momentum: a running average of recent steps
public function __construct(private int $hidden, private string $activation, int $seed)
{
seed_rng($seed);
for ($k = 0; $k < $hidden; $k++) {
$this->W1[$k] = [gauss() * 1.5, gauss() * 1.5];
$this->b1[$k] = gauss() * 0.5;
$this->W2[$k] = gauss() / sqrt($hidden);
}
$this->v = ['W1' => array_fill(0, $hidden, [0.0, 0.0]), 'b1' => array_fill(0, $hidden, 0.0),
'W2' => array_fill(0, $hidden, 0.0), 'b2' => 0.0];
}
private static function scale(float $v): float { return ($v - 5) / 5; }
// The activation: the bend. tanh squashes to −1..1, ReLU is max(0, z).
private function act(float $z): float { return $this->activation === 'relu' ? max(0.0, $z) : tanh($z); }
private function actSlope(float $z, float $a): float
{
return $this->activation === 'relu' ? ($z > 0 ? 1.0 : 0.0) : 1 - $a * $a;
}
// FORWARD PASS. Returns everything, because the backward pass needs it.
public function run(float $x, float $y): array
{
$sx = self::scale($x); $sy = self::scale($y);
$z = []; $a = []; $out = $this->b2;
for ($k = 0; $k < $this->hidden; $k++) {
$z[$k] = $this->W1[$k][0] * $sx + $this->W1[$k][1] * $sy + $this->b1[$k];
$a[$k] = $this->act($z[$k]);
$out += $this->W2[$k] * $a[$k];
}
return ['sx' => $sx, 'sy' => $sy, 'z' => $z, 'a' => $a, 'p' => sigmoid($out)];
}
public function predict(float $x, float $y): float { return $this->run($x, $y)['p']; }
public function hiddenOut(int $k, float $x, float $y): float { return $this->run($x, $y)['a'][$k]; }
// ONE TRAINING STEP: forward, measure the error, send it backwards, nudge.
public function trainStep(array $points, float $lr, float $momentum = 0.9): float
{
$H = $this->hidden;
$gW1 = array_fill(0, $H, [0.0, 0.0]); $gb1 = array_fill(0, $H, 0.0);
$gW2 = array_fill(0, $H, 0.0); $gb2 = 0.0; $loss = 0.0;
foreach ($points as $pt) {
$r = $this->run($pt['x'], $pt['y']);
$loss -= $pt['label'] ? log(max($r['p'], 1e-12)) : log(max(1 - $r['p'], 1e-12));
$dOut = $r['p'] - $pt['label']; // output neuron's error (lesson 5)
$gb2 += $dOut;
for ($k = 0; $k < $H; $k++) {
$gW2[$k] += $dOut * $r['a'][$k];
// BACKPROPAGATION: hidden neuron k's share of the blame =
// output error × the weight connecting them × its own slope
$dHidden = $dOut * $this->W2[$k] * $this->actSlope($r['z'][$k], $r['a'][$k]);
$gW1[$k][0] += $dHidden * $r['sx'];
$gW1[$k][1] += $dHidden * $r['sy'];
$gb1[$k] += $dHidden;
}
}
// Momentum: keep rolling in the direction we've been going.
$n = count($points);
for ($k = 0; $k < $H; $k++) {
for ($i = 0; $i < 2; $i++) {
$this->v['W1'][$k][$i] = $momentum * $this->v['W1'][$k][$i] - $lr * $gW1[$k][$i] / $n;
$this->W1[$k][$i] += $this->v['W1'][$k][$i];
}
$this->v['b1'][$k] = $momentum * $this->v['b1'][$k] - $lr * $gb1[$k] / $n;
$this->b1[$k] += $this->v['b1'][$k];
$this->v['W2'][$k] = $momentum * $this->v['W2'][$k] - $lr * $gW2[$k] / $n;
$this->W2[$k] += $this->v['W2'][$k];
}
$this->v['b2'] = $momentum * $this->v['b2'] - $lr * $gb2 / $n;
$this->b2 += $this->v['b2'];
return $loss / $n;
}
public function accuracy(array $points): float
{
$ok = 0;
foreach ($points as $p) $ok += (int) (($this->predict($p['x'], $p['y']) >= 0.5) === (bool) $p['label']);
return $ok / count($points);
}
public function parameterCount(): int { return $this->hidden * 4 + 1; }
}