One neuron | From a Dot to a Chatbot

What the neuron believes, everywhere

label 0 label 1dashed: where the neuron is exactly 50/50

Settings

Before training After 20 steps After 600 steps

98%points classified right

0.075loss (cross-entropy)

p = σ( 7.22·x + 8.19·y + −0.51 )

x and y are first rescaled from 0..10 to −1..1. σ(z) = 1 / (1 + e^−z) turns any number into a probability.

Loss during training

Inside the neuron

Blue wires push towards "label 1", red wires push towards "label 0". Thicker means a bigger weight.

What to notice

Training moves the line. Press "Before training": random weights put the line somewhere silly. After a few steps it swings into the gap between the two blobs.

XOR is impossible. Switch the dataset to XOR. Red sits in two opposite corners and blue in the other two. No single straight line can separate that, so accuracy stays near 50% however long you train. Moons and rings fail the same way.

This exact problem stalled AI research for years in the 1970s. The fix, stacking neurons in layers, is the next lesson.

The code that just ran

Everything on this page came from calling this code, which lives at the top of this same file.

Show or hide the source

final class Neuron
{
    public float $w1 = 0.0, $w2 = 0.0, $b = 0.0;

    public function __construct(int $seed)
    {
        seed_rng($seed);                 // small random starting weights
        $this->w1 = gauss() * 0.5;
        $this->w2 = gauss() * 0.5;
        $this->b  = gauss() * 0.1;
    }

    // Inputs go from 0..10 on the plot; the neuron sees them as −1..1.
    // (Lesson 1: keep numbers on a similar, small scale.)
    public static function scale(float $v): float { return ($v - 5) / 5; }

    // FORWARD: weighted sum, then squash.
    public function forward(float $x, float $y): float
    {
        $z = $this->w1 * self::scale($x) + $this->w2 * self::scale($y) + $this->b;
        return sigmoid($z);
    }

    // CROSS-ENTROPY LOSS: −log(probability given to the right answer).
    // Saying 0.99 when right costs 0.01. Saying 0.01 when right costs 4.6.
    // Confident mistakes are punished hard.
    public function loss(array $points): float
    {
        $sum = 0.0;
        foreach ($points as $p) {
            $q = $this->forward($p['x'], $p['y']);
            $sum -= $p['label'] ? log(max($q, 1e-12)) : log(max(1 - $q, 1e-12));
        }
        return $sum / count($points);
    }

    public function accuracy(array $points): float
    {
        $ok = 0;
        foreach ($points as $p) $ok += (int) (($this->forward($p['x'], $p['y']) >= 0.5) === (bool) $p['label']);
        return $ok / count($points);
    }

    // ONE STEP OF GRADIENT DESCENT over all points.
    // For sigmoid + cross-entropy the slope comes out beautifully simple:
    //   ∂loss/∂z = p − label     ("how wrong, and in which direction")
    // and then by the chain rule  ∂loss/∂w₁ = (p − label) · x,  ∂loss/∂b = (p − label)
    public function trainStep(array $points, float $lr): void
    {
        $g1 = $g2 = $gb = 0.0;
        foreach ($points as $p) {
            $err = $this->forward($p['x'], $p['y']) - $p['label'];
            $g1 += $err * self::scale($p['x']);
            $g2 += $err * self::scale($p['y']);
            $gb += $err;
        }
        $n = count($points);
        $this->w1 -= $lr * $g1 / $n;
        $this->w2 -= $lr * $g2 / $n;
        $this->b  -= $lr * $gb / $n;
    }

    public function train(array $points, float $lr, int $epochs): array
    {
        $history = [$this->loss($points)];
        for ($e = 0; $e < $epochs; $e++) {
            $this->trainStep($points, $lr);
            $history[] = $this->loss($points);
        }
        return $history;
    }
}