A degree-9 curve fitted to 15 points
Settings
Overfitting. Nearly perfect on data it has seen, much worse on data it hasn't. It learned the noise.
Error for every degree
What to notice
Training error only ever goes down. More knobs can always fit the seen points better. That's why it's useless for choosing a model.
Validation error is a U. Too simple, it misses the wave. Too complex, it chases the noise. The bottom of the U is the model you want.
Three fixes, all used for LLMs. More data: set 100 points and degree 15 behaves. Regularisation: set λ to medium and the wild swings calm down. Stopping early: in neural networks you watch validation loss during training and stop when it turns up.
The weights of the current model
When a model overfits, its weights usually become huge and cancel each other out. Regularisation keeps them small.
| x^0 | x^1 | x^2 | x^3 | x^4 | x^5 | x^6 | x^7 | x^8 | x^9 | |
|---|---|---|---|---|---|---|---|---|---|---|
| w | −0.2 | −3.3 | 10.8 | −16.1 | −42.7 | 105.6 | 55.0 | −165.9 | −22.6 | 80.8 |
Biggest weight: 165.9. LLMs are so large they could memorise a lot of their training text, and sometimes do. Keeping an honest held-out set is how labs measure whether they're really learning.
The code that just ran
Everything on this page came from calling this code, which lives at the top of this same file.
Show or hide the source
final class PolyRegression
{
private array $w = [];
public function __construct(private int $degree, private float $lambda = 0.0) {}
// x goes from 0..1; we use u = 2x − 1 (−1..1) so high powers don't explode.
private function features(float $x): array
{
$u = 2 * $x - 1; $f = []; $p = 1.0;
for ($k = 0; $k <= $this->degree; $k++) { $f[] = $p; $p *= $u; }
return $f;
}
// Least squares again (lesson 3), now with d+1 knobs, plus REGULARISATION:
// an extra cost of λ·(sum of squared weights). It says "prefer small weights",
// which means "prefer smooth curves unless the data really insists".
// Solves (XᵀX + λ·n·I) w = Xᵀy
public function fit(array $xs, array $ys): void
{
$d = $this->degree + 1; $n = count($xs);
$A = array_fill(0, $d, array_fill(0, $d, 0.0));
$b = array_fill(0, $d, 0.0);
foreach ($xs as $i => $x) {
$f = $this->features($x);
for ($r = 0; $r < $d; $r++) {
$b[$r] += $f[$r] * $ys[$i];
for ($c = 0; $c < $d; $c++) $A[$r][$c] += $f[$r] * $f[$c];
}
}
for ($r = 0; $r < $d; $r++) $A[$r][$r] += ($r > 0 ? $this->lambda * $n : 0) + 1e-10;
$this->w = self::solve($A, $b);
}
public function predict(float $x): float
{
$sum = 0.0;
foreach ($this->features($x) as $k => $f) $sum += $this->w[$k] * $f;
return $sum;
}
// Root-mean-square error: the typical size of a miss.
public function rmse(array $xs, array $ys): float
{
$s = 0.0;
foreach ($xs as $i => $x) $s += ($this->predict($x) - $ys[$i]) ** 2;
return sqrt($s / count($xs));
}
public function weights(): array { return $this->w; }
// Gaussian elimination: the school method for solving equations.
private static function solve(array $A, array $b): array
{
$n = count($b);
for ($col = 0; $col < $n; $col++) {
$piv = $col;
for ($r = $col + 1; $r < $n; $r++) if (abs($A[$r][$col]) > abs($A[$piv][$col])) $piv = $r;
[$A[$col], $A[$piv]] = [$A[$piv], $A[$col]];
[$b[$col], $b[$piv]] = [$b[$piv], $b[$col]];
if (abs($A[$col][$col]) < 1e-14) continue;
for ($r = $col + 1; $r < $n; $r++) {
$k = $A[$r][$col] / $A[$col][$col];
for ($c = $col; $c < $n; $c++) $A[$r][$c] -= $k * $A[$col][$c];
$b[$r] -= $k * $b[$col];
}
}
$x = array_fill(0, $n, 0.0);
for ($r = $n - 1; $r >= 0; $r--) {
$s = $b[$r];
for ($c = $r + 1; $c < $n; $c++) $s -= $A[$r][$c] * $x[$c];
$x[$r] = abs($A[$r][$r]) < 1e-14 ? 0.0 : $s / $A[$r][$r];
}
return $x;
}
}