Module 18 · Lesson 18.2
Overfitting and generalisation
Why training error falls forever, why test error turns round, and why the gap is invisible from the training data alone.
Why this matters
Overfitting is usually explained with a picture of a wiggly line through some dots. That gets the shape right and misses the consequence, which is: you cannot detect it from the data you trained on. The training error is falling the whole time. Everything looks like progress.
The only way to see it is to hold data back, and that is not a technicality — it is the entire argument for a test set.
By the end of this lesson you should be able to
- Explain why training error falls with every added parameter
- Read the point where test error turns round
- Explain the role of a held-back test set
- Recognise the engineering analogue in curve-fitting practice
The two curves
Fit a polynomial of increasing degree to twelve noisy points and track two things.
Training error — how well the fit reproduces the points it was given. This falls with every added parameter, and it must: a curve with more freedom can always match the data at least as well. With twelve points, a degree-eleven polynomial passes through every one exactly and the training error is zero.
Test error — how well the fit predicts points it never saw. This falls at first, as the model captures the real relationship. Then it turns round, as the extra freedom is spent reproducing the noise in the training points rather than the signal underneath.
The gap between the two curves is the model learning the noise. And nothing in the training data reveals it, because the training error is still falling.
What the numbers do
On the library's study — a straight-line truth with noise, twelve training points:
| Degree | Training error | Test error |
|---|---|---|
| 0 | 1.705 | 2.464 |
| 1 | 0.999 | 1.456 |
| 3 | 0.699 | 1.431 ← best |
| 5 | 0.444 | 1.515 |
| 8 | 0.147 | 1.525 |
| 10 | 0.089 | 1.657 |
Training error falls by a factor of nineteen from degree 0 to degree 10. Test error reaches its minimum at degree 3 and is worse at degree 10 than it was at degree 1.
A model chosen on training error alone would pick degree 10, and it would be the worst usable model in the table.
Why the underlying truth was a straight line
Worth noticing: the data came from a straight line plus noise, and the best model was degree 3, not degree 1. That is not a mistake in the study — it is what a finite noisy sample does. With twelve points, a little extra flexibility happens to fit this particular noise slightly better than the true relationship does.
Which is the honest version of the lesson. The best model on your data is not necessarily the true one; it is the one that best balances signal against noise for the amount of data you have. More data moves that balance towards the truth.
The engineering analogue
This is not new to structural engineering, and it is worth naming the version engineers already know.
Fit a curve through six test results and read a value between them: reasonable. Fit a fifth-order polynomial through the same six points so it passes exactly through each, and read a value between them: the curve will swing wildly between the points, and the reading will be worse than a straight line would have given.
Every engineer who has fitted a trend line has met this. The machine-learning version has more parameters and less supervision, and it is the same phenomenon.
Try it
Overfitting explorer
Fit a polynomial of increasing degree to noisy data drawn from a straight line. Two error curves, and only one of them turns round.
More data moves the best degree back towards the true relationship.
- Degree selected
- 3
- Training error
- 0.6990
- Test error
- 1.4310
- Test / training ratio
- 2.0
- Best degree by test error
- 3
- Points needed to fit exactly
- 12
4 parameters
a healthy model is near 1.5; nineteen is overfitting
degree 11 has one parameter per point
This is the best degree by test error. Note that the data came from a STRAIGHT LINE — the best model on a finite noisy sample is not necessarily the true one.
What to try
- Push the degree to 10 and watch the training error collapse while the test error rises. A model chosen on training error alone would pick this.
- Set the noise to zero. The two curves converge, because there is no noise for the extra parameters to learn.
- Increase the training points. The best degree moves back towards 1 — more data shifts the balance towards the true relationship.
- The polynomial fit itself is a least-squares solve, and at high degree the normal-equations matrix becomes ill-conditioned — the same arithmetic problem as Module 12's stiffness range.
What this shows: Training error falls with every added parameter; test error turns round — and nothing in the training data reveals the gap.
Worked example
Choosing a model on the wrong curve
Given
- The table above: training and test error against polynomial degree
- An engineer with access to the training data only
Find
What degree they would choose, and what it costs
Practice
A model achieves a training error of 0.089 and a test error of 1.657. What is the ratio of test error to training error?
Practice
Twelve training points are fitted with a polynomial. At what degree does the polynomial have exactly enough parameters to pass through every point?
Worked example
How many parameters is too many
Given
- A model is fitted to 40 measured floor deflections
- The candidate is a polynomial in span, depth and load, with all cross terms up to second order
Find
How many parameters that is, and whether 40 data points can support it
Practice
A full second-order polynomial in three variables has how many parameters, counting the constant?
Practice
40 data points are used to fit a model with 10 parameters. How many observations per parameter is that?
Practice
A model's training error is 0.20 and its test error is 1.60. What is the ratio, and roughly what does a healthy model give?
Check yourself
Why does adding parameters always reduce the training error?
Check yourself
An engineer tries fifteen model configurations and reports the test error of the best one. What is wrong with that figure?
Check yourself
What does cross-validation buy that a single train/test split does not?
Summary
- Training error falls with every added parameter, necessarily
- Test error falls then rises; the gap is the model learning the noise
- Overfitting is undetectable from the training data alone
- A test/train ratio of 19 is a warning; a ratio near 1.5 is healthy
- Choosing a configuration on the test set consumes it — use a third split
This is educational material. It uses simplified examples to teach principles, and must not be relied on for real design or safety-critical decisions. Module overview and checkpoint