Module 18 · Lesson 18.3
Bias, and what the model never saw
Class imbalance, distribution shift, and the two numbers that behave completely differently.
Why this matters
Every fitted model knows exactly one thing: the data it was shown. Two consequences follow, and both are routinely missed because the headline metric hides them.
A class that barely appeared in training will barely be recognised — while overall accuracy stays respectable. And a case unlike anything in training will get an answer anyway, delivered with the same confidence as every other.
By the end of this lesson you should be able to
- Distinguish accuracy from per-class recall and say which hides what
- Read a confusion matrix
- Explain distribution shift and why a model cannot flag it
- Name the classes of bias that matter in engineering data
Accuracy hides; recall shows
Accuracy is the fraction of all cases classified correctly. Recall for a class is the fraction of that class's cases that were found.
They come apart the moment the classes are unbalanced, and the demonstration is stark.
Take a classifier over four categories of model warning — restraint, units, mesh and connectivity — tested on sixty cases of each.
Trained on 80 examples of each class: - Overall accuracy 90.4 % - Recall on connectivity 95.0 %
Trained on 80, 80, 80 and just 3 connectivity examples: - Overall accuracy 75.0 % - Recall on connectivity 26.7 %
Accuracy fell by fifteen points, which anyone might accept as the cost of imperfect data. Recall on the starved class fell from 95 % to 27 %: it now misses nearly three connectivity faults in four.
If the reason you built the classifier was to catch connectivity faults, it is no longer fit for purpose — and the accuracy figure would not have told you.
Report per-class recall. Overall accuracy on unbalanced data is a summary of how common the common classes are.
The confusion matrix
Rows are what the case actually was; columns are what the model said. The diagonal is correct; everything off it is a specific, nameable error.
It is worth reading rather than summarising because the off-diagonal pattern is informative. A starved class does not get misclassified randomly — its cases pile up in whichever class it most resembles, and that tells you what the model has actually learned.
Distribution shift
The model saw a sample. The world it is used in has moved. Three ways that happens in structural work:
Novel structural form. A model trained on rectilinear steel frames, asked about a diagrid.
Changed practice. A model trained on pre-2020 detailing, asked about current detailing.
Changed materials. A model trained on Portland-cement concrete, asked about a high-replacement mix.
In every case the model answers. It has no notion of 'outside my experience', because a fitted function is defined everywhere its inputs are numbers.
Bias in engineering data
Five kinds that arise in real project data:
Selection bias. The projects in the dataset are the ones that were digitised, which are the recent, large, well-documented ones.
Survivor bias. Structures that failed are absent, and so are designs that were rejected at scheme stage. A model trained on what was built learns what gets built, not what works.
Practice bias. Data from one office encodes that office's habits, which will be reproduced confidently as though they were structural truths.
Measurement bias. Labels applied by different engineers, to different standards, at different times.
Temporal bias. Design practice, codes and material specifications all move. Old data describes an old world.
All five are properties of how the data came to exist, and none is visible in the data itself. Finding them requires knowing where the data came from, which is why data provenance is an engineering question rather than an administrative one.
Try it
Training data bias explorer
Starve one class in the training set, then test on a balanced set. Watch what happens to accuracy, and to that class's recall.
Connectivity faults are rare in an archive — that is why this happens.
| Class | Trained on | Recall | Precision |
|---|---|---|---|
| Missing or excessive restraint | 80 | 96.7 % | 56.9 % |
| Unit or scale error | 80 | 85.0 % | 87.9 % |
| Mesh density or quality | 80 | 91.7 % | 85.9 % |
| Disconnected or duplicated geometry | 3 | 26.7 % | 100.0 % |
- Overall accuracy
- 75.0 %
- Balanced training would give
- 90.4 %
- Worst class recall
- 26.7 %
- Accuracy cost of the imbalance
- 15.4 points
- Recall cost on the starved class
- 68.3 points
the headline figure
connectivity
Overall accuracy is 75.0 % — respectable enough to be accepted in a report. Recall on connectivity is 26.7 %, so the model misses most of the class it was probably built to catch.
Why this is the normal case, not the pathological one
- The rarest class in an archive is usually the one worth detecting — anomalies, defects and unusual failures are rare by definition.
- So the class most likely to be starved in training is the class the model exists to find.
- And overall accuracy is least sensitive to exactly that class, because it is the smallest share of the test set.
- With more classes it gets worse: a class collapsing costs 16 accuracy points out of four classes and 3 points out of twenty.
What this shows: Accuracy summarises how common the common classes are. Only per-class recall shows what happened to the one you built the model for.
Try it
Engineering classifier
Two abstract symptom scores, four fictional warning categories. Move the point and see which class it is assigned to — and which training examples decided it.
The features are deliberately unnamed. The point is about the data, and naming them would invite you to reason about the symptom instead of the distribution.
- Classified as
- Missing or excessive restraint
- Confidence
- 100 %
- Neighbours consulted
- 3
3 of 3 neighbours agreed
The neighbours agree — which tells you the point is well inside a cluster of training examples, and nothing more than that.
What confidence is and is not
- It is the proportion of the k nearest training examples that voted for the winning class. Nothing more.
- It is NOT a probability that the answer is right. Move the point far outside every cluster and the confidence can still be 100 %, because the nearest neighbours still agree with each other.
- That is the same failure as the surrogate's: a fitted method is defined everywhere, and has no representation of 'outside my experience'.
- k-nearest-neighbour is used here because every prediction is traceable to specific examples — the circled points are the ones that decided it.
What this shows: Confidence is the proportion of neighbours that agreed. It is not a probability that the answer is right.
Worked example
The classifier that stopped finding what it was built for
Given
- A classifier over four warning categories, tested on 60 balanced cases of each
- Version A trained on 80 examples of every class
- Version B trained on 80, 80, 80 and 3 — connectivity faults being rare in the archive
Find
What each version achieves, and which number reveals the difference
Practice
A classifier is tested on 60 connectivity cases and correctly identifies 16 of them. What is its recall on that class, as a percentage to one decimal place?
Practice
Three classes score 90 % recall and a fourth scores 26.7 %, on a balanced test set. What is the overall accuracy, as a percentage to one decimal place?
Check yourself
A model trained on data from completed projects is used to assess whether a proposed scheme is likely to be buildable. What bias should concern you most?
Practice
A class has 60 cases in the test set. The model correctly identifies 51 of them. What is its recall on that class, as a percentage?
Practice
Four classes are trained on 80, 80, 80 and 20 examples. What percentage of the training set does the smallest class make up?
Practice
Overall accuracy falls from 90.4 % to 75.0 % when one class is starved, while that class's recall falls from 95.0 % to 26.7 %. What is the ratio of the recall drop to the accuracy drop?
Check yourself
A defect classifier is trained on models from one practice, and deployed across a group of practices. What is the risk?
Check yourself
Why does re-balancing a training set by discarding majority-class examples have a cost?
Check yourself
A tool's worst-class recall is 27 %. What is it safe to use it for?
Summary
- Accuracy summarises how common the common classes are; recall shows what happened to each
- Starving one class in four dropped accuracy 15 points and recall on that class 68 points
- The rarest class is usually the one you built the model for
- A fitted function is defined everywhere, so it answers outside its experience without saying so
- Selection, survivor, practice, measurement and temporal bias are all properties of how the data came to exist
This is educational material. It uses simplified examples to teach principles, and must not be relied on for real design or safety-critical decisions. Module overview and checkpoint