95.0% on a sealed holdout, and the experiments that made it worse
We classify ICBC crash narratives for a Metro Vancouver municipality. Each narrative is a few lines of claims-handler shorthand. The pipeline reads it and returns each vehicle’s approach direction and movement, plus one of six crash configurations, such as rear end, side impact or same-direction sideswipe. The configuration decides which arrow goes on the collision diagram, so it is the number we tuned for.
Seal first, then tune
Before changing anything, we took 300 cases out of the hand-checked label set and sealed them. Nobody inspected them and nothing was tuned on them. We read them only to decide whether to keep going.
The stopping rule was written down before the first experiment. If the tuning set and the holdout drifted more than 5 points apart on two confirms in a row, we would stop and call it overfitting. The target was 92% configuration accuracy on the holdout.
The labels were wrong too
The labels came from the city’s own hand-checked QC sheet, 1,341 rows. We treated it as the answer key, and it turned out to have errors of its own.
Lining the model’s answers up against the sheet gave 527 disagreements, and we checked each one against the narrative. In 99, the narrative contradicts the human label, 30 of them with the wrong crash configuration. Another 10 incidents had been entered twice with conflicting labels. The human review had passed every one of them. They surfaced only because the model disagreed.
We corrected them, citing the narrative’s own words each time, and every number below is scored against the corrected labels. It changed how we think about the job. The model is not only something to check against the humans. It is also a second reader that catches what the humans missed.
The result
The holdout scored 95.0%, or 285 of 300. The 300 tuning cases scored 94.7%. On both confirms the holdout came in 0.3 points higher than the set we had tuned on, which is about as clean a sign as you get that the gains were real and not memorised. Pedestrian and cyclist involvement scored 100%.
Where the points came from
Holdout accuracy rose 6.0 points over the campaign. None of it came from changing the model. All of it came from the deterministic code around the model, which had been quietly throwing correct answers away.
We found those problems by reading old results, not by running new experiments. Nothing in a pipeline fails loudly when a check misfires or never runs at all. The output still looks plausible, and the cost shows up only as a few points of accuracy nobody can account for. Auditing what we already had was worth more than anything we added.
The experiments that lost ground
Then we tried to improve the model’s own reading. Every one of those changes lost ground on the screening set. Across all of them, not one case got fixed.
Looking back, the ideas were written without first auditing what was already covered. Most of what we set out to add was already there, so each experiment tested a small new fragment, and it did nothing or did harm. The worst of them also taught us that the parts behave as a system, because a change aimed at one kind of crash damaged others it was never meant to touch.
The model’s mistakes, it turned out, were consistent. It made the same wrong call on the same kind of narrative every time. Consistent errors are the checks’ job to catch and the engineer’s job to rule on, not a reason for another round of experiments on the model.
Where the engineer takes over
Of the 300 sealed cases, 15 were misclassified. Nine of them are side impacts read as same-direction sideswipes, where the narrative mentions the point of impact, front into side or flank along flank, only in passing. The needs-review flag catches 10 of the 15 and puts them on the review page, where the city’s engineer makes the final call.
That is the design. No crash goes into the report on a guess the pipeline cannot stand behind, and every call the engineer changes is printed in the report’s appendix.