Intersection crash report

This is the first delivery of the crash analysis service. What a city receives and how to engage: Crash analysis for cities.

Brief

A Metro Vancouver municipality had ten years of ICBC crash records for 35 intersections, 2013 to 2022. It wanted them in a form an engineer can act on, with one collision diagram per intersection, a report to read and a workbook to sort.

The records came as 1,331 free-text narratives written by claims handlers.

Constraint

Claims handlers write in their own shorthand, with abbreviations for the drivers, the crash and every direction of travel, and no two write it quite the same way. The city’s QC sheet codes vehicle movements in its own notation, which was not consistent either.

A diagram built from a wrong reading is worse than no diagram. So the classifier had to be accurate, say so when a narrative could not be decided, and leave every call open to checking by the city’s engineers.

What was built

We built a pipeline that cleans, classifies, validates, sends doubtful cases to review, and writes the report.

Each narrative is cleaned up before anything reads it, and every row of the city’s QC sheet reads into the same structure, so the model’s answer and the human label can be compared line by line.

Our own model, fine-tuned for claims-handler shorthand, reads each cleaned narrative and returns approach directions, movements, pedestrian or cyclist involvement, and a crash configuration. It runs on our hardware, so the narratives never leave the office. Checks against the intersection’s geometry then test every answer before it is accepted. Every result is stored with its input, so any answer can be traced back to the text it came from.

The report is LaTeX, 98 pages. Each intersection gets a two-page section with a collision diagram drawn from hand-made crash-arrow glyphs, followed by charts of crash type, severity, time of day and day of week. The Excel workbook holds the same data in rows.

Corrections go through a loop. Flagged cases land on a review page, one HTML file that opens in any browser with no server behind it. The engineer changes a call and downloads a corrections file. We apply it, the next run starts from the corrected answer, and the report prints every correction in an appendix.

What it measured

The target was 92% on crash configuration, scored against a sealed holdout of 300 cases. The result was 95.0%, 285 of 300. Nobody looked at the holdout, and nothing was tuned on it. We read it only to decide whether to stop or keep going. The set we tuned on scored 94.7%, up from 89.0% at the start, and the holdout came in 0.3 points higher than that set on both confirmation runs. Pedestrian and cyclist involvement scored 100%.

Of the 15 holdout misses, 9 are side impacts read as same-direction sideswipes, and the needs-review flag sends 10 of the 15 to the engineer’s review page.

The labels we scored against came from the city’s own hand-checked QC sheet, and the human process had made mistakes too. When we lined the model’s answers up against the sheet, 527 cases disagreed, and each one was checked against the narrative itself. In 99 of them the narrative contradicts the human label, 30 of those with the wrong crash configuration, and another 10 incidents had been entered twice with conflicting labels. The city’s QC had passed every one of them. They came to light only because the model disagreed. We corrected them, with the narrative’s own words as evidence, before scoring anything, so the 95.0% is measured against the corrected labels. The model now in service, our own fine-tuned model, classifies these narratives with much higher accuracy than the human process it was checked against.

Most of what we tried did not work. Every gain came from fixing the deterministic checks around the model, which had been throwing correct answers away, and we found those problems by auditing past results rather than running anything new. Every change aimed at the model itself lost ground, and none of them fixed a single case.