The dashboard was wrong for six weeks and nobody noticed

The worst data incidents are the quiet ones.

A schema change upstream turned an inner join into a row shredder. Nothing crashed. No alert fired. The revenue dashboard kept rendering every morning, numbers slightly low, well within the range where a human squinting at a chart says “hm, soft month” and moves on. Six weeks later someone in finance reconciled against the billing system and the gap was too big to be a soft month.

I did the postmortem. The pipeline was healthy by every measure we had, because every measure we had asked “did the job run?” None of them asked “is the output right?”

Those are wildly different questions. A cron job that exits 0 after loading half your rows is, by its own account, a success.

The fix was less glamorous than the incident. We added validation as a pipeline stage, not a dashboard. Row counts against source, reconciliation totals against the system of record, uniqueness and referential checks in dbt, freshness thresholds per table. The important design decision: a failed check blocks the publish. Data that fails validation never reaches a dashboard, so the failure mode changes from “wrong number quietly displayed for six weeks” to “stale number, loudly explained, fixed in hours.”

Stale beats wrong. It took me an embarrassingly long time to internalize that. A stakeholder looking at yesterday’s correct data makes better calls than one looking at today’s fiction.

Measured over the following months, dashboard accuracy went from roughly 88% to 98%, and the monthly rework on reports dropped by about a third. But the number I actually care about is subjective: people stopped opening tickets that said “this looks off.” When the data looked off, the pipeline had already said so first.

If your monitoring only knows whether jobs ran, you don’t have monitoring. You have a wall of green lights wired to nothing.