The anomaly detector that cried wolf

Early in my time doing financial risk analytics, I built an anomaly detection model I was proud of. Good precision-recall curve, sensible features over transaction and credit data, XGBoost because gradient boosting on tabular data remains the reigning champion no matter what the deep learning people say at parties.

It flagged around thirty items a day. The risk team investigated roughly none of them.

I took this personally for about a week, then actually talked to the analysts, which is what I should have done before building anything. Their capacity was maybe five investigations a day, each taking real time. A queue of thirty meant the queue was noise by definition. They’d sampled it, found mostly false alarms, and rationally concluded my model was a spam folder. An alert system that people have learned to ignore is worse than no system, because now “it was flagged” provides false comfort in the postmortem.

The fix was a reframe. The model’s job was never “find all anomalies.” Its job was “spend five investigations well.” So: rank by score, take the top five, suppress repeat flags on the same entity within a window, and add a one-line reason for each flag, because “why is this here” was the first question every analyst asked and shap values were sitting right there anyway.

Same underlying model. Completely different system.

People started working the queue. And once the queue was trusted, the model got better, because analysts began disputing individual flags (“this pattern is normal for that client type”), which became the feedback that actually improved the features. Trust unlocked the data flywheel, not the reverse.

The outcome that mattered: potential issues surfacing three to four weeks earlier than the old process caught them. Weeks matter enormously in credit risk. Problems compound.

What I took forward from this, into every alerting system I’ve built since, data-quality alarms included: the scarce resource is never detection. It’s attention. Design for the amount of attention that exists, not the amount your recall metric wishes existed.