The dependency map nobody drew

I drew our dependency map because I was tired of being surprised. That is the entire origin story. There was no initiative, no ticket, no moment of architectural clarity. There was a Tuesday where something broke that I hadn’t known was connected to another thing, then a second Tuesday, and somewhere around the third one I opened a file.

At my last job I consolidated 8+ source systems into a single analytics layer. Everyone frames that as a pipeline problem, and the pipelines were fine. Extraction is a solved genre at this point. What kept catching me was the graph nobody had drawn: which upstream change lands on which model, which model feeds which report. And then the column that turned out to matter most, which was who would be upset, and how fast.

Your lineage tool draws the middle of the graph. dbt gives you model-to-model lineage for free and it is genuinely good. It also stops at both boundaries, and both boundaries are where the surprises live. On the upstream end it knows a source exists and nothing about who operates it or what they’re planning next sprint. On the downstream end it knows a mart exists and nothing about the reports, the semantic models somebody copied last quarter, the spreadsheet a director maintains by pasting from a table export every Monday, and whatever else got built while I wasn’t looking.

The middle of the graph almost never surprised me. The middle was tested.

Every edge that matters is owned by a person. So my map had a column for that, and it held a person’s name. Team names are useless here. “The marketing ops team” cannot be pinged, and it does not know that a schema changed. Half those entries started as guesses. Filling them in took weeks and involved a slightly embarrassing number of messages beginning with “sorry, quick question, do you own…”

That exercise turned out to be the actual deliverable. I thought I was documenting a system. What I produced was a list of conversations that should have happened and hadn’t. Two of the eight sources were operated by people who had never spoken to anyone downstream of them. They weren’t being difficult. They had no idea anyone was there. One of them mentioned he’d been renaming export columns for tidiness for about a year, cheerfully, the way you’d rearrange a shelf.

Worry about degree, not complexity. My instinct was to guard the complicated pipelines, the ones with transformation logic I understood least. Wrong instinct. The complicated ones got care precisely because they were frightening. The dangerous nodes were the boring high-degree ones: a customer dimension that eleven downstream things depended on, joined by everybody, owned by nobody in particular, extended whenever someone needed one more attribute. A change there had a blast radius that no single person could hold in their head. That is more or less the definition of a node that needs to be written down.

I’d seen a smaller version of this at the agency I interned at, where marketing and customer data flowed into a central warehouse powering dashboards across 8+ accounts. Each account looked like its own tidy little graph. They shared plumbing, though, and the shared parts were invisible from inside any one account. A change made for one client could reach the other seven, and the only person positioned to see that was whoever had bothered to look across all of them at once. For a while that was nobody, and then briefly it was me, which is a promotion of sorts.

The map is worth less than the map-drawing. I say this having eventually produced a nice diagram. Almost nobody opened it. What survived was that I now knew, without looking, which four changes would ruin a week and which people to warn before making them. The document was a byproduct of learning the terrain.

A deprecation I ran later is the best evidence I have. I was about to retire a table I was confident had one consumer. The map said three. Two were live reports, and one of those belonged to someone who had left months earlier. Nobody had inherited it. It was still quietly on a team’s Monday routine, being read by people who assumed a responsible adult was maintaining it. Without the map I would have dropped the table and learned about this from a confused message two days later.

The other thing the map changed was who I talked to before shipping anything. Standing up a new model used to be a technical decision. After the map it came with a short list of people to mention it to, which sounds like bureaucracy and functioned like insurance. Most incidents I didn’t have are incidents I can’t prove, which is the permanent PR problem of this kind of work.

(The consolidation that all of this came out of is the case study on my home page, architecture and numbers included.)

My map is out of date now. It was out of date the week after I finished it, because that is what maps of living systems do. It still beat the alternative, which was being surprised on a schedule set by other people.