DuckDB is my new scratchpad

I resisted DuckDB for a while because the hype had that crypto-adjacent smell. Everyone insisting a thing is the future, few showing their homework. Then I had a debugging session that changed my routine.

The situation: a mart in the warehouse had a suspicious dip, and I had a hunch it came from one source file among a few hundred parquet exports. My old workflow would have been loading candidates into a scratch schema, waiting, paying, querying, repeating. Instead I pointed DuckDB at the directory on my laptop:

select filename, count(*), min(event_time), max(event_time)
from read_parquet('exports/*.parquet', filename = true)
group by 1 order by 3;

Found the gap in about ninety seconds. One file had a four-hour hole. The whole investigation never touched the warehouse.

That’s the shape of what DuckDB is for, at least in my hands. It hasn’t replaced Snowflake for anything and it isn’t supposed to. Production models, shared data, anything with concurrency or governance needs — warehouse, obviously. What it replaced is the awkward middle zone where I used to choose between pandas groaning through a few gigabytes and firing up warehouse credits to answer a question only I care about.

Prototyping dbt logic before it ships. Sanity-checking a vendor’s export against what they claim it contains. Sniffing at a dataset someone handed me on a call, while still on the call. Local, instant, free, and it speaks real SQL with window functions instead of making me remember pandas’ groupby-apply incantations.

The pattern worth naming: an enormous amount of data work is provisional. Questions you ask once, hypotheses you kill in ten minutes, drafts. Routing provisional work through production infrastructure was never a considered decision. It was just the only place SQL lived.

Now SQL lives on my laptop too, and the warehouse is reserved for work that’s actually meant to be shared. Correct tool sizes for correct problems. Took the industry a weirdly long time to offer this and it took me a weirdly long time to accept it.