The migration ran against the wrong database and reported failure

I ran a migration against staging. It failed, complaining about null values in a column it was trying to make unique.

Fine — that is what the documented backfill script is for. I ran the backfill.

It reported: backfilled 0 rows.

Those two results cannot both be true about one database. Either there are nulls or there are not.

Two commands, two databases

The migration script reads DATABASE_URL. The secret manager stores the connection string as DB_URL.

So DATABASE_URL was simply not set when the migration ran. That should have been a loud error. It was not, because the script had a fallback:

postgresql://localhost:5432/..._dev

A hardcoded local development database. The migration connected to my laptop, found the nulls that genuinely exist in dev data, and failed on those.

The backfill script — reading the right variable — connected to staging, found no nulls because staging was clean, and correctly reported zero.

Both outputs were accurate. They described different databases, and nothing in either message said which one.

The fallback is the bug

The missing variable was a small configuration mismatch, easily fixed once seen. The fallback is what made it cost an evening.

Without it, the migration would have said “DATABASE_URL is not set” and I would have fixed it in under a minute. With it, a missing variable became a wrong target success — the command connected, ran, and produced a plausible, detailed, completely misleading error about data.

That is the worst possible failure mode. Not a crash. Not a hang. A confident, specific error message about the wrong system, which sends you off debugging data that is fine.

Convenience defaults are appropriate for a dev server. They are never appropriate for anything that writes to a database. The whole point of a migration tool is that it modifies durable state, and it must be impossible to be uncertain about which state.

The corrected invocation, documented nowhere before this, maps one variable to the other explicitly at the call site.

What made it hard to see

A wrong hypothesis fits the evidence for a surprisingly long time. “Staging has dirty data” explained the migration failure perfectly. I only stopped believing it when the backfill contradicted it, and my first instinct was that the backfill was broken.

The habit worth building: when two tools disagree about the same system, first check they are talking to the same system. Not that one is wrong. Compatible statements about different targets look exactly like contradictory statements about one.

Faster diagnostic than any of my hypotheses: make the tool print the host it is connected to before it does anything. Three words of output that would have ended this immediately.

What I changed

Fallbacks removed from anything that writes. Missing configuration exits non-zero with the name of the variable it wanted. Loud beats convenient.

Every migration prints its target first. Host and database name, before the first statement. If you cannot say out loud which database you are about to change, you should not be changing it.

One name for the connection string. The mismatch between what the code read and what the secret store held existed because two people named the same thing differently at different times, and nothing forced a reconciliation.

Runbooks carry exact commands. The invocation that works is now written down in full. “Run the migration against staging” is not a runbook; it is a summary of one, and the difference is where this bug lived.

Related

• Enterprise deployment - the wider case for making the target of an action explicit.
• When the agent gets it wrong - reading a transcript when a plausible hypothesis fits the wrong evidence.