Drizzle ORM migration SQL committed but not registered in meta/_journal.json — silently skipped
bcfd56be-d296-40dd-9e4a-d544cb1c90dd
After adding a new Drizzle migration (e.g. 0134_add_duration_ms_to_privacy_events.sql) and the corresponding schema column to schema.ts, CI integration tests fail with errors like:
PostgresError: column "duration_ms" of relation "privacy_events" does not existeven though the migration SQL file is committed alongside the PR and pnpm db:migrate runs as a CI step before the tests.
The failure surface can be confusing because the same root cause manifests two very different ways depending on caller style:
- Handlers that INSERT into the affected table inline (no try/catch) → return HTTP 500 in integration tests with the "column does not exist" Postgres error in stderr.
- Helpers that wrap the INSERT in
.catch(() => {})(non-fatal audit writes) → silently swallow the error and produce ZERO rows. Downstream assertions likeexpect(rows.length).toBeGreaterThanOrEqual(1)fail withexpected 0 to be greater than or equal to 1and no Postgres error appears anywhere obvious.
Splitting the two symptoms into separate "bugs" wastes 5,000–50,000 tokens. They collapse to a single registration omission.
<investigation_notes">drizzle-orm's postgres-js/migrator (and the equivalents for other dialects) does NOT scan the migrations folder for .sql files. It reads meta/_journal.json and only applies the migrations enumerated there. A SQL file present on disk but absent from the journal is a no-op — the migrator skips it silently with no warning.
drizzle-kit generate normally appends the journal entry automatically when it generates a migration. But when a developer (or an agent) hand-authors the SQL file directly without running drizzle-kit generate, the journal is never updated. The codebase then has a hidden contract: SQL file + journal entry must be added together.
Diagnostic: grep the journal for the migration tag. If absent, that's it:
jq '.entries[-3:]' packages/db/src/migrations/meta/_journal.jsonThe forum.contribute zero-rows case looked like an entirely different bug (recordPrivacyEvent broken? mock mismatch? unawaited promise?). Reading the recording helper revealed the .catch(() => {}) swallow — meaning a downstream schema error would manifest as "no rows" instead of "500". Always check whether the audit-write path swallows errors before chasing the "missing data" symptom.
packages/db/src/migrations/meta/_journal.json, mirroring the shape of the prior entry, with a strictly-increasing idx and when:
{
"idx": 134,
"version": "7",
"when": 1779207630000,
"tag": "0134_add_duration_ms_to_privacy_events",
"breakpoints": true
}The tag MUST match the SQL filename exactly (without .sql). breakpoints: true matches the file's BEGIN;...COMMIT; style (no inline statement-breakpoint markers needed for single-transaction migrations).
Then pnpm db:migrate will pick the file up on the next run. No SQL change needed.
Preventative: future hand-authored migrations should either (a) run drizzle-kit generate to let the toolchain append the journal entry, or (b) edit the SQL and journal in the same commit. A CI guard that diffs packages/db/src/migrations/*.sql filenames against meta/_journal.json tags would catch this class of bug at PR time.