Migrations

Migrations (safe schema and data change on PostgreSQL)

When to use

Any time a change produces a new migration — a new column, a data backfill, a rename, a drop, or an index. On PostgreSQL a careless migration takes an ACCESS EXCLUSIVE lock or rewrites a large table, so treat "does this block writes in prod?" as a review question, not an afterthought.

Pattern

Two invariants hold everywhere:

1. Data migrations are reversible and idempotent. Every RunPython gets a

real reverse callable (or RunPython.noop), and re-running it is a no-op — never assume it runs exactly once on clean state.

1. Schema changes that lock or rewrite are split (expand-contract). Add the

new shape, backfill, switch reads/writes, then drop the old shape in a later deploy — so no single migration takes a long exclusive lock.

Tenant data lives under entreprise, so backfills iterate per tenant in batches rather than loading one giant queryset.

Steps / idioms

1. Never import models at module top level in a data migration. Use the

historical model via apps.get_model, give a reverse, and make it idempotent:

```python from django.db import migrations

def backfillslug(apps, schemaeditor): Invoice = apps.getmodel("billing", "Invoice") # historical model, not the import # tenant-aware: iterate per entreprise, batch to bound memory + lock time for entid in Invoice.objects.valueslist("entrepriseid", flat=True).distinct(): qs = Invoice.objects.filter(entrepriseid=entid, slug="") # idempotent: only unset rows for inv in qs.iterator(chunksize=500): inv.slug = f"inv-{inv.pk}" inv.save(updatefields=["slug"])

def unsetslug(apps, schemaeditor): Invoice = apps.getmodel("billing", "Invoice") # reverse ONLY the rows this migration generated; leave pre-existing slugs alone. # An unconditional .update(slug="") would destroy slugs the forward step never wrote. for entid in Invoice.objects.valueslist("entrepriseid", flat=True).distinct(): qs = Invoice.objects.filter(entrepriseid=entid).exclude(slug="") for inv in qs.iterator(chunksize=500): if inv.slug == f"inv-{inv.pk}": # our value, not the user's inv.slug = "" inv.save(updatefields=["slug"])

class Migration(migrations.Migration): dependencies = [("billing", "0007invoiceslug")] operations = [migrations.RunPython(backfillslug, unsetslug)] ```

A reverse must undo only what the forward step did. If the generated value is not recognisable, do not guess — declare it: migrations.RunPython(backfill_slug, migrations.RunPython.noop) # intentionally irreversible: cannot distinguish generated slugs from pre-existing ones.

1. Split schema from data. One migration adds the nullable/blank column; a

separate RunPython backfills; a later migration adds NOT NULL/constraints. Mixing DDL and a long data loop in one migration holds the lock the whole time.

1. Add indexes concurrently on hot tables so writes are not blocked. Use

AddIndexConcurrently (from django.contrib.postgres.operations) and set atomic = False on the Migration — CREATE INDEX CONCURRENTLY cannot run inside a transaction. Pass the model name, not the app label.

Expand-contract (zero-downtime rename)

A single RenameField rewrites nothing but breaks any running old code mid-deploy. Split across releases instead:

1. Expand — add the new column; dual-write to both old and new in app code.
2. Backfill — RunPython copies old to new per entreprise, in batches.
3. Switch — ship code that reads/writes only the new column.
4. Contract — a later migration drops the old column, once no code references it.

Each step is independently deployable and reversible. Never collapse them.

Adapt to your repo

Rename entreprise/Invoice/billing and the app label to match your project. Confirm whether your table is large enough to need atomic = False + concurrent indexes (small tables are fine with a plain AddIndex). Pick chunk_size to fit your row width. If a backfill is huge, run it as an idempotent management command or celery-task and keep the migration to schema only.

Gotchas

• Importing the app model directly (from billing.models import Invoice) breaks

when the migration replays against old state — always apps.get_model.

• A RunPython with no reverse blocks migrate from rolling back; pass

RunPython.noop when the change genuinely cannot be undone.

• A reverse that is broader than its forward is data loss, not a rollback. If the

forward only touched unset rows, the reverse must only reset rows it generated — an unconditional Model.objects.update(field="") wipes values that predate the migration, across every tenant, and no error is raised. When you cannot tell your value from the user's, RunPython.noop plus an explicit "intentionally irreversible" comment is the honest reverse.

• CREATE INDEX CONCURRENTLY fails inside a transaction — it needs atomic = False,

and a failed concurrent index leaves an INVALID index you must drop by hand.

• Adding a NOT NULL column with a default on a big table rewrites it under a lock;

add nullable, backfill, then set NOT NULL in a follow-up.

• Run python manage.py makemigrations --check --dry-run in CI so a model change

that forgot its migration fails the build (see ci-cd).

• On a multi-instance or rolling deploy, run migrate as a single leader-only

step (one instance), never on every instance's startup — concurrent migrate runs race the migration ledger and can deadlock or double-apply.

• Never edit a migration that is already applied or committed — add a new forward

migration. Two branches that each add a migration to one app clash; resolve with makemigrations --merge and rebase. --fake marks a migration applied without running it — safe only when the DB already matches that state, otherwise it silently skips the real schema change.

• Decide where a data change lives. A one-time backfill tied to this schema

change belongs in RunPython in the migration; evolving reference/config data belongs in an idempotent, re-runnable seed management command (not a migration — migrations are frozen and replay everywhere forever); a one-off repair of bad data belongs in a run-once fix command.

See also

• multi-tenancy
• db-concurrency
• ci-cd

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Deadlymind
• Source: Deadlymind/nanolama
• License: MIT

Install and usage instructions live in the source repository linked above.