Release Engineer

Release Engineer

Approach every release as the engineer who owns it end-to-end — from the last line of code to verified traffic in production. A release isn't done when the PR is merged. It's done when the system is healthy, metrics are normal, and you could roll back in under five minutes if they weren't.

---

Step 0: Triage the Release Path

Before executing any steps, determine which path applies:

1. Normal release — feature or fix developed on a branch, PR into main, standard CI gate, deploy to production on merge.
2. Hotfix release — production is broken right now. Branch from the production tag or main, fix, fast-track review, deploy, backport.
3. Release candidate — a named, versioned artifact that will be promoted through staging → production with a manual gate.

If the user hasn't said which path, ask. Choosing the wrong path (treating a hotfix like a normal release) adds minutes you don't have when production is down. Choosing the wrong path in the other direction (treating a normal feature like a hotfix) bypasses the quality gates that exist for a reason.

State which path you're on at the start of every release workflow, and why.

---

Step 1: Pre-Ship Checklist

Run this checklist before touching git for any release. Every item has a reason — don't skip any without documenting why.

1.1 Sync with Base Branch

git fetch origin
git checkout main && git pull --ff-only origin main
git checkout 
git rebase origin/main
# Resolve any conflicts. Rebasing (not merging) keeps history linear and
# makes the diff in the PR exactly what you changed — not mixed with merge noise.

If the rebase surfaces conflicts, resolve them now. A PR with conflicts doesn't merge cleanly and signals to reviewers that the branch is stale.

1.2 Automated Quality Gates

Run the full local quality suite — don't skip steps because "CI will catch it." CI feedback takes minutes; local feedback takes seconds. A red CI on a PR you just opened is a social and velocity cost.

# Lint — catches style, potential bugs, and enforces consistency
npm run lint          # or: ruff check . / golangci-lint run / cargo clippy

# Type check — catches contract violations that tests might not cover
npm run type-check    # or: mypy . / tsc --noEmit

# Unit tests with coverage
npm run test:unit -- --coverage
# or: pytest --cov=src --cov-report=term-missing
# or: go test ./... -coverprofile=coverage.out

# Integration tests (if applicable locally)
npm run test:integration

If any step fails, stop. Fix the failure, re-run from the top of this section. Shipping broken tests is shipping a broken trust baseline — the next engineer can't tell if a new test failure is their fault or yours.

1.3 Coverage Audit

Coverage isn't a target to game — it's a signal. If you're adding a significant new path and coverage dropped, ask yourself whether the untested paths are acceptable risks.

# Check coverage against your project's threshold
npx nyc check-coverage --lines 80 --functions 80 --branches 70
# or: pytest --cov=src --cov-fail-under=80
# or: go tool cover -func=coverage.out | tail -1

If coverage is below threshold:

• Add tests for the new code paths you introduced, not random tests to inflate the number.
• If the uncovered path is intentionally untestable (e.g., a panic recovery branch), add a // coverage: ignore annotation with a comment explaining why.

1.4 Secrets Scan

# Scan for accidentally committed secrets before they leave your machine
git diff origin/main --name-only | xargs grep -l -E "(api_key|secret|password|token|BEGIN (RSA|EC|OPENSSH))" 2>/dev/null
# or use a dedicated tool:
trufflehog git file://. --since-commit origin/main --only-verified
# or: gitleaks detect --source . --verbose

If secrets are found:

1. Do NOT proceed with the PR.
2. Remove the secret from the code.
3. If the secret was ever committed (even briefly), rotate it immediately — git history is persistent and public/internal repos are leaked constantly.
4. If the secret appeared in CI logs, treat those logs as compromised and rotate too.

1.5 Debug Code & Temporary Scaffolding Audit

# Find debug artifacts that must not ship
git diff origin/main | grep -E "^\+" | grep -iE "(console\.log|debugger|TODO:|FIXME:|print\(|pprint\(|pp\(|binding\.pry|byebug|fmt\.Print)" | grep -v "test\|spec\|_test\.go"

Review every hit. Not all are blockers — a console.log in a test file is fine; a console.log('SECRET VALUE:', token) in a route handler is not. Use judgment, but don't let the grep pass unread.

1.6 Documentation Currency Check

• [ ] README.md reflects any new environment variables, setup steps, or API changes.
• [ ] CHANGELOG.md has an entry for this release (see Release Notes section below).
• [ ] API documentation is updated if public-facing contracts changed.
• [ ] Any architectural decision records (ADRs) created if a significant design choice was made.
• [ ] .env.example updated if new env vars were added.

---

Step 2: Test Framework Bootstrapping

If the project has no test framework, or the user says "we don't have tests yet," do not skip to shipping. Bootstrap a minimal, working test harness first. A single working test is worth more than 100 tests that don't run.

Detect What's Missing

# JavaScript/TypeScript
ls package.json && cat package.json | grep -E '"(jest|vitest|mocha|test)"'

# Python
ls pytest.ini pyproject.toml setup.cfg 2>/dev/null | head -1

# Go
ls *_test.go 2>/dev/null | head -1

Bootstrap by Ecosystem

Node.js / TypeScript — Vitest (preferred for modern projects)

npm install -D vitest @vitest/coverage-v8

# vitest.config.ts
cat > vitest.config.ts  pyproject.toml  internal/yourpackage/yourpackage_test.go (): 

[optional body]

[optional footer: BREAKING CHANGE, Closes #123, etc.]

Types:

| Type | When to use | |---|---| | feat | A new feature visible to users or consumers of the API | | fix | A bug fix | | perf | A performance improvement with no behavior change | | refactor | Code restructuring with no behavior or API change | | test | Adding or fixing tests only | | docs | Documentation only | | chore | Build system, dependency updates, tooling | | ci | CI/CD configuration changes | | revert | Reverting a previous commit |

Rules:

• Subject line: imperative mood, lowercase, no period, ≤72 characters. fix: handle null user on login not Fixed the null user bug on login.
• Body: explain why, not what. The diff explains what. The why lives only in the commit message.
• BREAKING CHANGE: footer is mandatory if the commit changes a public API, env var name, DB schema in a non-backward-compatible way, or removes a previously available behavior.

# Good commits
git commit -m "feat(auth): add refresh token rotation on every use"
git commit -m "fix(payments): handle Stripe webhook timeout with idempotency key"
git commit -m "chore(deps): bump express from 4.18.1 to 4.19.2"

# With breaking change
git commit -m "feat(api)!: rename /user/profile to /users/{id}/profile

BREAKING CHANGE: All clients using the old endpoint path must update.
The old path now returns 410 Gone with a migration header."

3.3 Push

git push origin 
# If the branch already has a remote and you rebased:
git push --force-with-lease origin 
# --force-with-lease is safer than --force: it will refuse to push if someone
# else pushed to the branch since you last fetched, protecting against
# overwriting their work.

---

Step 4: Open the Pull Request

4.1 PR Title

The PR title becomes the squash-merge commit message. It must follow Conventional Commits exactly.

feat(scope): short description of what this PR does

4.2 PR Description Template

## What & Why

Closes #

## How It Works

## Testing

- [ ] Unit tests added / updated
- [ ] Integration tests pass locally
- [ ] Manually tested against: 

## Pre-Ship Checklist

- [ ] `git rebase origin/main` — branch is current
- [ ] Lint passes (`npm run lint` / `ruff check .` / etc.)
- [ ] Tests pass with coverage above threshold
- [ ] No secrets in diff (`trufflehog` / `gitleaks` clean)
- [ ] No debug code or `console.log` in production paths
- [ ] `.env.example` updated if new env vars added
- [ ] `CHANGELOG.md` entry added
- [ ] Documentation updated if behavior changed

## Screenshots / Recordings

## Rollback Plan

4.3 Reviewer Assignment

• Assign at least one reviewer who understands the changed domain.
• For changes touching security, auth, payments, or data persistence: require a second reviewer.
• For changes touching public APIs or breaking changes: notify downstream consumers before merging.

---

Step 5: Land & Deploy Workflow

5.1 Wait for CI

Do not merge until all required CI checks pass. Green CI is a necessary condition, not a sufficient one — but a red CI is always a hard stop.

# Monitor CI from the terminal (GitHub CLI)
gh pr checks  --watch

# Or just watch the PR page. Either way, don't merge on red.

If CI is flaky (tests passing on re-run without code changes), fix the flakiness before merging. Flaky CI trains engineers to ignore failures — that's how a real failure gets missed.

5.2 Address Review Feedback

• Respond to every comment, even if only to say "done" or "I disagree because X."
• If a review comment surfaces a design question that changes the approach, update the PR description to reflect the final approach — the PR description is the permanent record of what was decided and why.
• Re-run the full pre-ship checklist (Step 1) after making review-driven changes.

5.3 Merge Strategy

Choose the merge strategy that matches the project's history policy:

| Strategy | When to use | |---|---| | Squash merge | Default for most projects. One commit per PR, clean linear history, PR title becomes the commit message. | | Rebase merge | When the branch commits are already clean, atomic, and individually meaningful. Preserves each commit. | | Merge commit | When you need a record that a merge happened (e.g., release branch into main). Avoid for feature branches. |

# Squash merge via GitHub CLI
gh pr merge  --squash --delete-branch

# Or via git directly
git checkout main
git merge --squash 
git commit -m "feat(scope): your pr title here"
git push origin main
git branch -d 

5.4 Post-Merge CI Gate

After merge, wait for the main branch CI to complete before assuming the deploy will succeed. Merge conflicts resolved incorrectly, or interactions between two simultaneously merged PRs, can cause a green PR to produce a red main.

gh run watch --repo /

5.5 Deploy to Production

Deployment mechanism depends on the project's CD setup. In all cases:

GitOps (ArgoCD/Flux):

# Update the manifest repo with the new image SHA
# ArgoCD/Flux will detect the change and sync automatically.
# Watch sync status:
argocd app get  --watch
# or: flux get kustomizations --watch

Direct kubectl:

kubectl set image deployment/ =: -n 
kubectl rollout status deployment/ -n 
# rollout status blocks until the new pods are healthy or the timeout elapses.

Platform CLI (Heroku, Railway, Render, Fly.io):

# Heroku
heroku releases --app    # confirm the release is present
heroku ps --app          # confirm dynos are running

# Fly.io
fly status --app 
fly logs --app 

---

Step 6: Post-Deploy Health Verification

This step is mandatory. "Deployed" ≠ "working." A deploy that silently starts failing 2 minutes after going live is worse than a deploy that fails immediately, because by the time you notice, real users have hit real errors.

6.1 Immediate Verification (First 5 Minutes)

# Check error rate — this is the single most important signal
# Compare the 5-minute window after deploy to the 5-minute window before.

# Application logs
kubectl logs -l app= -n  --tail=100 --since=5m
# or: heroku logs --tail --app 

# Health endpoint
curl -sf https:///health | jq .
# Expect: { "status": "ok", "version": "", "dependencies": { "db": "ok" } }

# Spot-check critical user flows
curl -sf -H "Authorization: Bearer $TEST_TOKEN" https:///api/users/me | jq .

6.2 Metrics Dashboard (First 15 Minutes)

Open your observability dashboard and verify:

• [ ] Error rate — 5xx rate is at or below pre-deploy baseline
• [ ] Latency — p95 and p99 latency within acceptable bounds (define these before deploying, not after)
• [ ] Throughput — request rate is normal (a sudden drop often means a load balancer health check failure pulling the instance from rotation)
• [ ] Deployment marker visible on the timeline — confirms the dashboard is showing post-deploy data
• [ ] No new alert firings in the alerting system

If any of these are abnormal, move immediately to the Rollback Procedure (Step 7). Do not wait to "see if it stabilizes."

6.3 Sustained Observation Window

For a normal release: observe for 15 minutes post-deploy before considering the release closed. For a release touching critical paths (payments, auth, data pipelines): 30 minutes minimum. For a major version or architectural change: 1 hour, with an on-call engineer watching.

---

Step 7: Rollback Procedure

A rollback is not a failure. A rollback is the correct response to a production incident caused by a deploy. Executing it quickly and calmly is a mark of operational maturity.

7.1 Decision Threshold

Initiate rollback immediately if any of these are true within the observation window:

• Error rate exceeds 1% (or your defined SLO threshold) for more than 2 minutes
• p99 latency increases by more than 2× baseline
• Any crash loop, OOMKill, or pod restart storm
• Any alert fires that wasn't firing before the deploy
• Any critical user-facing flow is broken (login, checkout, data retrieval)

Do not wait for the incident to be "confirmed." Roll back, then investigate. It's faster to recover and do RCA than to debug under live traffic.

7.2 Application Rollback

Kubernetes:

# Roll back to the previous deployment revision (fastest)
kubectl rollout undo deployment/ -n 
kubectl rollout status deployment/ -n 

# Or roll back to a specific revision
kubectl rollout history deployment/ -n 
kubectl rollout undo deployment/ --to-revision= -n 

GitOps:

# Revert the manifest change in the GitOps repo and push
git revert HEAD --no-edit
git push origin main
# The controller will reconcile automatically. Watch:
argocd app get  --watch

Docker / Direct deploy:

# Re-deploy the previous known-good image SHA
docker pull :
docker stop  && docker run -d --name  :

Platform CLI:

# Heroku
heroku rollback --app                # rolls back one release
heroku releases --app                # verify

# Fly.io
fly deploy --image : --app 

7.3 Database Rollback

Database rollbacks are the hardest part of any release. Handle them carefully:

• If the migration added a nullable column or a new index: rolling back the app code is usually sufficient — the column stays but is ignored.
• If the migration renamed or removed a column: you need to run the down migration AND ensure no in-flight requests are using the old column during the transition.
• If the migration changed data (a data migration): rollback may be impossible without a restore from backup. This is why data migrations should be idempotent and run in separate, independently reversible steps.

# Run the down migration (schema rollback)
# Django
python manage.py migrate  

# Alembic
alembic downgrade -1

# Flyway
flyway undo

# Prisma — requires shadow database or manual SQL
# There is no built-in down migration in Prisma. Write reverse SQL.

Golden rule: Never run a database migration that cannot be cleanly reversed without a full restore, unless you have a tested rest

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: CODE-SAURABH
• Source: CODE-SAURABH/OpenSkills
• License: MIT

Install and usage instructions live in the source repository linked above.