Aws Infra Migration

AWS infra migration — fenced one-writer state cutover

Move a Terraform root's live remote state from one CI/repo (the source) to another (the destination) without split-brain, keeping every managed resource's identity so running workloads never notice.

The one invariant: exactly one writer for a given state key at every instant. Source and destination use different backend keys, so the DynamoDB lock table locks them independently — there is no shared lock. "Copy the object and leave the old workflow armed" is split-brain. You make one writer, fence the other, and only ever move forward.

Orient first (do not reinvent)

The per-root procedure should be authoritative in the destination infra repo, conventionally at docs/runbooks/fenced-cutover.md, with a transfer-manifest.template.md and filled exemplars under docs/runbooks/manifests/. Read the runbook and the most recent exemplar; copy the exemplar's shape. This skill is the operator's wrapper around that runbook — it does not replace it. If they disagree, the runbook wins (and fix this skill).

If the destination repo has no such runbook yet, the first migration should produce one; a per-root manifest is the only rollback path you will have.

Actor split (who can do what)

| Action | Actor | | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | Edit .tf / backend key, open PRs, drive CI, fence source, fill manifest | dev (you) | | Backend state object ops (head-object / copy-object / snapshot) + hand-apply bootstrap/ + policy-simulation | admin credentials (a privileged SSO/role profile) — the dev role cannot touch backend state | | Go / no-go at every gate and before every state mutation | the human owner |

Check the admin profile is live before you rely on it (aws sts get-caller-identity --profile ); if expired, ask the owner to re-login. Never admin-merge (gh pr merge --admin) past branch protection on repos that enforce reviews or commit signatures.

Per-root loop

For each root, in order (PAUSE for the owner's go before any live mutation — fence toggle, bootstrap apply, copy-object, destination apply/merge, source retire):

1. Pick the next root by correctness, not effort. Prefer clean single-root

canaries first (no cross-state ownership, no secret-bearing state, no live prod-user dependency, doesn't own the backend bucket). Leave the cluster root and the backend-bucket root for last.

1. Pre-gates (below) — all green or STOP.
2. Fence the source apply path (see Fence strategy).
3. Drain in-flight source runs; confirm the source key's lock is released.
4. Snapshot (admin): source VersionId + ETag + length (head-object);

lineage + serial + instance count; config-dir + .terraform.lock.hcl hashes. Record every value in the manifest — the version IDs are the only rollback path.

Never read state with a bare terraform state pull | jq. An unfiltered jq is the identity filter: it prints the whole state document, including every plaintext sensitive attribute, to your terminal and into any CI log or shell history that captures it. Stream the object and project only the three scalars you need — this form is safe on every root, so use it as the default rather than remembering to switch on the secret-bearing ones:

``bash aws s3 cp "s3:///" - --profile \ | jq '{lineage, serial, instances: ([.resources[].instances | length] | add)}' ``

1. Copy the exact version (admin, after go). Gate 3 established the

destination key was absent, but that was a check at a point in time — between the gate and the copy, anything with write access could create it. Make the copy itself refuse to overwrite rather than trusting the earlier check:

``bash aws s3api copy-object --profile \ --copy-source "/?versionId=" \ --bucket --key "" \ --if-none-match '*' ``

--if-none-match '*' fails the request with PreconditionFailed if the destination key already exists, closing the time-of-check/time-of-use window. Record the VersionId the copy returns — post-apply the destination has a newer version, and rollback needs to name the exact one it is restoring from. Then verify dest lineage/serial/count/ETag EQUAL the recorded values. Any mismatch = STOP.

1. Add infra// to the destination (fresh worktree): byte-identical

.tf except the backend key; keep .terraform.lock.hcl. Open the PR → destination plan must be 0/0/0 (read the actual plan comment — "No changes" — not just a green check; plan exits 0 on a diff). Merge → sole-writer apply 0/0/0. Verify post-apply serial unchanged.

1. Observe, then retire the source .tf (tombstone; delete the source key

only after observed-green — versioning is the safety net). Finalize the manifest.

Pre-gates (the make-or-break part)

Gate 1 — CI policy for the root's services, hand-applied FIRST. The destination CI plan/apply roles need least-privilege permissions for exactly this root's services before its first plan/apply.

• Derive the action set from what the provider ACTUALLY calls, not from CRUD.

Enumerated CRUD is not enough — the AWS provider makes implicit read calls on every refresh that fail the first plan with AccessDenied if missing. The reliable way to enumerate them is a TF_LOG=trace plan under admin creds, then grep the signed requests for the action names.

Treat the specific call-level claims below as observations, not as a spec. Which implicit reads a refresh makes is provider-implementation behavior and changes across provider versions — the AWS authorization references (action names, ARN forms) are stable and authoritative, but "this resource's Read also calls X" is only true of the provider version you traced. Record the provider version alongside the grant in the manifest, and re-trace rather than re-use the list after a provider major bump. Three classes bite repeatedly:

• Tag reads — but check HOW the service returns tags first. If the root's

provider.tf sets default_tags (or a resource sets tags), the mutating side (TagResource/UntagResource) is needed on the apply role for any tag change. For the READ side it depends on the service:

• Services whose Read/Describe does not return tags make a separate

tag-read call on every refresh via the transparent-tagging interceptor. Miss it and the first plan AccessDenieds. Grant the read to both roles.

• Services whose Read/Describe returns tags inline make no separate

call — e.g. secretsmanager:DescribeSecret returns Tags in its response, so default_tags adds no read action. Do not invent a GetTags-style grant it will never use.

• **The tag-read VERB SPELLING is per-service — a plausible guess grants

nothing**, because IAM silently no-ops an action name that doesn't exist for that service. These are all real, all different, and all mean "read this resource's tags":

| Service | Tag-read action | | ------------------- | --------------------- | | CloudWatch (alarms) | ListTagsForResource | | CloudWatch Logs | ListTagsForResource | | RDS | ListTagsForResource | | S3 Control | ListTagsForResource | | Glue | GetTags | | DynamoDB | ListTagsOfResource |

Look the verb up in the service authorization reference per namespace; don't pattern-match it from the last root. Note CloudWatch and CloudWatch Logs are different services that happen to share a verb spelling — granting one is not granting the other.

• Sub-resource reads the provider makes unconditionally. A resource's Read

can fan out to describe/list calls for sub-resources that don't exist in your configuration — e.g. Glue's GetPartitionIndexes fires on every aws_glue_catalog_table read even when no partition index is defined. These never appear in a CRUD-derived action list.

• **Encryption-at-rest pulls in a KMS read even when the root declares no KMS

resource. A resource encrypted with SSE-KMS resolves its key during Read, so the refresh calls kms:DescribeKey — including for AWS-managed** keys (e.g. the aws/dynamodb key behind an SSE-KMS table). Grant kms:DescribeKey on that key. Note this is strictly the metadata read: kms:Decrypt stays absent, so CI still cannot read the encrypted data.

• Cross-check the currently-working source role. The source CI role that plans

this root today already holds the exact permission set a working plan needs (often a broad :Get*/List*). Pull its policy and make sure your tightened enumerated grant covers everything its wildcard would.

• Policy-simulate under the real roles. aws iam simulate-custom-policy (or

simulate-principal-policy post-apply) — assert: plan reads allowed / plan mutations implicit-deny (read-only preserved); apply in-scope mutations allowed / out-of-scope resources implicit-deny; any trust-anchor protect-deny still explicit-deny. A green admin plan does NOT prove the CI role's scoping.

• KMS stays hand-applied in bootstrap/ — CI gets zero KMS-create.
• The bootstrap/ root is the trust anchor: hand-applied by admin, never CI.

Verify its plan is only your intended statements (e.g. 0 add / 2 change / 0 destroy = the two inline role policies), no trust change, no drift.

Gate 2 — source 0/0/0 vs live, at a pinned freeze SHA (source main tip). The migration zero-diff gate can't tell expected-state from un-captured drift, so reconcile any drift into source first. If the source drift-detection workflow is fenced/disabled, either temporarily re-enable it, dispatch a single read-only plan for this root, confirm "No changes", and re-disable; or run terraform plan locally under admin creds. Record the run/evidence.

Gate 3 — destination key absent (head-object → 404). This is a point-in-time check, not a guarantee: pair it with --if-none-match '*' on the copy (step 6) so the write itself is what enforces non-overwrite.

Gate 4 — backend bucket versioning is Enabled.

aws s3api get-bucket-versioning --bucket  --profile  --query Status

Every rollback instruction in this document assumes a prior object version can be retrieved by ID. If versioning is Suspended or absent, there is no rollback path and the whole procedure is a one-way door — stop and fix that before touching any state. Confirm the value; do not infer it from the bucket having been created by a module that usually enables it.

Hard rules / gotchas

• Verify the root's ACTUAL resources — trust neither its name nor its README.

Grep the .tf for ^resource/^data; a root can be misnamed (e.g. one called "Athena" that declares only Glue catalog resources) and its README can describe an aspirational design that was never built. Scope the IAM grant to what the code declares, confirmed against live resource names. Also check for a for_each/module that inflates one ^resource block into many live instances (e.g. a "VPC" root that is one aws_vpc_endpoint block plus a terraform-aws-modules/vpc/aws module = subnets/NAT/IGW/EIP/route-tables/flow-log role) — the instance count and the policy surface follow the live state, not the block count.

• Scope to exact resource ARNs when the naming namespace is shared. If the root

owns only some resources under a name prefix and other, unmanaged resources share that prefix, a /* grant over-reaches (worst case: the apply role could delete an unmanaged prod secret/bucket/queue). Enumerate the exact ARNs the root declares (list them live and diff against the .tf), not the prefix. Shape of the problem: a Secrets Manager root declares some secrets under per-environment prefixes, but those same prefixes also hold datastore and third-party secrets it does not manage — so the grant must enumerate one exact secret:-* ARN per declared secret (the -* matches Secrets Manager's random 6-char suffix), and a policy-simulation asserting that an unmanaged same-prefix secret is DENIED is what proves no leak. When the root does own a whole dedicated namespace with nothing else in it, a /* grant is fine.

• Implicit reads with irregular ARN forms + multi-resource auth. Some services'

refresh reads authorize against ARN shapes you won't guess, and against several resource types at once — get either wrong and the first plan AccessDenieds.

• Identity Store ARNs are irregular. identitystore:GetGroupId/DescribeGroup

authorize against Identitystore = arn:aws:identitystore:::identitystore/ (empty region, carries the account) OR Group = arn:aws:identitystore:::group/ (no region, no account, no store-id in the path). Grant both forms; the "standard" arn:aws:identitystore:::… shape denies.

• Multi-resource actions need every authorizing ARN listed.

sso:ListAccountAssignments authorizes against Instance / Account / PermissionSet together — list all three (instance/…, account/, permissionSet/…) or the refresh may deny depending on which the service evaluates.

• simulate-custom-policy can give a false verdict here: a deny-scoping

assertion ("the other permission set must deny") only holds once the co-authorizing ARN (e.g. the account/) is also present in the policy under test — omit it and the simulator's implicit-deny can flip. Treat post-apply simulate-principal-policy against the live role as authoritative; use simulate-custom-policy only as a pre-apply sketch, always with every co-authorizing ARN present. Two mechanics: it rejects --policy-input-list file://… ("invalid content") — pass the JSON inline via "$(cat file)" — and it enforces a per-member length limit, so simulate a trimmed slice of the relevant statements for a large policy.

• **explicitDeny vs implicitDeny mean different things — assert the one you

actually expect per role, never "either". explicitDeny means an applicable Deny statement matched, whether or not an Allow also matched. implicitDeny means nothing matched at all: no Allow and no Deny. They both block the call today, which is why it is tempting to accept either — but they are not interchangeable evidence. implicitDeny says only "this role was never granted the action", which stops being true the moment someone widens an unrelated wildcard; explicitDeny says "a protection statement is in force here". An assertion written to accept either would report a missing protection policy** as a pass.

So when the same assertion returns explicitDeny on the apply role and implicitDeny on the plan role, do not wave it through as a quirk — that is telling you the Deny is present on one role's policy and absent from the other's. Decide deliberately which roles must carry the Deny (for a trust anchor, normally both), assert explicitDeny on each of thos

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: loomantix
• Source: loomantix/claude-platform
• License: Apache-2.0
• Homepage: https://github.com/loomantix/claude-platform

Install and usage instructions live in the source repository linked above.