# Midnight Ops Doctor

> Diagnostic playbook for Midnight Network backend ops. Triggers on wallet sync stalls (appliedIndex, isStrictlyComplete, Custom error 139), deploy hangs (deployContract, watchForTxData), three-address confusion (Zswap coinPublicKey, NightExternal bech32m, Dust publicKey), DUST zero-balance and registerNightUtxosForDustGeneration, cloud-IP RPC 403 from AWS ELB on GCP/DO/AWS, AWS WAF 8KB submitTx (u…

- **Type:** Skill
- **Install:** `agentstack add skill-samuelarogbonlo-midnight-ops-doctor-midnight-ops-doctor`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [samuelarogbonlo](https://agentstack.voostack.com/s/samuelarogbonlo)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [samuelarogbonlo](https://github.com/samuelarogbonlo)
- **Source:** https://github.com/samuelarogbonlo/midnight-ops-doctor

## Install

```sh
agentstack add skill-samuelarogbonlo-midnight-ops-doctor-midnight-ops-doctor
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# midnight-ops-doctor

Notes from running a Midnight Network backend in production. Look up your symptom in the triage table, follow the link to the runbook.

---

## Triage

Match the user's words against the left column. Load only the doc on the right. Don't pre-load others.

| Dev says... | Load |
|---|---|
| "stuck at X%", "wallet not syncing", "appliedIndex", "isStrictlyComplete" | `references/wallet-lifecycle.md` § Sync stalls |
| "Custom error: 139", "transaction rejected by node", "Invalid Transaction" | `references/symptom-catalog.md` § err-139 |
| "deployContract hangs", "watchForTxData never returns", "deploy timed out" | `references/wallet-lifecycle.md` § Six-phase deploy |
| "403", "WAF", "ELB", "blocked from my server", "GCP/AWS/DO + Midnight" | `references/network-chooser.md` § Cloud-IP block |
| "wrong address", "faucet didn't arrive", "balance shows 0", "which address" | `references/three-addresses.md` |
| "DUST", "NIGHT registration", "dust generation", "dustReceiverAddress", "tDUST" | `references/dust-night-registration.md` |
| "verifier returns false", "vk mismatch", "Groth16 deploy verification" | run `scripts/deploy-verifier.mjs`; for incident response read `references/groth16-vk-mismatch.md` |
| "persistentHash", "SHA256 doesn't match", "hashlock mismatch", "cross-family" | `references/cross-family-hashlocks.md` |
| "ERR_UNSUPPORTED_DIR_IMPORT", "ESM in CJS", "@midnight-ntwrk import fails" | `references/symptom-catalog.md` § cjs-esm |
| "submitTx timeout", "tx >8KB fails", "AWS WAF on RPC" | `references/symptom-catalog.md` § waf-8kb |
| "snapshot won't restore", "GCS scope", "wallet warm-restart broken" | `references/symptom-catalog.md` § snapshot-gcs |
| "what proof-server version", "ledger-v7 vs v8", "SDK compat" | `references/version-matrix.md` |

Routing rules:
- One symptom, one doc. Never broadcast-load.
- If the user describes two symptoms, fix the one that blocks the other first (sync before deploy, network before sync).
- If no row matches, ask one clarifying question. Don't guess.

---

## Three addresses

The single most-mis-applied concept in Midnight backend code. Inline here so triage doesn't require loading a reference doc.

### One seed, three addresses

A single `MIDNIGHT_SEED` (32-byte or 64-byte hex) derives **three distinct addresses** through `HDWallet.fromSeed(seed).selectAccount(0).selectRoles([Roles.Zswap, Roles.NightExternal, Roles.Dust]).deriveKeysAt(0)`. The roles enum is exported by `@midnight-ntwrk/wallet-sdk-hd`. Each role yields a separate keypair with a different on-chain semantic.

```typescript
import { HDWallet, Roles } from '@midnight-ntwrk/wallet-sdk-hd';
import {
  ZswapSecretKeys,
  DustSecretKey,
} from '@midnight-ntwrk/ledger-v8';
import { createKeystore, PublicKey } from '@midnight-ntwrk/wallet-sdk-address-format';
import { MidnightBech32m, DustAddress } from '@midnight-ntwrk/wallet-sdk-address-format';

const seed = Buffer.from(process.env.MIDNIGHT_SEED!, 'hex');
const hd = HDWallet.fromSeed(new Uint8Array(seed));
if (hd.type !== 'seedOk') throw new Error(`HD seed failed: ${hd.type}`);

const derived = hd.hdWallet
  .selectAccount(0)
  .selectRoles([Roles.Zswap, Roles.NightExternal, Roles.Dust] as const)
  .deriveKeysAt(0);
if (derived.type !== 'keysDerived') throw new Error(`derive failed: ${derived.type}`);

const networkId = 'preview'; // or 'preprod' / 'mainnet'

const zswapKeys = ZswapSecretKeys.fromSeed(derived.keys[Roles.Zswap]);
const dustSecret = DustSecretKey.fromSeed(derived.keys[Roles.Dust]);
const unshieldedKs = createKeystore(derived.keys[Roles.NightExternal], networkId);

const shieldedCoinPublicKey = zswapKeys.coinPublicKey;                     // 32-byte hex
const unshieldedAddress = unshieldedKs.getBech32Address().toString();      // mn_addr_1...
const dustAddress = MidnightBech32m
  .encode(networkId, new DustAddress(dustSecret.publicKey))
  .toString();                                                             // mn_dust_1...

hd.hdWallet.clear();
```

### Address shapes

| Role | Format | Example |
|---|---|---|
| `Roles.Zswap` | 32-byte hex (no prefix in SDK; backend pads to `0x` + 64 hex) | `4f1c...3a9b` |
| `Roles.NightExternal` | bech32m | `mn_addr_preview1qwerty...` / `mn_addr_preprod1...` |
| `Roles.Dust` | bech32m | `mn_dust_preview1...` / `mn_dust_preprod1...` |

### Operation, address, why

| Operation | Use this address | Why |
|---|---|---|
| Faucet POST `address` field | unshielded bech32m | Faucet drops land in the unshielded UTXO set |
| Lace "Receive" tab display | unshielded bech32m | Lace's UI is unshielded-default |
| Indexer balance query | unshielded bech32m | Public chain state is keyed on unshielded |
| Native NIGHT transfer (send) | unshielded bech32m | NIGHT lives unshielded |
| Shielded zswap tx, `coinPublicKey` API params | shielded hex (Zswap) | Shielded ledger uses coin public keys |
| HTLC `sender` / `receiver` Bytes arg | shielded hex (Zswap) | HTLC contract takes raw 32-byte keys |
| DUST balance lookup, dust-credit recipient | dust bech32m | Dust accrual uses the dust public key |
| `dustReceiverAddress` arg to `registerNightUtxosForDustGeneration` | dust bech32m | Designation targets the dust address |

### The `getWalletAddress()` trap

Many wallet adapter wrappers expose a `getWalletAddress()` method that returns ONLY the shielded coin public key (32-byte hex padded to `0x` + 64 hex). Backend code that says "the wallet address" almost always means this one. It's correct for HTLC `sender`/`receiver` args, and wrong for everything balance-related (faucet, Lace, indexer).

When in doubt, expose all three on a diagnostics endpoint and pick by use case:

```typescript
{
  shieldedCoinPublicKey: '0x4f1c...3a9b',
  unshieldedAddress: 'mn_addr_preview1...',
  dustAddress: 'mn_dust_preview1...',
}
```

### Seed normalization gotcha

A 24-word BIP39 mnemonic can be normalized into a seed three ways. Only one matches Lace.

| Normalization | Bytes | Matches Lace? |
|---|---|---|
| `bip39.mnemonicToSeedSync(mnemonic, '')` (PBKDF2 full) | 64 | YES |
| First 32 of PBKDF2 output | 32 | NO |
| `bip39.mnemonicToEntropy()` (BIP39 entropy) | 32 | NO |

If your derived addresses don't match what Lace shows, you almost certainly used the wrong normalization. The bundled `scripts/address-derive.mjs` runs the canonical PBKDF2-full path.

For deeper diagnostics (faucet didn't arrive, balance still zero, address mismatch), load `references/three-addresses.md`.

---

## Quick fixes

Top twelve errors. One-line diagnosis, minimum-viable fix. If the fix doesn't stick, escalate to the deeper doc.

### `Custom error: 139`

Diagnosis: wallet submitted a transaction before chain sync completed. Node rejected stale UTXO inputs.

Fix:
```typescript
await waitForWalletSyncState(WALLET_SYNC_TIMEOUT_MS, 'startup');
// only then: submitTx, deploy, lock, etc.
```

Deeper: `references/wallet-lifecycle.md` § Sync-completion check.

### `ERR_UNSUPPORTED_DIR_IMPORT` from `@midnight-ntwrk/*`

Diagnosis: `@midnight-ntwrk/*` packages are ESM-only. CJS `require()` fails on bare-directory imports.

Fix:
```typescript
// In a CJS file, use type-only static imports + dynamic import for runtime.
import type { WalletFacade } from '@midnight-ntwrk/wallet-sdk-facade';
const { WalletFacade } = await import('@midnight-ntwrk/wallet-sdk-facade');
```

Deeper: `references/symptom-catalog.md` § cjs-esm.

### `submitTx` times out around 25-30s, tx is ~90 KB

Diagnosis: the SDK's HTTP `submitTx` posts to RPC. AWS WAF rejects bodies >8 KB. Deploy txs are typically 50-100 KB.

Fix: route submission through `wallet.submitTransaction(tx)`. That uses the WebSocket relay (PolkadotNodeClient), bypassing the HTTP body limit.

```typescript
const midnightProvider = {
  async submitTx(tx) { return wallet.submitTransaction(tx); },
};
```

Deeper: `references/symptom-catalog.md` § waf-8kb.

### `403 Forbidden` from `https://rpc.preprod.midnight.network` (only on cloud VMs)

Diagnosis: AWS ELB on the preprod RPC blocks cloud-provider IP ranges (GCP, DO, AWS, etc.). `server: awselb/2.0` in the response headers confirms it. Indexer is unblocked. Only RPC is.

Fix: prefer **preview** network (typically reachable from cloud VMs; verify with `scripts/rpc-reachability-probe.mjs`) or run a Cloudflare Worker reverse-proxy from `assets/cloudflare-worker-template/`.

```bash
# preview env
MIDNIGHT_NETWORK_ID=preview
MIDNIGHT_NODE_RPC=wss://rpc.preview.midnight.network
MIDNIGHT_INDEXER_URL=https://indexer.preview.midnight.network/api/v3/graphql
```

Deeper: `references/network-chooser.md` § Cloud-IP block.

### `deployContract()` hangs forever (no logs, no progress)

Diagnosis: the SDK's `deployContract()` calls `watchForTxData()` internally with no timeout. If the tx never lands on-chain, it waits indefinitely.

Fix: don't use `deployContract()`. Use the six-phase manual flow with explicit timeouts on each phase.

```typescript
const unproven = await createUnprovenDeployTx(providers, { compiledContract, ... });
const proven = await providers.proofProvider.proveTx(unproven.private.unprovenTx);
const balanced = await walletProvider.balanceTx(proven);
const txId = await wallet.submitTransaction(balanced);
const finalized = await Promise.race([
  publicDataProvider.watchForTxData(txId),
  new Promise((_, r) => setTimeout(() => r(new Error('confirm timeout')), 90_000)),
]);
```

Deeper: `references/wallet-lifecycle.md` § Six-phase deploy.

### Faucet POST succeeded but balance is still zero

Diagnosis: faucet was sent to the wrong address (probably the shielded hex from `getWalletAddress()` instead of the unshielded bech32m).

Fix:
```bash
curl -H "x-midnight-sidecar-token: $TOKEN" http://127.0.0.1:8090/wallet/diagnostics | jq
# verify addresses.unshieldedAddress matches the address you sent to
```

Deeper: `references/three-addresses.md` § Diagnostic recipes.

### Wallet sync stuck >30 min on a fresh container

Diagnosis: cold sync on preprod is genuinely slow. Linear in chain depth, no birthday/fast-sync primitive. Preview is faster (~20 min).

Fix: wait. If still stuck after 60 min, check indexer reachability and the dust sub-wallet specifically. Dust is usually the bottleneck.

Deeper: `references/wallet-lifecycle.md` § Cold-sync expectations.

### Dust balance is zero after sync completes

Diagnosis: the wallet has NIGHT but `registerNightUtxosForDustGeneration` was never called. There is no programmatic faucet for DUST.

Fix: open the same seed/mnemonic in **Lace wallet** → Midnight tab → tNIGHT Designation. Lace can run the registration even with zero existing dust (works on preview, preprod is intermittent). Backend inherits the on-chain designation automatically; first dust appears in ~90 seconds.

Deeper: `references/dust-night-registration.md`.

### `persistentHash` output doesn't match Node `crypto.createHash('sha256')`

Diagnosis: you called `persistentHash(rawBytes)` instead of `persistentHash(new CompactTypeBytes(32), rawBytes)`. Compact's persistent hash requires a type tag. Without it the hash is a different domain.

Fix:
```typescript
import { persistentHash, CompactTypeBytes } from '@midnight-ntwrk/compact-runtime';
const bytesType = new CompactTypeBytes(32);
const hashlock = persistentHash(bytesType, preimage);  // matches sha256(preimage)
```

Deeper: `references/cross-family-hashlocks.md`.

### Snapshot upload fails with "Provided scope(s) are not authorized"

Diagnosis: GCE VM's default service account scopes include `devstorage.read_only` but not `read_write`. Local snapshot still works. Only GCS upload fails.

Fix:
```bash
gcloud compute instances stop  --zone=
gcloud compute instances set-service-account  \
  --zone= \
  --scopes=devstorage.read_write,logging-write,monitoring-write
gcloud compute instances start  --zone=
```

Deeper: `references/symptom-catalog.md` § snapshot-gcs.

### Proof-server returns "version mismatch", or proofs verify locally but reject on-chain

Diagnosis: the proof-server image version, the ledger version (`ledger-v7` vs `ledger-v8`), and the on-chain Groth16 verifier's vk all need to match. One drift gives silent verification failure.

Fix: check `references/version-matrix.md` for the pinned compat row. Re-run `scripts/deploy-verifier.mjs` to confirm the on-chain vk matches your local zkey.

Deeper: `references/version-matrix.md`, `references/symptom-catalog.md` § err-vk-mismatch, and `references/groth16-vk-mismatch.md` (incident playbook if the script reports vk byte-equality FAIL).

### Lace shows "Failed to clone intent" when signing

Diagnosis: Lace wallet state race. Usually multiple tabs or a pending intent from a prior session.

Fix: close all Lace tabs except one, wait until Lace shows "Idle" status, retry. Not a protocol-level error.

Deeper: `references/symptom-catalog.md` § lace-clone-failure.

---

## Reference docs at a glance

Brief overviews. Load the matching reference for full runbooks.

### Three addresses (`references/three-addresses.md`)

One seed gives Zswap shielded hex, NightExternal unshielded bech32m, and Dust bech32m. `getWalletAddress()` returns only the shielded hex; backend code referring to "the address" is usually wrong for unshielded ops. Faucet drops, Lace Receive, indexer queries all use the unshielded bech32m. Load this when balance/faucet/which-address questions come up.

### Wallet lifecycle (`references/wallet-lifecycle.md`)

`WalletFacade.init()` builds the facade; `wallet.start(shieldedSecretKeys, dustSecretKey)` is a separate explicit step in SDK 3.x. Sync completion requires `appliedId >= highestTransactionId` AND `isStrictlyComplete: true` per sub-wallet. Submitting before that yields `Custom error: 139`. Deploy must use the six-phase manual flow because the SDK's `deployContract()` hangs forever on `watchForTxData`. Load this for any sync, deploy, submit, or warm-restart issue.

### Network chooser (`references/network-chooser.md`)

Preview vs preprod vs local-playground vs mainnet selection. Cloud VMs (GCP/DO/AWS) hit `awselb/2.0` 403 on preprod RPC. Preview is typically reachable from cloud VMs (verify with `scripts/rpc-reachability-probe.mjs` before relying on it). Cloudflare Worker reverse-proxy in `assets/cloudflare-worker-template/` is the workaround for preprod-only deployments. Load this for any "blocked from my server" or "which network should I use" question.

### DUST + NIGHT registration (`references/dust-night-registration.md`)

DUST is not transferable. Only redirectable via `dustReceiverAddress` during `registerNightUtxosForDustGeneration`. Fresh wallets bootstrap dust through Lace's tNIGHT Designation flow. About 12 hours for full ramp, ~90s for first dust to appear. Load this for "DUST balance zero", "how do I send DUST", or "registerNightUtxosForDustGeneration fails with InsufficientDust" questions.

### Cross-family hashlocks (`references/cross-family-hashlocks.md`)

Midnight's `persistentHash(CompactTypeBytes(32), bytes)` matches Node SHA-256 byte-for-byte ONLY with the type tag. Cross-family swaps (Midnight ↔ EVM SHA-256 corridor) require both sides compute the same hashlock from the same preimage. The bundled self-test (`scripts/persistent-hash-self-test.mjs`) confirms parity. Load this when hashlock parity, EVM/Midnight preimage matching, or persistentHash output is in question.

---

## The deploy verifier

`scripts/deploy-verifier.mjs` is the post-deploy smoke test. Run it after every Midnight HTLC deploy and before any go-live cutover.

### When to run

- Immediately after the deploy script reports a contract address.
- Before pointing the backend at a new contract address.
- After a network migration (preprod ↔ preview).
- As a CI gate before promoting an env file.

### Invocation

```bash
node scripts/deploy-verifier.mjs assets/deploy-manifest.example.json
```

The manifest shape is documented inline in `assets/deploy-manifest.example.json`. Required fields: `contractAddress`, `networkId`, `indexerUrl`, `nodeRpcUrl`, `expectedZkeyDige

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [samuelarogbonlo](https://github.com/samuelarogbonlo)
- **Source:** [samuelarogbonlo/midnight-ops-doctor](https://github.com/samuelarogbonlo/midnight-ops-doctor)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-samuelarogbonlo-midnight-ops-doctor-midnight-ops-doctor
- Seller: https://agentstack.voostack.com/s/samuelarogbonlo
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.