Midnight Ops Doctor

midnight-ops-doctor

Notes from running a Midnight Network backend in production. Look up your symptom in the triage table, follow the link to the runbook.

---

Triage

Match the user's words against the left column. Load only the doc on the right. Don't pre-load others.

| Dev says... | Load | |---|---| | "stuck at X%", "wallet not syncing", "appliedIndex", "isStrictlyComplete" | references/wallet-lifecycle.md § Sync stalls | | "Custom error: 139", "transaction rejected by node", "Invalid Transaction" | references/symptom-catalog.md § err-139 | | "deployContract hangs", "watchForTxData never returns", "deploy timed out" | references/wallet-lifecycle.md § Six-phase deploy | | "403", "WAF", "ELB", "blocked from my server", "GCP/AWS/DO + Midnight" | references/network-chooser.md § Cloud-IP block | | "wrong address", "faucet didn't arrive", "balance shows 0", "which address" | references/three-addresses.md | | "DUST", "NIGHT registration", "dust generation", "dustReceiverAddress", "tDUST" | references/dust-night-registration.md | | "verifier returns false", "vk mismatch", "Groth16 deploy verification" | run scripts/deploy-verifier.mjs; for incident response read references/groth16-vk-mismatch.md | | "persistentHash", "SHA256 doesn't match", "hashlock mismatch", "cross-family" | references/cross-family-hashlocks.md | | "ERRUNSUPPORTEDDIR_IMPORT", "ESM in CJS", "@midnight-ntwrk import fails" | references/symptom-catalog.md § cjs-esm | | "submitTx timeout", "tx >8KB fails", "AWS WAF on RPC" | references/symptom-catalog.md § waf-8kb | | "snapshot won't restore", "GCS scope", "wallet warm-restart broken" | references/symptom-catalog.md § snapshot-gcs | | "what proof-server version", "ledger-v7 vs v8", "SDK compat" | references/version-matrix.md |

Routing rules:

• One symptom, one doc. Never broadcast-load.
• If the user describes two symptoms, fix the one that blocks the other first (sync before deploy, network before sync).
• If no row matches, ask one clarifying question. Don't guess.

---

Three addresses

The single most-mis-applied concept in Midnight backend code. Inline here so triage doesn't require loading a reference doc.

One seed, three addresses

A single MIDNIGHT_SEED (32-byte or 64-byte hex) derives three distinct addresses through HDWallet.fromSeed(seed).selectAccount(0).selectRoles([Roles.Zswap, Roles.NightExternal, Roles.Dust]).deriveKeysAt(0). The roles enum is exported by @midnight-ntwrk/wallet-sdk-hd. Each role yields a separate keypair with a different on-chain semantic.

import { HDWallet, Roles } from '@midnight-ntwrk/wallet-sdk-hd';
import {
  ZswapSecretKeys,
  DustSecretKey,
} from '@midnight-ntwrk/ledger-v8';
import { createKeystore, PublicKey } from '@midnight-ntwrk/wallet-sdk-address-format';
import { MidnightBech32m, DustAddress } from '@midnight-ntwrk/wallet-sdk-address-format';

const seed = Buffer.from(process.env.MIDNIGHT_SEED!, 'hex');
const hd = HDWallet.fromSeed(new Uint8Array(seed));
if (hd.type !== 'seedOk') throw new Error(`HD seed failed: ${hd.type}`);

const derived = hd.hdWallet
  .selectAccount(0)
  .selectRoles([Roles.Zswap, Roles.NightExternal, Roles.Dust] as const)
  .deriveKeysAt(0);
if (derived.type !== 'keysDerived') throw new Error(`derive failed: ${derived.type}`);

const networkId = 'preview'; // or 'preprod' / 'mainnet'

const zswapKeys = ZswapSecretKeys.fromSeed(derived.keys[Roles.Zswap]);
const dustSecret = DustSecretKey.fromSeed(derived.keys[Roles.Dust]);
const unshieldedKs = createKeystore(derived.keys[Roles.NightExternal], networkId);

const shieldedCoinPublicKey = zswapKeys.coinPublicKey;                     // 32-byte hex
const unshieldedAddress = unshieldedKs.getBech32Address().toString();      // mn_addr_1...
const dustAddress = MidnightBech32m
  .encode(networkId, new DustAddress(dustSecret.publicKey))
  .toString();                                                             // mn_dust_1...

hd.hdWallet.clear();

Address shapes

| Role | Format | Example | |---|---|---| | Roles.Zswap | 32-byte hex (no prefix in SDK; backend pads to 0x + 64 hex) | 4f1c...3a9b | | Roles.NightExternal | bech32m | mn_addr_preview1qwerty... / mn_addr_preprod1... | | Roles.Dust | bech32m | mn_dust_preview1... / mn_dust_preprod1... |

Operation, address, why

| Operation | Use this address | Why | |---|---|---| | Faucet POST address field | unshielded bech32m | Faucet drops land in the unshielded UTXO set | | Lace "Receive" tab display | unshielded bech32m | Lace's UI is unshielded-default | | Indexer balance query | unshielded bech32m | Public chain state is keyed on unshielded | | Native NIGHT transfer (send) | unshielded bech32m | NIGHT lives unshielded | | Shielded zswap tx, coinPublicKey API params | shielded hex (Zswap) | Shielded ledger uses coin public keys | | HTLC sender / receiver Bytes arg | shielded hex (Zswap) | HTLC contract takes raw 32-byte keys | | DUST balance lookup, dust-credit recipient | dust bech32m | Dust accrual uses the dust public key | | dustReceiverAddress arg to registerNightUtxosForDustGeneration | dust bech32m | Designation targets the dust address |

The getWalletAddress() trap

Many wallet adapter wrappers expose a getWalletAddress() method that returns ONLY the shielded coin public key (32-byte hex padded to 0x + 64 hex). Backend code that says "the wallet address" almost always means this one. It's correct for HTLC sender/receiver args, and wrong for everything balance-related (faucet, Lace, indexer).

When in doubt, expose all three on a diagnostics endpoint and pick by use case:

{
  shieldedCoinPublicKey: '0x4f1c...3a9b',
  unshieldedAddress: 'mn_addr_preview1...',
  dustAddress: 'mn_dust_preview1...',
}

Seed normalization gotcha

A 24-word BIP39 mnemonic can be normalized into a seed three ways. Only one matches Lace.

| Normalization | Bytes | Matches Lace? | |---|---|---| | bip39.mnemonicToSeedSync(mnemonic, '') (PBKDF2 full) | 64 | YES | | First 32 of PBKDF2 output | 32 | NO | | bip39.mnemonicToEntropy() (BIP39 entropy) | 32 | NO |

If your derived addresses don't match what Lace shows, you almost certainly used the wrong normalization. The bundled scripts/address-derive.mjs runs the canonical PBKDF2-full path.

For deeper diagnostics (faucet didn't arrive, balance still zero, address mismatch), load references/three-addresses.md.

---

Quick fixes

Top twelve errors. One-line diagnosis, minimum-viable fix. If the fix doesn't stick, escalate to the deeper doc.

Custom error: 139

Diagnosis: wallet submitted a transaction before chain sync completed. Node rejected stale UTXO inputs.

Fix:

await waitForWalletSyncState(WALLET_SYNC_TIMEOUT_MS, 'startup');
// only then: submitTx, deploy, lock, etc.

Deeper: references/wallet-lifecycle.md § Sync-completion check.

ERR_UNSUPPORTED_DIR_IMPORT from @midnight-ntwrk/*

Diagnosis: @midnight-ntwrk/* packages are ESM-only. CJS require() fails on bare-directory imports.

Fix:

// In a CJS file, use type-only static imports + dynamic import for runtime.
import type { WalletFacade } from '@midnight-ntwrk/wallet-sdk-facade';
const { WalletFacade } = await import('@midnight-ntwrk/wallet-sdk-facade');

Deeper: references/symptom-catalog.md § cjs-esm.

submitTx times out around 25-30s, tx is ~90 KB

Diagnosis: the SDK's HTTP submitTx posts to RPC. AWS WAF rejects bodies >8 KB. Deploy txs are typically 50-100 KB.

Fix: route submission through wallet.submitTransaction(tx). That uses the WebSocket relay (PolkadotNodeClient), bypassing the HTTP body limit.

const midnightProvider = {
  async submitTx(tx) { return wallet.submitTransaction(tx); },
};

Deeper: references/symptom-catalog.md § waf-8kb.

403 Forbidden from https://rpc.preprod.midnight.network (only on cloud VMs)

Diagnosis: AWS ELB on the preprod RPC blocks cloud-provider IP ranges (GCP, DO, AWS, etc.). server: awselb/2.0 in the response headers confirms it. Indexer is unblocked. Only RPC is.

Fix: prefer preview network (typically reachable from cloud VMs; verify with scripts/rpc-reachability-probe.mjs) or run a Cloudflare Worker reverse-proxy from assets/cloudflare-worker-template/.

# preview env
MIDNIGHT_NETWORK_ID=preview
MIDNIGHT_NODE_RPC=wss://rpc.preview.midnight.network
MIDNIGHT_INDEXER_URL=https://indexer.preview.midnight.network/api/v3/graphql

Deeper: references/network-chooser.md § Cloud-IP block.

deployContract() hangs forever (no logs, no progress)

Diagnosis: the SDK's deployContract() calls watchForTxData() internally with no timeout. If the tx never lands on-chain, it waits indefinitely.

Fix: don't use deployContract(). Use the six-phase manual flow with explicit timeouts on each phase.

const unproven = await createUnprovenDeployTx(providers, { compiledContract, ... });
const proven = await providers.proofProvider.proveTx(unproven.private.unprovenTx);
const balanced = await walletProvider.balanceTx(proven);
const txId = await wallet.submitTransaction(balanced);
const finalized = await Promise.race([
  publicDataProvider.watchForTxData(txId),
  new Promise((_, r) => setTimeout(() => r(new Error('confirm timeout')), 90_000)),
]);

Deeper: references/wallet-lifecycle.md § Six-phase deploy.

Faucet POST succeeded but balance is still zero

Diagnosis: faucet was sent to the wrong address (probably the shielded hex from getWalletAddress() instead of the unshielded bech32m).

Fix:

curl -H "x-midnight-sidecar-token: $TOKEN" http://127.0.0.1:8090/wallet/diagnostics | jq
# verify addresses.unshieldedAddress matches the address you sent to

Deeper: references/three-addresses.md § Diagnostic recipes.

Wallet sync stuck >30 min on a fresh container

Diagnosis: cold sync on preprod is genuinely slow. Linear in chain depth, no birthday/fast-sync primitive. Preview is faster (~20 min).

Fix: wait. If still stuck after 60 min, check indexer reachability and the dust sub-wallet specifically. Dust is usually the bottleneck.

Deeper: references/wallet-lifecycle.md § Cold-sync expectations.

Dust balance is zero after sync completes

Diagnosis: the wallet has NIGHT but registerNightUtxosForDustGeneration was never called. There is no programmatic faucet for DUST.

Fix: open the same seed/mnemonic in Lace wallet → Midnight tab → tNIGHT Designation. Lace can run the registration even with zero existing dust (works on preview, preprod is intermittent). Backend inherits the on-chain designation automatically; first dust appears in ~90 seconds.

Deeper: references/dust-night-registration.md.

persistentHash output doesn't match Node crypto.createHash('sha256')

Diagnosis: you called persistentHash(rawBytes) instead of persistentHash(new CompactTypeBytes(32), rawBytes). Compact's persistent hash requires a type tag. Without it the hash is a different domain.

Fix:

import { persistentHash, CompactTypeBytes } from '@midnight-ntwrk/compact-runtime';
const bytesType = new CompactTypeBytes(32);
const hashlock = persistentHash(bytesType, preimage);  // matches sha256(preimage)

Deeper: references/cross-family-hashlocks.md.

Snapshot upload fails with "Provided scope(s) are not authorized"

Diagnosis: GCE VM's default service account scopes include devstorage.read_only but not read_write. Local snapshot still works. Only GCS upload fails.

Fix:

gcloud compute instances stop  --zone=
gcloud compute instances set-service-account  \
  --zone= \
  --scopes=devstorage.read_write,logging-write,monitoring-write
gcloud compute instances start  --zone=

Deeper: references/symptom-catalog.md § snapshot-gcs.

Proof-server returns "version mismatch", or proofs verify locally but reject on-chain

Diagnosis: the proof-server image version, the ledger version (ledger-v7 vs ledger-v8), and the on-chain Groth16 verifier's vk all need to match. One drift gives silent verification failure.

Fix: check references/version-matrix.md for the pinned compat row. Re-run scripts/deploy-verifier.mjs to confirm the on-chain vk matches your local zkey.

Deeper: references/version-matrix.md, references/symptom-catalog.md § err-vk-mismatch, and references/groth16-vk-mismatch.md (incident playbook if the script reports vk byte-equality FAIL).

Lace shows "Failed to clone intent" when signing

Diagnosis: Lace wallet state race. Usually multiple tabs or a pending intent from a prior session.

Fix: close all Lace tabs except one, wait until Lace shows "Idle" status, retry. Not a protocol-level error.

Deeper: references/symptom-catalog.md § lace-clone-failure.

---

Reference docs at a glance

Brief overviews. Load the matching reference for full runbooks.

Three addresses (references/three-addresses.md)

One seed gives Zswap shielded hex, NightExternal unshielded bech32m, and Dust bech32m. getWalletAddress() returns only the shielded hex; backend code referring to "the address" is usually wrong for unshielded ops. Faucet drops, Lace Receive, indexer queries all use the unshielded bech32m. Load this when balance/faucet/which-address questions come up.

Wallet lifecycle (references/wallet-lifecycle.md)

WalletFacade.init() builds the facade; wallet.start(shieldedSecretKeys, dustSecretKey) is a separate explicit step in SDK 3.x. Sync completion requires appliedId >= highestTransactionId AND isStrictlyComplete: true per sub-wallet. Submitting before that yields Custom error: 139. Deploy must use the six-phase manual flow because the SDK's deployContract() hangs forever on watchForTxData. Load this for any sync, deploy, submit, or warm-restart issue.

Network chooser (references/network-chooser.md)

Preview vs preprod vs local-playground vs mainnet selection. Cloud VMs (GCP/DO/AWS) hit awselb/2.0 403 on preprod RPC. Preview is typically reachable from cloud VMs (verify with scripts/rpc-reachability-probe.mjs before relying on it). Cloudflare Worker reverse-proxy in assets/cloudflare-worker-template/ is the workaround for preprod-only deployments. Load this for any "blocked from my server" or "which network should I use" question.

DUST + NIGHT registration (references/dust-night-registration.md)

DUST is not transferable. Only redirectable via dustReceiverAddress during registerNightUtxosForDustGeneration. Fresh wallets bootstrap dust through Lace's tNIGHT Designation flow. About 12 hours for full ramp, ~90s for first dust to appear. Load this for "DUST balance zero", "how do I send DUST", or "registerNightUtxosForDustGeneration fails with InsufficientDust" questions.

Cross-family hashlocks (references/cross-family-hashlocks.md)

Midnight's persistentHash(CompactTypeBytes(32), bytes) matches Node SHA-256 byte-for-byte ONLY with the type tag. Cross-family swaps (Midnight ↔ EVM SHA-256 corridor) require both sides compute the same hashlock from the same preimage. The bundled self-test (scripts/persistent-hash-self-test.mjs) confirms parity. Load this when hashlock parity, EVM/Midnight preimage matching, or persistentHash output is in question.

---

The deploy verifier

scripts/deploy-verifier.mjs is the post-deploy smoke test. Run it after every Midnight HTLC deploy and before any go-live cutover.

When to run

• Immediately after the deploy script reports a contract address.
• Before pointing the backend at a new contract address.
• After a network migration (preprod ↔ preview).
• As a CI gate before promoting an env file.

Invocation

node scripts/deploy-verifier.mjs assets/deploy-manifest.example.json

The manifest shape is documented inline in assets/deploy-manifest.example.json. Required fields: contractAddress, networkId, indexerUrl, nodeRpcUrl, `expectedZkeyDige

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: samuelarogbonlo
• Source: samuelarogbonlo/midnight-ops-doctor
• License: MIT

Install and usage instructions live in the source repository linked above.