Stable
Sourced from: docs/02-NODE-ARCHITECTURE.md §5 (crash recovery, boot sequence), docs/DECISIONS.md D1.8 D6.4 D8.7 D3.8, docs/10-SECURITY-SPEC.md §10

Incidents

This page covers the two things you actually need to know when something looks wrong: what your node fixes automatically, and what needs you.

The short version

  • Your node restarted or crashed? It reconciles everything against the live chain on its own at boot, before it even starts accepting new checkout traffic. In the overwhelming majority of cases you don't need to do anything — see "Crash recovery" below.
  • Something says NEEDS_ATTENTION? That's the one state where the node is explicitly asking for a human decision because it couldn't safely determine an outcome on its own. See "Reading NEEDS_ATTENTION" below.
  • Funds are never silently lost. A wallet only ever loses money when a transaction actually settles on-chain, and settlement only happens after your node's own signature is on it. The worst realistic outcome of a crash or an ambiguous gateway response is a wallet sitting locked and idle for a while — the money is still there, under a key your node still holds.

Reading NEEDS_ATTENTION

NEEDS_ATTENTION lives on individual sweep and outbound transfer rows — it is never a wallet-level state. The wallet involved stays locked (a sub-wallet stays SWEEPING, a treasury stays checked out) while the row sits in this state, which is exactly why it matters: that wallet can't participate in any other transfer until the row is resolved, one way or another.

Why a transfer lands here, specifically: your node never blind-retries a submission whose outcome is unclear (see treasury-operations.md for the full rule). When a submission's result can't be confidently determined — most commonly because the SplitChain gateway was unreachable at the moment your node tried to check — the node refuses to guess. It marks the row NEEDS_ATTENTION instead of either assuming success (risking a stuck wallet) or blindly retrying (risking a double-spend).

There are two flavors, and they resolve differently:

  • Gateway-unreachable. The node itself keeps re-checking this automatically, roughly once a minute, and heals itself the moment the gateway is reachable again — a transient network blip on either side usually clears within a minute or two with zero operator action.
  • Genuinely unclear even with the gateway reachable. This is the case that actually needs you. Open the sweep or outbound transfer detail from the admin dashboard; it carries the reconcile context the node captured when it flagged the row. From there:
    1. Check the wallet's real balance and last settled transaction directly (the dashboard shows both, sourced live from the gateway).
    2. If the transfer clearly did land (the destination balance reflects it), mark it resolved — nothing further to do, the funds are exactly where intended.
    3. If it clearly did not land, use the retry action — this re-runs the same reconcile-before-retry check your node runs automatically, so it's safe even if you're not certain; it will not double-submit.
    4. If you genuinely cannot tell, that's a case for support rather than a guess — don't force a retry on a row you can't confidently reason about.

You'll also get an admin alert the moment a sweep exhausts its automatic backoff retries, and for any transfer/wallet that enters NEEDS_ATTENTION — you don't have to go looking for these proactively, but it's worth knowing where to look (Sweeps / Outbound Transfers screens) if you want to check proactively during a known gateway incident.

Crash recovery at boot

Every time your node starts — first deploy, a routine restart, recovering from a crash — it runs a recovery pass before it starts any background engine work (sweeps, autoscale, webhook delivery) or accepts new payment traffic. This is fully automatic; there is no operator action here under normal circumstances. What it does, in order:

  1. Reconciles every wallet that was mid-transfer when the node went down — a sub-wallet that was mid-sweep, or a treasury wallet that was checked out for a sweep-target receive, a consolidation, or an outbound send. For each one, it checks the real chain state:
    • If the transfer actually landed, it finalizes it exactly as the normal success path would (records the settled balance and transaction, completes the sweep/outbound row).
    • If it provably did not land, the transfer is requeued and the wallet is freed (a sub-wallet goes back to FUNDS_HELD, a treasury's checkout is released).
    • If it's ambiguous, the row is marked NEEDS_ATTENTION and the wallet stays locked — same as the ambiguous case above, just discovered at boot instead of mid-operation.
  2. Re-syncs balances for any wallet it just unstranded, straight from the gateway, before freeing it for new work.
  3. Releases stale checkouts — any treasury or sub-wallet lock whose owning sweep/outbound/session already reached a terminal state (settled or failed) gets its lock cleared, even if the lock itself somehow survived the crash.
  4. Leaves funded-but-unswept sub-wallets alone — a sub-wallet sitting in FUNDS_HELD is safe (the funds are under a key your node controls); the normal sweep process just picks it up on its next pass. Nothing special happens to these at boot beyond the balance re-sync above.
  5. Releases expired reservations — any sub-wallet reserved for a checkout session that already expired while the node was down gets released back to available.
  6. Resets stuck background jobs — webhook and reporting-sync jobs that were recorded as "running" when the node went down get reset to pending so they retry normally.

The invariant underneath all of this: coins only ever move on a fully settled, node-co-signed transaction, and a wallet holding funds is always a wallet whose key your node holds. A crash can strand a wallet in a locked or unswept state for a while, but it can't lose money — the worst case is a FUNDS_HELD wallet sitting a little longer than usual, which the sweep loop drains as soon as it's freed.

When to contact support versus act yourself

Act yourself when:

  • A NEEDS_ATTENTION row's real chain state is clearly readable from the dashboard (funds obviously landed, or obviously didn't) — resolve or retry as described above.
  • A sweep failed and simply needs a manual retry after a known transient gateway issue has cleared.
  • The node is slow to become ready after a restart — this is normal boot recovery work; give it a minute before assuming something's wrong.

Contact support when:

  • A NEEDS_ATTENTION row's chain state doesn't match either the "clearly landed" or "clearly didn't land" case, and you're not confident forcing a retry is safe.
  • The node fails to reach /health/ready well beyond a normal boot window.
  • You suspect a wallet key may have been compromised (see the incident response steps in key-rotation.md) — this is urgent, don't wait for a scheduled check-in.
  • Anything involving a discrepancy between what the dashboard shows and what you can independently verify on-chain for a wallet's balance.