Skip to main content

Checkpoints & Recovery

Deploying the full system to Hedera is dozens of transactions. Networks hiccup, accounts run low on HBAR, RPC relays rate-limit. The checkpoint system makes every deployment and upgrade workflow resumable: if a run fails or you interrupt it, you fix the cause and re-run the same command — completed steps are skipped, saving time and gas.

For the full reference, see the in-repo scripts/CHECKPOINT_GUIDE.md.

How it works​

  • After each successful step (ProxyAdmin, BLR, each facet batch, each configuration, Factory), the workflow writes its progress to a checkpoint file under deployments/<network>/.checkpoints/.
  • On the next run, the workflow detects a resumable checkpoint, skips completed steps, and continues from where it stopped.
  • On success, the checkpoint is marked complete; you can clean it up afterwards.
# A run gets interrupted at step 3…
Step 1: Deploy ProxyAdmin... ✅
Step 2: Deploy BLR... ✅
Step 3: Deploy Facets... ❌ ERROR: Transaction failed

# …fix the cause, then just run the same command again:
npm run deploy:hedera:testnet
# ✅ Resumes from step 3, skipping steps 1–2.

You do not pass any special flag to resume — re-running the deploy command is the resume.

Recovering from a failed deployment​

1. See what happened​

# List checkpoints for the network
npm run checkpoint:list -- hedera-testnet

# Show full details of the failed checkpoint (which step, the error, completed/pending steps)
npm run checkpoint:show -- <checkpoint-id>

2. Fix the root cause​

ErrorRoot causeFix
Insufficient gas / gas limitLimit too low for a complex stepLower BATCH_SIZE, or raise the gas limit in your env
Nonce too lowPending transactions / mempoolWait 1–2 minutes, retry
Network unreachableRPC node down or misconfiguredCheck your *_JSON_RPC_ENDPOINT
Transaction underpricedGas price too lowRaise gas price in the network config
Facet batch failedTransient network failureRe-run; deployed facets are skipped

3. Resume​

# Same command as the original deployment
npm run deploy:hedera:testnet

The workflow detects the failed checkpoint, asks you to confirm the resume (showing the failure details), skips completed steps, retries the failed one, and continues to completion.

4. Verify​

cat "$(ls -t deployments/hedera-testnet/newBlr-*.json | head -1)" | jq '.infrastructure'

Checkpoint management CLI​

npm run checkpoint:list    -- <network>          # list checkpoints for a network
npm run checkpoint:show    -- <checkpoint-id>    # show details of a checkpoint
npm run checkpoint:delete  -- <checkpoint-id>    # delete a specific checkpoint
npm run checkpoint:cleanup -- <network> <days>   # delete completed checkpoints older than N days
npm run checkpoint:reset   -- <checkpoint-id>    # reset a failed checkpoint to in-progress

Typical maintenance after a successful deployment:

# Remove old failed attempts, keep recent history
npm run checkpoint:cleanup -- hedera-testnet 7

CI/CD (non-interactive)​

In a non-TTY environment the resume prompt can't be answered interactively. Two patterns:

  • Auto-resume — let the workflow detect and resume the checkpoint automatically (the default in non-interactive mode).

  • Fresh start — delete old checkpoints first, then deploy:

    npm run checkpoint:delete -- <checkpoint-id>
    npm run deploy:hedera:testnet

Best practices​

  • ✅ Review the failure (checkpoint:show) before resuming — fix the root cause, don't retry blindly.
  • ✅ Let the system skip completed steps; don't redeploy by hand.
  • ✅ Clean up old checkpoints after a successful deployment.
  • ❌ Don't hand-edit checkpoint files — it's easy to create inconsistent state and break resume safety.
  • ❌ Don't change network configuration between resume attempts.
  • ❌ Don't delete a checkpoint immediately after a failure — you'll usually want to resume from it.