Operational runbook

Use this runbook for day-2 operations after launch.

Watch items

• operator API auth failures
• wallet adapter timeouts
• balance operations in unknown or manual_review
• webhook deliveries in failed or dead_letter
• unexpected market visibility complaints

First-response flow

1. identify affected user or event
2. capture UTC time window
3. locate idempotency_key, event_id, order_id, or delivery ID
4. determine if money movement may be affected
5. stop retries or replays until state is understood

Wallet incidents

If balance operation is unknown:

1. query operation lookup by same idempotency_key
2. confirm final remote result
3. only then decide whether replay, compensation, or manual review is needed

If balance operation is manual_review:

• gather operator-side wallet logs
• gather related trade or redemption record
• escalate with exact IDs

Webhook incidents

If delivery is failed:

• inspect last status code and response body sample
• fix receiver issue
• wait for retry or replay manually if needed

If delivery is dead_letter:

• confirm receiver fix deployed
• replay from dashboard or operator API
• confirm latest attempt becomes sent

Key rotation runbook

1. create replacement key
2. deploy secret
3. smoke one endpoint
4. revoke old key
5. monitor for INVALID_API_KEY spikes