orchestrator/docs/provisioning/OPERATOR-RUNBOOK.md

OPERATOR-RUNBOOK — R720 agent-team coordinator incident handling

The on-call runbook for the always-on agent-team-coordinator daemon on the sh-secrev R720 VM. Covers pipeline stalls, stuck/parked tasks, failed human-in-the-loop resumes, budget exhaustion mid-pipeline, transport outages, and the COMPLACENCY / COVERAGE alarms. Grounds every recovery in real code (design §5 escalation ladder + §6.6 contention/park policy; Phase-6 requirement).

CLI used throughout: run-team.py (the entry CLI), run from ~/orchestrator/agent-team with the venv active so it hits the default ledger (state/agent_team.sqlite) and audit log (state/audit.log.jsonl). Do not use agent_team/operator_cli.py — it has divergent verbs (no show, required --db/--audit-log, and a force-resume that supersedes). See DEPLOY-AUDIT.md.

ssh -i ~/.ssh/r720_seahaven adam@10.10.60.120
cd ~/orchestrator/agent-team && . .venv/bin/activate

Verb reference (all confirmed in run-team.py:build_parser)

Verb	Effect	Destructive?
list / list --all / list --parked / list --status <state>	read pending questions (default open)	no
show <question_id>	print one ledger row (JSON)	no
redeliver <question_id>	clear channel_ref so the reconcile loop re-posts an open question	no (audit-logged)
expire <question_id> --confirm	force open→expired	yes
answer <question_id> --answer <p> [--via <id>] --confirm	answer-on-behalf (first-answer-wins CAS)	yes
force-resume <question_id> --confirm	reopen an expired (parked) question; for answered records resume intent	yes
supersede <question_id> --confirm	mark a stale open/answered row superseded	yes
start --task "..." [--transport ...] [--dry-run]	start one task to the human gate	no

--operator <name> (global) sets the audit attribution; it defaults to the OS login. Destructive verbs require --confirm and write an attempt-then-outcome record to the audit log before mutating.

First triage for any incident:

systemctl status agent-team-coordinator.service
journalctl -u agent-team-coordinator.service -e --since "-2h" | tail -100
python3 run-team.py list --all          # full ledger snapshot
python3 run-team.py list --parked       # non-open rows (parked-task context)

---

Incident 1 — Pipeline stall (tasks not advancing)

Symptoms: list shows open (or answered) rows that never progress; the journal shows no tick activity or repeated errors.

Diagnose:

systemctl is-active agent-team-coordinator.service     # "active" expected
journalctl -u agent-team-coordinator.service -e | tail -60
# Daemon dead/looping on restart? Check the unit + deps:
.venv/bin/python -c "import langgraph, slack_sdk, slack_bolt, requests; print('deps ok')"

Recover:

1. If the daemon is dead and Restart=on-failure is flapping, read the journal for the import/auth error. A missing venv dep (D-4/D-5) or a missing CLAUDE_CODE_OAUTH_TOKEN is the usual cause — fix the env / venv, then:

   sudo systemctl restart agent-team-coordinator.service
   

2. The startup recover() sweep re-drives answered-but-unresumed rows and re-posts open rows that lost their channel_ref, so a clean restart converges from the durable ledger. Confirm with journalctl ... | tail and list --all.
3. A single task stuck open with a stale/missing post: re-deliver it.

   python3 run-team.py show <qid>
   python3 run-team.py redeliver <qid>     # clears channel_ref; reconcile re-posts
   

Escalate (see the ladder) if a restart does not clear it within one tick cadence and the journal shows a non-transient error.

---

Incident 2 — Stuck / parked task

A task parks (design §6.6) when its clarifier question expires with no answer, or its budget headroom drops below reserve, or a checkpoint is corrupt. Parked = the durable ledger row is no longer open (it is expired), and an ALARM was raised, not spun on.

Diagnose:

python3 run-team.py list --parked
python3 run-team.py show <qid>          # status, thread_id, deadline_at, answered_via

Recover — depends on why it parked:

• Expired with no answer (the common case) — un-park by reopening the expired question; it is then re-delivered for an answer:

  python3 run-team.py force-resume <qid> --confirm
  # "force-resume: reopened expired question <qid>; it will be re-delivered"
  python3 run-team.py show <qid>         # status flips back to "open"
  

  Then answer it (Slack or CLI) to drive it forward.

• Answered but not yet resumed — the recovery sweep handles it; force-resume records intent and reports that (no mutation):

  python3 run-team.py force-resume <qid> --confirm
  # "force-resume: question <qid> is answered and pending resume; the recovery
  #  sweep will resume it (intent recorded)"
  sudo systemctl restart agent-team-coordinator.service   # forces the recover() sweep now
  

• Stale / wrong question that should be abandoned — supersede it so it stops surfacing as parked context:

  python3 run-team.py supersede <qid> --confirm
  

MAX_PARK FIFO-aging: a task that exceeds the park window escalates (ALARM + a Jira ticket per the ladder) rather than starving silently.

---

Incident 3 — Failed human-in-the-loop resume

Symptoms: an answer was submitted (Slack or CLI) but the graph did not advance.

Diagnose:

python3 run-team.py show <qid>          # is status "answered"? what answered_via?
journalctl -u agent-team-coordinator.service -e | grep -iE "resume|answer|<qid>"

Recover:

• Status is answered but no resume drained — the resume queue is in-process; a daemon restart triggers the recover() sweep that re-drives answered rows via the turn-guarded ResumeWorker (idempotent — an already-advanced thread supersedes-and-skips):

  sudo systemctl restart agent-team-coordinator.service
  python3 run-team.py show <qid>        # confirm it advanced
  

• Live Slack answer never registered — the listener fails closed. Check:

  journalctl -u agent-team-coordinator.service -e | grep -i "Slack answer"
  # "rejecting Slack answer: owner allowlist is unconfigured ..." -> set
  #   AGENT_TEAM_SLACK_OWNER_IDS in ~/secrev.env and restart.
  # "rejecting Slack answer ... unauthorized sender" -> the answerer's user id is
  #   not in the allowlist. Add it, or answer via the CLI on their behalf:
  python3 run-team.py answer <qid> --answer "<payload>" --via "cli:adam" --confirm
  

• Answer lost the compare-and-set (not open) — the row was already answered/expired/superseded. Inspect with show; if it parked, go to Incident 2.

---

Incident 4 — Budget exhaustion mid-pipeline

The shared Claude budget ledger (budget_ledger table) enforces per-call + total nightly caps (design §6.1, §6.6). A stage that would breach the reserve parks the task (deferred, not dropped) and ALARMs — it never loop-drains the pool.

Diagnose:

journalctl -u agent-team-coordinator.service -e | grep -iE "budget|reserve|park"
# Inspect today's spend directly (no CLI verb for the budget ledger; read it).
# Columns (agent_team/db/schema.py budget_ledger DDL): thread_id, stage, model,
# billing_mode, input_tokens, output_tokens, usd_cost, recorded_at, day_bucket.
sqlite3 state/agent_team.sqlite \
  "SELECT day_bucket, thread_id, ROUND(SUM(usd_cost),4) AS usd, SUM(input_tokens) AS in_tok
   FROM budget_ledger GROUP BY day_bucket, thread_id ORDER BY day_bucket DESC LIMIT 20;"

Recover:

• Wait for the next budget window — budget-exhausted roles/tasks are deferred via the rotation pointer and picked up next cycle; this is the intended behavior, not a failure. The parked task surfaces in list --parked.
• Force a specific parked task forward now (e.g. it is urgent and headroom has since freed): un-park it and let the daemon re-run the stage within the remaining cap:

  python3 run-team.py force-resume <qid> --confirm
  

• Confirm ANTHROPIC_API_KEY is absent — its presence would silently meter to API rates and blow the budget model (the billing seam pops it defensively, but it must not be set):

  grep -c ANTHROPIC_API_KEY ~/secrev.env ~/orchestrator/.env   # both must print 0
  

Do not raise the cap to push a task through without Adam's decision — budget realism is a deliberate guardrail. Escalate per the ladder if a task repeatedly parks on budget across nights.

---

Incident 5 — Transport outage (Slack / GitHub down or misconfigured)

Symptoms: clarifier posts fail; the journal shows transport errors or the inbound listener is not started.

Diagnose:

journalctl -u agent-team-coordinator.service -e | grep -iE "Slack|listener|post|transport"
# Did the inbound listener start?
journalctl -u agent-team-coordinator.service -e | grep -i "inbound Slack listener"
#   "started (Socket Mode ...)"  -> inbound up
#   "not started (... SLACK_APP_TOKEN is unset)" -> Slack inbound intentionally off

Recover:

• Outbound post failed (Slack API / network) — the ledger row stays open with no channel_ref (the responder leaves it for reconcile). After the transport recovers, the recover() sweep (on restart) or a manual redeliver re-posts:

  python3 run-team.py redeliver <qid>
  

• Inbound listener down / never started — the daemon still posts and expires; it just cannot hear Slack. The CLI answer path is the outage fallback — it runs the identical compare-and-set with no socket:

  python3 run-team.py answer <qid> --answer "<payload>" --via "cli:adam" --confirm
  

  Fix SLACK_APP_TOKEN / SLACK_BOT_TOKEN in ~/secrev.env, then sudo systemctl restart agent-team-coordinator.service to re-establish Socket Mode. (D-1: the listener starts only when the transport is live Slack AND SLACK_APP_TOKEN is set; it is optional by design.)
• Slack listener crash-looping — the listener runs on an isolated daemon thread; a crash is logged (inbound Slack listener thread exited with an error) and takes down only the inbound socket, not the maintenance loop. Restart the service to relaunch the thread once the cause is fixed.

---

Incident 6 — COMPLACENCY / COVERAGE alarms (Plane-1 checkers)

These come from the nightly checker run, not the coordinator daemon (design §6.4, §6.6):

• COMPLACENCY ALARM — a checker role missed a planted canary fault. That role is skipped for the night (it never runs silently degraded). Recover: inspect the canary corpus / the role's checker under security-review/checkers/, fix the regression, re-run that checker's dry-run, and confirm the canary passes before re-enabling.
• COVERAGE ALARM — a role slipped its rotation slot (e.g. budget-deferred). It is deferred via the rotation pointer, never dropped, and picked up next cycle. Recover: confirm the rotation pointer advanced (it is rebuildable from report history) and that the role runs on the next cadence; investigate only if it slips repeatedly.

Both alarms are report + ALARM-only (design D3): nothing posts on a clean state, no auto-Jira/Notion writes from the checker itself. Persistent alarms follow the escalation ladder below.

---

Escalation ladder (design §5 / §6.6, resolves Q3)

Anything that does not clear on the first ALARM escalates — but ALARM-only in spirit (nothing posts on a clean state):

1. Re-alarm on a backoff. A confirmed critical (or a COMPLACENCY / COVERAGE alarm) that persists re-alarms to Slack each night it is still unresolved, on a backoff so it does not spam.
2. Open a Jira tracking ticket after N nights (default N = 3). If the condition still has not cleared, the coordinator opens an INFRA Jira ticket so it cannot quietly linger. The same ladder applies to a parked task that exceeds MAX_PARK.
3. Human (Adam) takes it from the Jira ticket. For a parked task, recover via Incidents 2–4 above; for a checker alarm, via Incident 6.

When you resolve an incident, record the action — destructive CLI verbs already write an attributable attempt+outcome record to state/audit.log.jsonl; for non-CLI recoveries note it on the Jira ticket.

---

Post-incident

• Confirm the daemon is active and the ledger has no unexpected parked rows (list --parked).
• If you restored the VM snapshot or wiped the ledger, re-run the relevant PROVISIONING-RUNBOOK steps.
• Update project_r720_agent_team memory if the incident revealed a durable fact (a new failure mode, a config that must change).