adam/orchestrator
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
This repository has been archived on 2026-08-04. You can view files and clone it, but cannot push or open issues or pull requests.
orchestrator/docs/provisioning/DEPLOY-AUDIT.md
Adam Moussa 2d1dca0804
feat(agent-team): deploy-readiness — serve starts Slack listener + systemd + provisioning docs (#23) 
* fix(agent-team): serve() starts the inbound Slack listener (D-1)

Coordinator.serve() now constructs and starts the SlackListener concurrently
with the tick/drain loop on a background daemon thread, but ONLY when the live
transport is a SlackTransport AND SLACK_APP_TOKEN is configured. When Slack is
not the transport or the app token is absent, serve() behaves exactly as before
(tick/recover only) — Slack is never made mandatory.

- New injectable build_listener seam + default_slack_listener_factory sharing
  the coordinator's own transport, ledger db_path, and resume_queue put.
- AUTHZ-01 owner-allowlist + open-status CAS untouched: serve() sources
  AGENT_TEAM_SLACK_OWNER_IDS in SlackListener.serve, which still fails closed.
- SlackListener.close() added for clean Socket Mode teardown on shutdown;
  serve() stops the listener + joins the thread in a finally.
- Tests: start-when-Slack+app-token, no-start otherwise, clean shutdown,
  idempotent start, serve start/stop around the loop, listener close().

* fix(agent-team): systemd unit loads ~/orchestrator/.env + uses venv python (D-2/D-7)

D-2: add EnvironmentFile=-/home/adam/orchestrator/.env (optional '-') so the P2
GPT-4.1 review loop's cross_reviewer sub-process can read the non-Claude provider
key once a task reaches REVIEW. Mirrors the sea-haven-secrev unit.

D-7: point ExecStart at the agent-team venv interpreter
(/home/adam/orchestrator/agent-team/.venv/bin/python) instead of
/usr/bin/env python3, which resolved the system interpreter without the
installed deps under systemd's PATH.

All hardening (NoNewPrivileges / ProtectSystem=full / ProtectHome=read-only /
ReadWritePaths) is retained unchanged (locked decision).

* docs(agent-team): land provisioning + operator runbooks under docs/provisioning

- PROVISIONING-RUNBOOK.md: merged final state (6 checkers, dep-bump fixer, P5
  intake-checker loop), SLACK_CHANNEL_ID, the gated P3-live flip steps (GitHub
  App + agent-apply env + gated_build_verify_wiring), and D-1/D-2/D-7 marked
  FIXED so the demo can use the live Slack answer path.
- P1-DEMO-SCRIPT.md: live Slack answer path now available (D-1 fixed); both the
  Slack and operator-CLI answer paths documented for all four exit criteria.
- DEPLOY-AUDIT.md: D-1/D-2/D-7 RESOLVED (this PR); D-4/D-5 dep pinning and the
  operator-CLI divergence kept as provisioning notes.
- OPERATOR-RUNBOOK.md (new): incident handling for pipeline stalls, parked tasks,
  failed HITL resumes, budget exhaustion, transport outages, and
  COMPLACENCY/COVERAGE alarms — each grounded in real run-team.py verbs, plus the
  re-alarm-backoff -> Jira-after-N-nights escalation ladder (design §5/§6.6).

* fix(agent-team): supervise the Slack listener thread — recurring ALARM + respawn

sh-security-review (logic) MEDIUM: a crashed listener thread was logged once,
then the daemon ran on 'deaf' — posting clarifier questions but receiving no
answers, every gate silently parking, process never exiting so systemd
Restart=on-failure never fired. serve() now calls _supervise_slack_listener()
each pass: when the listener is enabled but its thread is dead, it emits a
recurring ERROR ALARM and respawns via the idempotent starter (self-heal).
No-op when alive or disabled. +3 tests. (authz detector: wiring clean — AUTHZ-01
fail-closed allowlist + open-status CAS intact, dead listener fails SAFE.)
2026-06-18 16:56:21 -04:00

9.5 KiB
Raw Blame History

DEPLOY-AUDIT — agent-team/DEPLOY-R720.md + systemd unit vs. actual code

Cross-check of the drafted deploy doc (agent-team/DEPLOY-R720.md) and the systemd unit (agent-team/systemd/agent-team-coordinator.service) against the actual current code in agent-team/agent_team/** and agent-team/run-team.py.

Each finding: location → claimed → actual → fix. Severity: 🔴 blocker / 🟠 should-fix / 🟡 nit. Items confirmed clean are stated explicitly.

Status (deploy-readiness PR): the three deploy-correctness bugs that blocked the live provisioning session — D-1, D-2, D-7 — are RESOLVED in this PR (feature/agent-team-deploy-readiness). The remaining items (D-4 / D-5 dependency pinning, the operator-CLI divergence) are kept below as provisioning notes — they do not block the coordinator deploy.

✅ D-1 — serve now starts the Slack inbound listener — RESOLVED (this PR)

  • Location: agent_team/coordinator.py (Coordinator.serve + _maybe_start_slack_listener / _slack_listener_enabled / _stop_slack_listener / default_slack_listener_factory); agent_team/transport/slack_listener.py (SlackListener.serve + new close).
  • Was: Coordinator.serve() did only bind_subscription_invoker(), setup(), recover(), then an infinite tick/sleep loop — it never constructed or started SlackListener, so a deployed daemon posted clarifier questions and expired them on deadline but could not hear Slack answers.
  • Now: serve() starts the SlackListener on a background daemon thread, concurrently with the tick/drain loop, when the live transport is a SlackTransport AND SLACK_APP_TOKEN is set. It shares the coordinator's own transport, ledger db_path, and resume_queue put; on shutdown it calls the listener's new close() and joins the thread in a finally. When Slack is not the transport or the app token is absent, no listener starts and serve behaves exactly as before — Slack is never made mandatory. The AUTHZ-01 owner allowlist + the open-status compare-and-set are untouched (still fail closed on an empty AGENT_TEAM_SLACK_OWNER_IDS).
  • Tests: tests/test_coordinator.py — start-when-Slack+app-token, no-start without the token, no-start when the transport is not Slack, idempotent start, clean shutdown, serve start/stop around the loop; tests/test_slack_listener.py — close() no-op + handler teardown.

✅ D-2 — coordinator unit loads ~/orchestrator/.env — RESOLVED (this PR)

  • Location: agent-team/systemd/agent-team-coordinator.service; run-team.py:_build_coordinator (always wires default_review_wiring); coordinator.py:default_review_wiring → review_loop_llm.default_plan_reviewer (shells the local orchestrator run.py → cross_reviewer GPT-4.1).
  • Was: the unit loaded only ~/secrev.env. run-team.py serve builds the coordinator with the P2 review loop wired, and once a task reaches REVIEW the reviewer shells the orchestrator run.py, whose GPT-4.1 call reads the non-Claude provider key from ~/orchestrator/.env. The review path would fail to authenticate.
  • Now: the unit adds EnvironmentFile=-/home/adam/orchestrator/.env (optional -, mirroring the sea-haven-secrev unit). Does not affect the P1 demo (P1 stops at PLAN before REVIEW); closes the latent P2 break.

🟠 D-4 — pip install list omits requests (provisioning note)

  • Location: agent_team/transport/github_live.py / github_intake.py (import requests).
  • Actual: the GitHub transport + intake require requests; the Slack-first path does not hit it, but any --transport github / intake-github use fails with a clear RuntimeError without it.
  • Fix: PROVISIONING-RUNBOOK Step 4 installs requests into the venv. requests is also absent from requirements.txt (see D-5).

🟠 D-5 — agent-team runtime deps are not pinned in requirements.txt (provisioning note)

  • Location: requirements.txt (repo root).
  • Actual: requirements.txt pins langgraph==1.1.10 and langgraph-checkpoint-sqlite==3.1.0, but the agent-team runtime deps claude-agent-sdk, slack_sdk, slack_bolt, requests (and anthropic for api mode) are not in requirements.txt at all — they are installed ad-hoc into the agent-team venv by the runbook. There is no pinned, reproducible source of truth for the box's runtime set.
  • Fix (deferred): add an agent-team/requirements.txt (or extras group) pinning these, version-matched to the root requirements.txt langgraph pin. Until then, PROVISIONING-RUNBOOK Step 4 pins langgraph==1.1.10 / langgraph-checkpoint-sqlite==3.1.0 explicitly so the unpinned pip install cannot pull a newer, untested major. Do NOT modify requirements.txt or the checkers in this PR (out of scope).

✅ D-6 — slack_bolt is now exercised by the daemon — RESOLVED (consequence of D-1)

  • Location: slack_listener.py:serve (the only slack_bolt import).
  • Now: with D-1 fixed, Coordinator.serve() starts SlackListener.serve(), which imports + uses slack_bolt for the Socket Mode handler. The dep is right and now actually exercised on the live Slack path.

✅ D-SLACKVAR (clean) — SLACK_CHANNEL_ID matches

  • run-team.py:_build_transport reads exactly os.environ.get("SLACK_CHANNEL_ID"). The runbook, the unit comment, and the code all use SLACK_CHANNEL_ID (not SLACK_CHANNEL). CLEAN.

✅ D-ENV-SLACKBOT / OAUTH / OWNERS / APPTOKEN (clean) — names match

  • SLACK_BOT_TOKEN ↔ slack_live.py + slack_listener env read. CLEAN.
  • CLAUDE_CODE_OAUTH_TOKEN ↔ invoker.py. CLEAN.
  • AGENT_TEAM_SLACK_OWNER_IDS ↔ slack_listener.py (name + fail-closed semantics). CLEAN — now read by the running daemon (D-1 fixed).
  • SLACK_APP_TOKEN — now read in two places: coordinator._slack_listener_enabled gates the listener on its presence, and default_slack_listener_factory / SlackListener.serve source it to open the socket. CLEAN (read site exists now that D-1 is fixed).

✅ D-7 — ExecStart uses the venv interpreter — RESOLVED (this PR)

  • Location: agent-team/systemd/agent-team-coordinator.service ExecStart.
  • Was: ExecStart=/usr/bin/env python3 run-team.py serve resolved the system interpreter under systemd's PATH — not the venv where the deps were installed, so the daemon would fail at import.
  • Now: ExecStart=/home/adam/orchestrator/agent-team/.venv/bin/python run-team.py serve (matches the runbook venv path + WorkingDirectory).

✅ D-SUBCMD (mostly clean) — run-team.py subcommands referenced exist

Cross-checked every run-team.py <sub> the deploy doc + demo name against run-team.py:build_parser: init-db, serve, list (+ --all / --parked), show, expire, answer, redeliver, supersede, force-resume, start, intake-github, intake-checker, fix — all exist. No invented verbs.

Operator-CLI divergence (provisioning note)

Two operator CLIs exist with different verb names:

  • run-team.py (the entry CLI): init-db, list, show, redeliver, expire, answer, supersede, force-resume, start, serve, intake-github, intake-checker, fix. Has show and --parked; --db / --audit-log default sensibly.
  • agent_team/operator_cli.py: list, redeliver, force-expire, answer-on-behalf, force-resume — no show, --db / --audit-log are required, and its force-resume supersedes (unlike run-team.py's, which reopens an expired row and never supersedes).

Use run-team.py for provisioning + the demo + incident recovery. The docs reference only run-team.py. Reconciling the two CLIs is a follow-up.

✅ D-PYTHONPKG (clean) — package import bootstrap is correct

run-team.py inserts its own dir into sys.path so the hyphenated script imports the agent_team package without an editable install. CLEAN.

✅ D-HARDENING (clean, and matches the locked decision)

  • Unit: NoNewPrivileges=true, ProtectSystem=full, ProtectHome=read-only, ReadWritePaths=/home/adam/orchestrator/agent-team/state. Retained unchanged in this PR (locked decision — do not revert to secrev parity).
  • The ReadWritePaths carve-out matches the ledger + audit-log location (state/agent_team.sqlite, state/audit.log.jsonl). CLEAN.

Summary table

ID Sev Status One-line
D-1 🔴 ✅ RESOLVED (PR) serve starts SlackListener (Slack + app-token gated; Slack stays optional)
D-2 🔴 ✅ RESOLVED (PR) unit loads ~/orchestrator/.env for the P2 GPT-4.1 review provider key
D-7 🟡 ✅ RESOLVED (PR) ExecStart points at the agent-team venv interpreter
D-6 🟡 ✅ RESOLVED slack_bolt now exercised by the daemon (consequence of D-1)
D-4 🟠 NOTE pip list omits requests — runbook Step 4 installs it
D-5 🟠 NOTE agent-team runtime deps not pinned in requirements.txt — runbook pins langgraph
operator-CLI — NOTE run-team.py vs operator_cli.py divergent verbs — use run-team.py
D-SLACKVAR ✅ CLEAN SLACK_CHANNEL_ID matches everywhere
D-ENV-* ✅ CLEAN bot/oauth/owner/app-token env names match; all read sites now exist
D-SUBCMD ✅ CLEAN every run-team.py verb/flag the docs cite exists
D-HARDENING ✅ CLEAN unit hardening retained unchanged (locked decision)