2d1dca0804
* fix(agent-team): serve() starts the inbound Slack listener (D-1) Coordinator.serve() now constructs and starts the SlackListener concurrently with the tick/drain loop on a background daemon thread, but ONLY when the live transport is a SlackTransport AND SLACK_APP_TOKEN is configured. When Slack is not the transport or the app token is absent, serve() behaves exactly as before (tick/recover only) — Slack is never made mandatory. - New injectable build_listener seam + default_slack_listener_factory sharing the coordinator's own transport, ledger db_path, and resume_queue put. - AUTHZ-01 owner-allowlist + open-status CAS untouched: serve() sources AGENT_TEAM_SLACK_OWNER_IDS in SlackListener.serve, which still fails closed. - SlackListener.close() added for clean Socket Mode teardown on shutdown; serve() stops the listener + joins the thread in a finally. - Tests: start-when-Slack+app-token, no-start otherwise, clean shutdown, idempotent start, serve start/stop around the loop, listener close(). * fix(agent-team): systemd unit loads ~/orchestrator/.env + uses venv python (D-2/D-7) D-2: add EnvironmentFile=-/home/adam/orchestrator/.env (optional '-') so the P2 GPT-4.1 review loop's cross_reviewer sub-process can read the non-Claude provider key once a task reaches REVIEW. Mirrors the sea-haven-secrev unit. D-7: point ExecStart at the agent-team venv interpreter (/home/adam/orchestrator/agent-team/.venv/bin/python) instead of /usr/bin/env python3, which resolved the system interpreter without the installed deps under systemd's PATH. All hardening (NoNewPrivileges / ProtectSystem=full / ProtectHome=read-only / ReadWritePaths) is retained unchanged (locked decision). * docs(agent-team): land provisioning + operator runbooks under docs/provisioning - PROVISIONING-RUNBOOK.md: merged final state (6 checkers, dep-bump fixer, P5 intake-checker loop), SLACK_CHANNEL_ID, the gated P3-live flip steps (GitHub App + agent-apply env + gated_build_verify_wiring), and D-1/D-2/D-7 marked FIXED so the demo can use the live Slack answer path. - P1-DEMO-SCRIPT.md: live Slack answer path now available (D-1 fixed); both the Slack and operator-CLI answer paths documented for all four exit criteria. - DEPLOY-AUDIT.md: D-1/D-2/D-7 RESOLVED (this PR); D-4/D-5 dep pinning and the operator-CLI divergence kept as provisioning notes. - OPERATOR-RUNBOOK.md (new): incident handling for pipeline stalls, parked tasks, failed HITL resumes, budget exhaustion, transport outages, and COMPLACENCY/COVERAGE alarms — each grounded in real run-team.py verbs, plus the re-alarm-backoff -> Jira-after-N-nights escalation ladder (design §5/§6.6). * fix(agent-team): supervise the Slack listener thread — recurring ALARM + respawn sh-security-review (logic) MEDIUM: a crashed listener thread was logged once, then the daemon ran on 'deaf' — posting clarifier questions but receiving no answers, every gate silently parking, process never exiting so systemd Restart=on-failure never fired. serve() now calls _supervise_slack_listener() each pass: when the listener is enabled but its thread is dead, it emits a recurring ERROR ALARM and respawns via the idempotent starter (self-heal). No-op when alive or disabled. +3 tests. (authz detector: wiring clean — AUTHZ-01 fail-closed allowlist + open-status CAS intact, dead listener fails SAFE.)
187 lines
9.5 KiB
Markdown
187 lines
9.5 KiB
Markdown
# DEPLOY-AUDIT — `agent-team/DEPLOY-R720.md` + systemd unit vs. actual code
|
|
|
|
Cross-check of the drafted deploy doc (`agent-team/DEPLOY-R720.md`) and the
|
|
systemd unit (`agent-team/systemd/agent-team-coordinator.service`) against the
|
|
**actual current code** in `agent-team/agent_team/**` and `agent-team/run-team.py`.
|
|
|
|
Each finding: **location → claimed → actual → fix**. Severity: 🔴 blocker /
|
|
🟠 should-fix / 🟡 nit. Items confirmed clean are stated explicitly.
|
|
|
|
> **Status (deploy-readiness PR):** the three deploy-correctness bugs that
|
|
> blocked the live provisioning session — **D-1, D-2, D-7** — are **RESOLVED** in
|
|
> this PR (`feature/agent-team-deploy-readiness`). The remaining items
|
|
> (**D-4 / D-5** dependency pinning, the **operator-CLI divergence**) are kept
|
|
> below as **provisioning notes** — they do not block the coordinator deploy.
|
|
|
|
---
|
|
|
|
## ✅ D-1 — `serve` now starts the Slack inbound listener — RESOLVED (this PR)
|
|
|
|
- **Location:** `agent_team/coordinator.py` (`Coordinator.serve` +
|
|
`_maybe_start_slack_listener` / `_slack_listener_enabled` / `_stop_slack_listener`
|
|
/ `default_slack_listener_factory`); `agent_team/transport/slack_listener.py`
|
|
(`SlackListener.serve` + new `close`).
|
|
- **Was:** `Coordinator.serve()` did only `bind_subscription_invoker()`,
|
|
`setup()`, `recover()`, then an infinite tick/sleep loop — it never constructed
|
|
or started `SlackListener`, so a deployed daemon posted clarifier questions and
|
|
expired them on deadline but could **not hear Slack answers**.
|
|
- **Now:** `serve()` starts the `SlackListener` on a background **daemon thread**,
|
|
concurrently with the tick/drain loop, **when** the live transport is a
|
|
`SlackTransport` AND `SLACK_APP_TOKEN` is set. It shares the coordinator's own
|
|
transport, ledger `db_path`, and `resume_queue` put; on shutdown it calls the
|
|
listener's new `close()` and joins the thread in a `finally`. When Slack is not
|
|
the transport or the app token is absent, no listener starts and `serve` behaves
|
|
exactly as before — **Slack is never made mandatory**. The AUTHZ-01 owner
|
|
allowlist + the open-status compare-and-set are untouched (still fail closed on
|
|
an empty `AGENT_TEAM_SLACK_OWNER_IDS`).
|
|
- **Tests:** `tests/test_coordinator.py` — start-when-Slack+app-token, no-start
|
|
without the token, no-start when the transport is not Slack, idempotent start,
|
|
clean shutdown, serve start/stop around the loop; `tests/test_slack_listener.py`
|
|
— `close()` no-op + handler teardown.
|
|
|
|
---
|
|
|
|
## ✅ D-2 — coordinator unit loads `~/orchestrator/.env` — RESOLVED (this PR)
|
|
|
|
- **Location:** `agent-team/systemd/agent-team-coordinator.service`;
|
|
`run-team.py:_build_coordinator` (always wires `default_review_wiring`);
|
|
`coordinator.py:default_review_wiring` → `review_loop_llm.default_plan_reviewer`
|
|
(shells the local orchestrator `run.py` → `cross_reviewer` GPT-4.1).
|
|
- **Was:** the unit loaded only `~/secrev.env`. `run-team.py serve` builds the
|
|
coordinator with the P2 review loop wired, and once a task reaches REVIEW the
|
|
reviewer shells the orchestrator `run.py`, whose GPT-4.1 call reads the
|
|
non-Claude provider key from `~/orchestrator/.env`. The review path would fail
|
|
to authenticate.
|
|
- **Now:** the unit adds `EnvironmentFile=-/home/adam/orchestrator/.env`
|
|
(optional `-`, mirroring the `sea-haven-secrev` unit). Does not affect the P1
|
|
demo (P1 stops at PLAN before REVIEW); closes the latent P2 break.
|
|
|
|
---
|
|
|
|
## 🟠 D-4 — pip install list omits `requests` (provisioning note)
|
|
|
|
- **Location:** `agent_team/transport/github_live.py` / `github_intake.py`
|
|
(`import requests`).
|
|
- **Actual:** the GitHub transport + intake require `requests`; the Slack-first
|
|
path does not hit it, but any `--transport github` / `intake-github` use fails
|
|
with a clear RuntimeError without it.
|
|
- **Fix:** PROVISIONING-RUNBOOK Step 4 installs `requests` into the venv.
|
|
`requests` is also absent from `requirements.txt` (see D-5).
|
|
|
|
---
|
|
|
|
## 🟠 D-5 — agent-team runtime deps are not pinned in `requirements.txt` (provisioning note)
|
|
|
|
- **Location:** `requirements.txt` (repo root).
|
|
- **Actual:** `requirements.txt` pins `langgraph==1.1.10` and
|
|
`langgraph-checkpoint-sqlite==3.1.0`, but the agent-team runtime deps
|
|
`claude-agent-sdk`, `slack_sdk`, `slack_bolt`, `requests` (and `anthropic` for
|
|
api mode) are **not in `requirements.txt` at all** — they are installed ad-hoc
|
|
into the agent-team venv by the runbook. There is no pinned, reproducible source
|
|
of truth for the box's runtime set.
|
|
- **Fix (deferred):** add an `agent-team/requirements.txt` (or extras group)
|
|
pinning these, version-matched to the root `requirements.txt` langgraph pin.
|
|
Until then, PROVISIONING-RUNBOOK Step 4 pins `langgraph==1.1.10` /
|
|
`langgraph-checkpoint-sqlite==3.1.0` explicitly so the unpinned `pip install`
|
|
cannot pull a newer, untested major. **Do NOT modify `requirements.txt` or the
|
|
checkers in this PR** (out of scope).
|
|
|
|
---
|
|
|
|
## ✅ D-6 — `slack_bolt` is now exercised by the daemon — RESOLVED (consequence of D-1)
|
|
|
|
- **Location:** `slack_listener.py:serve` (the only `slack_bolt` import).
|
|
- **Now:** with D-1 fixed, `Coordinator.serve()` starts `SlackListener.serve()`,
|
|
which imports + uses `slack_bolt` for the Socket Mode handler. The dep is right
|
|
and now actually exercised on the live Slack path.
|
|
|
|
---
|
|
|
|
## ✅ D-SLACKVAR (clean) — `SLACK_CHANNEL_ID` matches
|
|
|
|
- `run-team.py:_build_transport` reads exactly `os.environ.get("SLACK_CHANNEL_ID")`.
|
|
The runbook, the unit comment, and the code all use `SLACK_CHANNEL_ID` (not
|
|
`SLACK_CHANNEL`). **CLEAN.**
|
|
|
|
---
|
|
|
|
## ✅ D-ENV-SLACKBOT / OAUTH / OWNERS / APPTOKEN (clean) — names match
|
|
|
|
- **`SLACK_BOT_TOKEN`** ↔ `slack_live.py` + `slack_listener` env read. **CLEAN.**
|
|
- **`CLAUDE_CODE_OAUTH_TOKEN`** ↔ `invoker.py`. **CLEAN.**
|
|
- **`AGENT_TEAM_SLACK_OWNER_IDS`** ↔ `slack_listener.py` (name + fail-closed
|
|
semantics). **CLEAN** — now read by the running daemon (D-1 fixed).
|
|
- **`SLACK_APP_TOKEN`** — now read in two places: `coordinator._slack_listener_enabled`
|
|
gates the listener on its presence, and `default_slack_listener_factory` /
|
|
`SlackListener.serve` source it to open the socket. **CLEAN** (read site exists
|
|
now that D-1 is fixed).
|
|
|
|
---
|
|
|
|
## ✅ D-7 — `ExecStart` uses the venv interpreter — RESOLVED (this PR)
|
|
|
|
- **Location:** `agent-team/systemd/agent-team-coordinator.service` ExecStart.
|
|
- **Was:** `ExecStart=/usr/bin/env python3 run-team.py serve` resolved the
|
|
**system** interpreter under systemd's PATH — not the venv where the deps were
|
|
installed, so the daemon would fail at import.
|
|
- **Now:** `ExecStart=/home/adam/orchestrator/agent-team/.venv/bin/python run-team.py serve`
|
|
(matches the runbook venv path + `WorkingDirectory`).
|
|
|
|
---
|
|
|
|
## ✅ D-SUBCMD (mostly clean) — run-team.py subcommands referenced exist
|
|
|
|
Cross-checked every `run-team.py <sub>` the deploy doc + demo name against
|
|
`run-team.py:build_parser`: `init-db`, `serve`, `list` (+ `--all` / `--parked`),
|
|
`show`, `expire`, `answer`, `redeliver`, `supersede`, `force-resume`, `start`,
|
|
`intake-github`, `intake-checker`, `fix` — all exist. No invented verbs.
|
|
|
|
> ### Operator-CLI divergence (provisioning note)
|
|
>
|
|
> Two operator CLIs exist with **different verb names**:
|
|
>
|
|
> - `run-team.py` (the entry CLI): `init-db, list, show, redeliver, expire,
|
|
> answer, supersede, force-resume, start, serve, intake-github, intake-checker,
|
|
> fix`. Has `show` and `--parked`; `--db` / `--audit-log` default sensibly.
|
|
> - `agent_team/operator_cli.py`: `list, redeliver, force-expire,
|
|
> answer-on-behalf, force-resume` — **no `show`**, `--db` / `--audit-log` are
|
|
> **required**, and its `force-resume` **supersedes** (unlike `run-team.py`'s,
|
|
> which reopens an expired row and never supersedes).
|
|
>
|
|
> **Use `run-team.py` for provisioning + the demo + incident recovery.** The
|
|
> docs reference only `run-team.py`. Reconciling the two CLIs is a follow-up.
|
|
|
|
---
|
|
|
|
## ✅ D-PYTHONPKG (clean) — package import bootstrap is correct
|
|
|
|
`run-team.py` inserts its own dir into `sys.path` so the hyphenated script
|
|
imports the `agent_team` package without an editable install. **CLEAN.**
|
|
|
|
---
|
|
|
|
## ✅ D-HARDENING (clean, and matches the locked decision)
|
|
|
|
- Unit: `NoNewPrivileges=true`, `ProtectSystem=full`, `ProtectHome=read-only`,
|
|
`ReadWritePaths=/home/adam/orchestrator/agent-team/state`. **Retained unchanged**
|
|
in this PR (locked decision — do not revert to secrev parity).
|
|
- The `ReadWritePaths` carve-out matches the ledger + audit-log location
|
|
(`state/agent_team.sqlite`, `state/audit.log.jsonl`). **CLEAN.**
|
|
|
|
---
|
|
|
|
## Summary table
|
|
|
|
| ID | Sev | Status | One-line |
|
|
|---|---|---|---|
|
|
| D-1 | 🔴 | ✅ RESOLVED (PR) | `serve` starts `SlackListener` (Slack + app-token gated; Slack stays optional) |
|
|
| D-2 | 🔴 | ✅ RESOLVED (PR) | unit loads `~/orchestrator/.env` for the P2 GPT-4.1 review provider key |
|
|
| D-7 | 🟡 | ✅ RESOLVED (PR) | `ExecStart` points at the agent-team venv interpreter |
|
|
| D-6 | 🟡 | ✅ RESOLVED | `slack_bolt` now exercised by the daemon (consequence of D-1) |
|
|
| D-4 | 🟠 | NOTE | pip list omits `requests` — runbook Step 4 installs it |
|
|
| D-5 | 🟠 | NOTE | agent-team runtime deps not pinned in `requirements.txt` — runbook pins langgraph |
|
|
| operator-CLI | — | NOTE | `run-team.py` vs `operator_cli.py` divergent verbs — use `run-team.py` |
|
|
| D-SLACKVAR | ✅ | CLEAN | `SLACK_CHANNEL_ID` matches everywhere |
|
|
| D-ENV-* | ✅ | CLEAN | bot/oauth/owner/app-token env names match; all read sites now exist |
|
|
| D-SUBCMD | ✅ | CLEAN | every `run-team.py` verb/flag the docs cite exists |
|
|
| D-HARDENING | ✅ | CLEAN | unit hardening retained unchanged (locked decision) |
|