Hermetic e2e tests driving the WHOLE stack composed together — real
build_graph(plan_gate=True) + real Coordinator + real SQLite ledger + real
review_loop router, with only the LLM nodes stubbed — through the daemon API
(start_task/submit_answer/tick), never nodes directly.
Flows: approve settles at BUILD; request-changes via RAW PROSE through the real
SlackListener -> _resolve_payload -> map_plan_decision proves the prose maps to
request_changes (NOT FAILED) and the notes reach the planner; abandon -> FAILED;
repeated request_changes terminates at MAX_PLAN_GATE_VISITS -> PARKED; and a
legacy (no-kind) ledger migrates in place then routes clarify vs plan_decision
correctly. 1459 passed.
Turn a human's Slack interaction at the plan gate into a structured decision the
graph can route, with the kind-aware mapping that closes a silent-FAIL hazard.
- KIND-AWARE NORMALIZATION (load-bearing): map_plan_decision() in slack_adapter
maps a reply to {"decision","notes"} — approve ∈ {approve,approved,yes,ok,lgtm,
ship}; abandon ∈ {abandon,reject,cancel,stop,kill}; EVERYTHING ELSE →
request_changes with the full reply as notes (never accidental abandon). Wired
in the listener's _resolve_payload for plan_decision rows ONLY (clarify passes
through). Without this, arbitrary change-notes hit the graph's
unrecognized-verb→FAILED path and silently fail the task. Anti-FAIL tests
assert prose → request_changes (!= abandon) at both the mapper and the
end-to-end listener seam; a regression test guards clarify pass-through.
New find_open_question_kind_by_channel_ref (anti-replay, status='open') powers
the thread-reply fallback's kind lookup.
- BUTTONS + MODAL: build_plan_decision_blocks() renders Approve (primary) /
Request changes / Abandon (danger+confirm); question_id double-anchored in
message metadata AND each button value ("<verb>:<question_id>"). Approve/abandon
submit via the existing @app.action(.*); request_changes has a dedicated
handler that AUTHORIZES before views_open (proven by test) and opens a notes
modal (private_metadata carries the id) → view_submission → request_changes +
notes. Free-text reply stays the always-available equal path. AUTHZ-01 ordering
preserved.
- No manifest change (views.open needs no extra scope).
1454 passed (1412 + 42).
Connect the graph plan-gate (B2a) to the durable ledger + Slack presentation.
- setup() passes build_graph(plan_gate=True) only on the wired review path
(plan_gate flag ANDed with review_node present); P1/stub paths force it off.
- _post_resume_followups detects a settled interrupt by payload
kind == PLAN_DECISION_KIND (NOT status, which still reads 'parked' at the
gate per B2a) and posts the decision gate: opens a pending_questions row with
kind='plan_decision' (24h deadline, threaded, channel_ref = root ts) and
presents _summarize_plan + findings + reply instructions, truncated to a
~2700-char Slack budget. A clarify/legacy interrupt keeps the existing path.
- Single-open-gate invariant: the opener skips if any open row already exists
for the thread (one row, one presentation).
- Expiry: _park posts a plan-decision-specific recovery notice (re-assign /
force-resume) for an expired gate row.
- Resume path unchanged: the decision answer flows through submit_answer →
ResumeWorker → plan_gate_node with no resume-worker special-casing.
notify_question has no kind param in this tree, so the opener calls
ledger.post_question(kind=...) + ledger.set_channel_ref directly; the clarifier
path still uses notify_question unchanged.
Tests: gate row+presentation+threading, approve/request_changes/abandon via
submit_answer, single-open-gate skip, expiry notice. 1412 passed.
Replace the terminal review-cap PARK with a resumable human decision gate,
opt-in via build_graph(plan_gate=True) (default False → all existing P1/P2/P3
wiring unchanged).
- plan_gate_node interrupt()s mirroring the clarifier contract (same payload
keys → existing pending_question() extractor + turn-guarded ResumeWorker drive
it with zero special-casing) plus a kind="plan_decision" discriminator and the
plan + latest review findings as context.
- Decision contract {"decision": approve|request_changes|abandon, "notes": ...}:
approve → the same terminal state an auto-approved plan reaches (ACTIVE/BUILD);
request_changes → append a synthetic human verdict to review_verdicts (so the
planner's _format_review_feedback surfaces the notes) and loop back to PLAN;
abandon / unrecognized → terminal FAILED (safe default, never accidental
approve).
- Bounded termination: MAX_PLAN_GATE_VISITS=3 combined ceiling on plan_gate_visits
(new channel on PipelineState + TaskRecord); on exhaustion the gate goes
terminal PARKED ("revision ceiling reached") WITHOUT interrupting. Proven by a
loop-past-ceiling test.
Notes for the coordinator wiring (B2b): while suspended at the gate the status
channel still reads 'parked' (carried over from review_node's escalate branch) —
the load-bearing "awaiting decision, not terminal" signal is the live pending
interrupt + kind="plan_decision", NOT the status channel.
Tests: interrupt-at-cap, approve/request_changes(notes)/abandon routing,
ceiling-terminates, auto-approve still bypasses the gate. 1404 passed.
The plan-review gate (coming next) needs to tell its decision questions apart
from clarifier questions in the durable ledger. Add a `kind` column to
pending_questions (values 'clarify' | 'plan_decision').
- Fresh DBs: `kind TEXT NOT NULL DEFAULT 'clarify'` (+ CHECK) in the DDL.
- Live ledger: idempotent additive migration (SCHEMA_VERSION 3→4) — a guarded
ALTER (PRAGMA table_info) run from both migrate() and init_db; legacy rows
take the 'clarify' default, never null. (SQLite can't add a CHECK via ALTER,
so the migrated column is NOT NULL DEFAULT only; value constraint is enforced
on fresh DBs by the CHECK and on all writes by the typed helper.)
- ledger.post_question gains a keyword-only `kind="clarify"` (backward
compatible — existing callers unchanged); PendingQuestion.from_row reads it.
Tests: fresh-DB column+default, idempotent init_db, legacy-DB backfill to
'clarify', plan_decision round-trip. 1396 passed.
The planner's single-shot Claude call intermittently failed with "Reached
maximum number of turns (1)" — it needs slightly more headroom than the
clarifier to finish emitting its JSON. Phase A of the planner-reliability plan:
- plan_node now invokes with max_turns=4 (allowed_tools stays []; the extra
turns buy completion, not exploration).
- build_plan_prompt instructs the model to use no tools and return only JSON
(a tool_use would consume the single turn before the plan is emitted).
- plan_node auto-retries the model call exactly once on a TRANSIENT failure
(turn-cap exhaustion or an empty reply), and fails fast on DETERMINISTIC ones
(malformed JSON, missing/blank phases) — a retry would just reproduce those.
review_loop_llm.py is intentionally GPT-4.1 cross-family (no claude_invoke), so
it gets no turn-budget change. verifier_llm.py does use claude_invoke but is P3
build/verify scope — left for a follow-up.
Tests: max_turns passthrough; retry on turn-cap and on empty; no retry on
malformed JSON; the no-tools prompt line. 1387 passed.
- run-team.py: add the 'dispatch <thread_id>' operator command (P3 option-b).
The read-only box parks at DISPATCH; this completes it with a just-in-time
WRITE token: reads candidate_diff + scope from the checkpoint (or --diff/--scope
files), pushes the head branch + fires workflow_dispatch via dispatch_apply_verify,
prints the located run_id, and (--write-back) writes it into the task checkpoint
so VERIFY binds. +2 tests.
- OPERATOR-RUNBOOK: fix the misleading 'systemctl show -p Environment' check (it
does NOT show EnvironmentFile= vars) -> use /proc/<MainPID>/environ +
_p3_env_is_configured(); document the operator-initiated dispatch flow + the
fine-grained-token write-probe caveat.
Suite green, ruff clean. Branch only; not merged.
The no-write-token detector's test fixtures + a doc comment contained contiguous
'-----BEGIN ... PRIVATE KEY-----' literals that tripped the repo's gitleaks
pre-push backstop (a false positive on a secret-DETECTOR's own test data). Build
the PEM markers at runtime so the source carries no contiguous literal; the
runtime values are still full PEM blocks (what the detector under test sees).
No behavior change; 39 no-write-token tests pass.
High-recall /sh-security-review fan-out + proof-or-kill verifier found two
confirmed HIGH; both now closed (verified empirically against the working tree):
- LOGIC-RACE-01 (HIGH, CWE-835): the build-loop budget was structurally dead
(verifier read a shared wiring-time VerifierConfig.build_loops, always 0, so
the max_build_loops park never fired -> a perpetually-failing task looped
BUILD->DISPATCH->VERIFY forever, force-pushing + firing a CI run each round).
Threaded build_loops through durable PipelineState/TaskRecord; verifier reads
state.get('build_loops',0), writes the incremented count back on each FAIL, and
PARKS at max_build_loops. Parks after exactly N failures, never unbounded.
- SEC-01 (HIGH, CWE-532) + SEC-02 (MED, CWE-214): p3_rollback.sh echoed the live
App JWT to stdout in default dry-run and passed it as a gh argv literal. Added
redact_secrets (Bearer/Authorization/ghX_/PEM masking) through run_or_plan; the
App uninstall now uses curl -H @<0600 tempfile> (JWT never on argv), shredded
after. Empirical: app/incident/all dry-runs leak 0 JWT occurrences.
- SEC-03 (MED, CWE-798): assert_no_write_token now applies the PEM regex + the
configured App-ID to env/config VALUES (not just files) — an App private key
under a benign env name is caught.
- SEC-04 (LOW) + P3-IAC-08 (LOW): tightened the box GITHUB_TOKEN fallback /
value-scan; staged-only WARN on the live workflow revert.
Suite: 1382 passed, ruff clean. Branch only; not merged/deployed.
NOTE: re-verifier flagged SEC-01 as open by grepping COMMITTED blobs (the fix was
uncommitted working-tree state); independently confirmed closed empirically.
GPT-4.1 cross-family review (APPROVE, no critical/high) raised two MEDIUMs on the
rollback tooling; addressed both:
- require_keys: each restore_* asserts its required baseline keys up front and
refuses a PARTIAL (silently-weaker) restore. environment accepts ids OR logins
(equivalent); a missing protection.full now REFUSES the enforce_admins-only
degrade unless P3_ROLLBACK_ALLOW_PARTIAL=1 is set (loud DEGRADED warning).
- out-of-band ACK: an --apply that needs a MANUAL App neutralise (no APP JWT, or
action=out-of-band) refuses unless P3_ROLLBACK_OOB_ACK=1 — so the App is never
left un-neutralised without a conscious operator sign-off; with the ack the
other surfaces still restore.
Documented both env vars in usage. +3 tests (required-key refuse, partial-protection
ack, oob ack). Suite: 1362 passed, ruff clean.
Wire the box-side build->dispatch->verify run identity so the verifier gate
can bind to the CI run the dispatcher triggered:
- task_model: add run_id / ci_correlation_tag / dispatched_at to TaskRecord +
PipelineState (+ dict round-trip).
- dispatcher: RunLocator seam + DispatchResult; dispatch_apply_verify stamps a
dispatched-at watermark, fires, then resolves the run via the workflow
run-name (gh run list; the per-task_id concurrency group makes it
unambiguous). Fails closed to run_id=None.
- dispatch_invoker: persist run_id/dispatched_at/ci_correlation_tag into state.
- workflow: additive run-name surfacing inputs.task_id as the correlation key
(flagged for the C1 /sh-security-review + GPT-4.1 cross-review re-run).
- docs: P3-PHASE0-DESIGN.md records the async-resume design decision.
Part of Phase 0 (feat/agent-team-p3-box-integration). No behavior change on the
default path: P3 wiring is still opt-in/inert.
Two gaps surfaced by a live /new-task (task 9bce78ad): the plan was produced
and approved, but the "plan ready" notice posted top-level (not in the task
thread) and contained no plan to review.
1. THREADING — `run-team.py` `_build_notifiers` exposed `notify(message)` with
no `thread_ts`. The coordinator's `_emit` calls `notify(message,
thread_ts=root)`; that raised TypeError, and `_emit`'s fallback re-posted
TOP-LEVEL. So every lifecycle milestone (plan-ready / parked / failed) landed
unthreaded, despite the coordinator computing the root ts. (The clarifier
QUESTION threaded fine — different path.) Fix: the sink now accepts and
forwards `thread_ts` into the chat.postMessage payload (build_slack_poster
already forwards the key).
2. PRESENTATION — the plan-ready milestone was a bare one-liner. It now posts a
CONDENSED plan (summary + numbered phase names; step detail stays on the
status dashboard) via new `Coordinator._summarize_plan`, so the plan is
actually reviewable in-thread.
Tests: notify sink forwards thread_ts (and omits it for top-level); condensed
plan renders summary + phase names (not steps); malformed plan falls back;
plan-ready milestone threads under the task root AND carries the plan. 1193 pass.
A live task (d30b697c) on the R720 crashed the coordinator: the planner's
single-shot Claude call raised "Reached maximum number of turns (1)", the
exception propagated out of `drain_resumes` through the serve loop, and systemd
restarted the daemon — with no Slack notice, so the failure was silent.
Defense in depth:
1. invoker: `_collect_subscription_text` now tolerates the single-shot turn cap.
When the Agent SDK raises "Reached maximum number of turns" mid-stream it
salvages the assistant text already collected (the JSON the planner needs)
instead of propagating. An empty salvage or any non-turn error still raises.
2. coordinator: `drain_resumes` wraps the per-job `resume()` so an unhandled
node exception fails THAT task instead of the daemon — it supersedes the
answered question (so the startup recovery sweep cannot re-drive it into the
same crash on reboot), marks the task FAILED via `update_state` with a short
`failure_reason`, and surfaces an honest "❌ FAILED" line to Slack.
3. task_model: add `failure_reason` to PipelineState + TaskRecord (kept in sync)
so the terminal-failure detail persists as a real graph channel.
4. resume_worker: add `ResumeOutcome.FAILED`.
Tests: invoker salvage/re-raise/propagate paths; drain_resumes fails-not-crashes,
supersedes the question, notifies, and one failing task does not block others.
1189 passed.
- render_html includes the inline SVG map + inline fetch('/api/state') poller
+ tooltip mount + live clock, stays offline (no CDN), keeps the <noscript>
meta-refresh fallback, and renders the map even on a not-ok snapshot.
- snapshot_to_dict: expected top-level keys, one entry per STAGE in order,
per-phase grouping, awaiting-human classification (open pending_question),
per-node model/agent role labels, summary counts, not-ok serializability.
- Descriptions escaped in BOTH the HTML and the JSON-in-script seed
(</script><script> breakout neutralised; \u003c form present); the raw
/api/state JSON round-trips the description for textContent rendering.
- End-to-end snapshot_to_dict over the seeded ledger.
WS Slack-UX Feature 2. When the inbound listener acts on an answer in a task
thread, it adds a 👍 reaction to that reply so the human sees the machine
received it.
- SlackListener gains an optional reactor seam; handle_event reacts to the
inbound reply message (channel + event ts) AFTER the AUTHZ-01 owner check
passes — a non-owner message is rejected and never reacted to. Best-effort:
any reaction failure (notably a missing scope) is swallowed and never breaks
handle_event or the listen loop.
- /new-task is NOT reacted to (a slash command has no reactable message); its
"📥 Task received" root post is the acknowledgement.
- build_slack_reactor wraps WebClient.reactions_add(name="thumbsup"); the
default listener factory wires it best-effort from SLACK_BOT_TOKEN.
- Adds reactions:write to the bot scopes in agent-team-manifest.json.
NOTE: the new reactions:write scope requires Adam to re-apply the manifest to
app A0BCC7TTU66 and reinstall the app. Until then reactions.add returns
missing_scope, which the listener swallows (the reaction silently no-ops) —
answer handling is unaffected.
AUTHZ-01 and the first-answer-wins CAS remain unchanged.
WS Slack-UX Feature 1. A /new-task task now maps to ONE Slack thread instead of
several top-level messages.
- /new-task posts an immediate root "📥 Task received: …" ack and captures its
ts (root_ts); this is the instant acknowledgement.
- root_ts is plumbed into start: new PipelineState/TaskRecord channel
slack_thread_ts, seeded by graph.start_task and threaded through
Coordinator.start_task. The NewTaskCallback is now (task_text, via, root_ts).
- All clarifier questions for the task post as THREADED REPLIES under root_ts
(chat.postMessage thread_ts=root_ts), and each question's ledger channel_ref
is set to root_ts (NOT the reply's own ts). Because answer-mapping resolves a
reply via find_open_question_by_channel_ref(thread_ts), a reply in the root
thread (thread_ts==root_ts) maps to the task's currently-open question with NO
change to the mapping logic or the first-answer-wins CAS. The open-only
partial-unique index still holds (one open question per task at a time).
- Lifecycle milestones (parked / plan-ready / needs-input) and follow-up
questions thread under root_ts too; the notify sink gained an optional
thread_ts kwarg (degrades to top-level on a sink that doesn't accept it).
notify failures still never break tick.
- SlackTransport.post_question + the live poster accept/forward thread_ts.
- No root_ts (non-/new-task origin) ⇒ top-level posts exactly as before.
AUTHZ-01 (owner-allowlist-first, fail-closed) and the atomic open→answered
compare-and-set are unchanged.
Adds plumbing for the inbound-ack reactor seam used by Feature 2 (dormant until
a reactor is injected). Tests cover thread_ts forwarding, channel_ref=root_ts,
graph seeding, and coordinator threading.
Two root causes behind 'every task parks, and slowly':
1. SINGLE-SHOT INVOKER: subscription Claude calls ran as 40-turn, tool-enabled
agentic sessions (--max-turns 40, $2 budget) for what are pure reasoning->JSON
completions — minutes-long, and Claude wandered/returned unparseable output.
_DEFAULT_MAX_TURNS 40->1 + allowed_tools=[] -> fast deterministic single turn.
2. PLANNER FEEDBACK KEY MISMATCH: the review stage writes verdict/findings, but
_format_review_feedback read decision/notes/comment (never present) -> the
planner re-planned with EMPTY feedback, re-introduced the rejected flaw
('assumptions persist'), hit the review cap, parked. Now reads verdict/findings
(old keys kept as fallback) so GPT-4.1's objections reach the re-plan.
Plus: park notifications infer the phase the task was IN (review/plan/clarify)
instead of the terminal 'parked'.
Tests: planner real-verdict-keys regression + coordinator phase-inference. 1149 pass.
Park notifications were opaque ('Task 6c3c3202 parked — needs your attention'):
no idea what the task is, where it got to, or what's blocking it. Now each
message names the task DESCRIPTION (not just the short id), the PHASE it reached,
and the actual BLOCKER — _summarize_blocker() pulls the last review_verdict's
findings (the GPT-4.1 REQUEST_CHANGES text, collapsed + truncated), falling back
to 'no plan built' / 'revision cap hit'. Applies to parked + needs-more-input +
plan-ready emits. +1 test (description + phase + blocker present). 1148 passed.
The bot only ever posted clarifier questions; parks/completions were silent and
multi-turn follow-up questions were never posted during normal operation (only
the startup recover sweep posted them). So an answered task was a black box.
- Coordinator gains a 'notify' sink + _emit() (guarded, never breaks the loop).
- tick() now runs _post_resume_followups(results) after the drain: for each
resumed thread it (a) POSTS a newly-pending clarifier question (fixes silent
multi-turn — the drain path left it unposted) and emits 'needs more input';
else emits 'parked — needs attention' or 'plan ready for review' from the
settled state.
- run-team serve wires notify -> Slack channel (build_slack_poster) and an
alarm_hook that logs the deadline-park WARNING AND posts a parked notice.
Live-Slack only; dry-run/non-Slack/no-channel = silent (None), no token needed.
Tests: +4 (needs-input/parked/plan-ready emits + notify-failure swallow);
_FakeCoordinator gains notify/alarm_hook. 1147 passed, ruff clean.
/new-task (and every intake: GitHub issue, /sh-assign-task) reached the clarifier
with NO description -> the clarifier asked 'no task description provided'. Root
cause: coordinator.start_task only LOGGED task_text (a P1-era decision when the
deterministic clarifier didn't consume a description), graph.start_task took no
task arg, and PipelineState/TaskRecord had no 'task' channel at all.
Fix: add a first-class 'task' field to PipelineState + TaskRecord (+ round-trip
in task_from_dict); graph.start_task seeds task into the initial invoke (persists
through intake_node's partial-state return into CLARIFY); coordinator.start_task
passes task=task_text. The clarifier already reads state['task'] via
_task_description, so it now sees the real description.
Test: start_task(task='build a login form') -> suspended CLARIFY state carries
task. 1143 passed, ruff clean.
Gap-audit findings:
- G1 (blocks-feature): make_cross_reviewer_invoker / make_fast_coder_invoker did
'from models import' without putting the orchestrator root on sys.path. The
run-team serve daemon only bootstraps agent-team/, so on the live box every
GPT-4.1 plan review hit ModuleNotFoundError -> review_plan's blanket except
silently fail-closed to REQUEST_CHANGES (GPT-4.1 never actually ran). Both
in-process invokers now call invoker_multi._ensure_orchestrator_on_path()
before the deferred import. WS1 introduced this when it swapped the review
default from the subprocess invoker to in-process.
- G4 (degrades): the handbook context_provider was wired into the planner only;
default_clarify_node_factory now accepts + forwards it, and run-team wires it
into build_clarify_node too, so clarifying questions are handbook-aware.
Tests: +2 regression tests (path-bootstrap, clarifier threading); _FakeCoordinator
gains build_clarify_node. 1142 passed, ruff clean.
Integration branch combining WS0-WS5 (PRs #43-#46) + the activation wiring that
flips the safe seams ON in the run-team serve path:
- WS5 (D10): inject the Sea Haven handbook conventions into the planner prompt
via context_provider (zero-arg handbook loader; fail-safe to '' when absent).
- WS2: an allowlisted Slack /new-task starts a task on this coordinator
(set_new_task_callback adapter -> start_task; AUTHZ-01 gates it upstream).
- WS1 bind_multi_invoker() is already wired in _cmd_serve.
Coordinator gains a new_task_callback param + set_new_task_callback() (resolves
the constructor chicken-and-egg of referencing the coordinator's own start_task);
default_slack_listener_factory forwards it to the SlackListener.
NOT wired (deliberately): the P3 dispatch_node / build_verify path. Activating
it correctly needs a per-task expected_run_id bound into gated_build_verify_wiring
(plumbing that does not exist yet) AND the CI trust-boundary security re-review.
It stays inert pending that work.
Tests: +5 activation-wiring tests; _FakeCoordinator stub gains
set_new_task_callback. Full agent-team suite: 1140 passed, ruff clean.
Security follow-up from the per-PR review (non-blocking, defense-in-depth):
- Replace the save_memory name blocklist with an allowlist regex
(^[A-Za-z0-9][A-Za-z0-9._-]*$, max 128) so dot-only/hidden/backslash/NUL/
over-long names are rejected outright, not written as malformed-but-contained
files.
- Write via os.open(..., O_NOFOLLOW): the open fails (ELOOP) if the final
path component is a pre-planted symlink, closing the TOCTOU where a symlink
in _box-drafts/ could redirect the write outside the dir. O_CREAT|O_TRUNC
keeps overwrite-on-resave for regular files.
Tests: adds allowlist-rejection + symlink-refusal cases (21 pass).
Security follow-ups from the per-PR review (non-blocking MEDIUMs):
- Eager _get_token() at make_app build time so a missing AGENT_TEAM_API_TOKEN
fails fast instead of serving requests first (matches the docstring contract).
- Disable /docs, /redoc, /openapi.json (no auth dependency in FastAPI) — the
API is VPN-only/127.0.0.1 and should not expose its schema unauthenticated.
- Scrub raw exception text and subprocess stderr from 500 response bodies;
log server-side instead (avoid internal-path/state disclosure).
- Bound /orchestrator/invoke concurrency with a semaphore (429 over the cap)
so an authenticated caller cannot exhaust the box via many 600s subprocesses.
Also pin fastapi/uvicorn in requirements.txt (WS1 dep). With fastapi now
installed in CI, the previously skip-guarded TestClient tests run for real;
the importorskip guard stays as a no-op safety net.
Tests: 23 pass (adds docs-disabled + concurrency-429 cases).
Reworked per GPT-4.1 cross-family review BLOCK. The original PR removed the
agent-apply GitHub Environment (the live required-reviewer human gate) and
replaced it with a Slack notice that FAILS OPEN when its webhook secret is
absent (which it is) plus an audit-log line. The cross-review correctly
flagged this as trading a preventive control for detective controls, one of
which silently no-ops.
This commit:
- Restores environment: agent-apply on gate-and-pr (the human approval pause).
- Drops the fail-open Slack notify step.
- Keeps the unconditional audit-log step as an additive detective control.
- Restores the MANDATORY-INVARIANT assertion (env must be present) and adds
an assertion that the audit step is retained.
WS3's auto-dispatch (dispatch_invoker.py + graph/coordinator wiring) is
unchanged: it fires workflow_dispatch, which now pauses at the restored gate
for human approval — auto-dispatch up to the approval, then one click.
CI has no fastapi (box-only dependency); api.py imports it lazily. Guard the
7 TestClient smoke tests with skipif(find_spec('fastapi') is None) so they
skip in CI instead of failing collection, leaving the 14 invoker_multi tests
running. Also apply ruff format to the 5 WS1 files CI flagged.
- invoker_multi.py: multi_invoke(prompt, *, model) dispatches to GPT-4.1
(cross_reviewer), DeepSeek (fast_coder), or Gemini (scanner) in-process;
bind_multi_invoker() wires the review seam; lazy models import
- review_loop_llm: swap default_plan_reviewer from make_run_py_invoker()
to make_cross_reviewer_invoker() (in-process GPT-4.1); subprocess path
retained as make_run_py_invoker() for opt-in use
- builders_llm: add make_fast_coder_invoker() in-process DeepSeek path;
rename subprocess path to subprocess_build (opt-in fallback); default_build
now delegates to make_fast_coder_invoker()
- api.py: FastAPI app with bearer-token auth (AGENT_TEAM_API_TOKEN env),
POST /tasks, GET /tasks/{id}, POST /orchestrator/invoke; binds 127.0.0.1;
code only — not started here
- run-team.py: bind_multi_invoker() called in serve() alongside
bind_subscription_invoker()
- tests: 21 new WS1 tests + updated builders_llm tests (1065 total, all passing)
- retriever: add save_memory() writing to _box-drafts/ review queue, add
memory_dir param to retrieve() for isolated test routing
- handbook: new load_handbook_conventions() with safe no-op contract (returns
"" when dir absent/empty/unreadable, never raises)
- clarifier_llm: add context_provider=None seam to ClaudeClarifier and
build_claude_clarifier_callables(); failure in provider is silent
- planner: add context_provider=None seam to build_plan_prompt() and plan_node()
- coordinator: thread context_provider through default_plan_node_factory(),
only pass kwarg when non-None to preserve stub-monkeypatching in tests
- tests: 19 new WS5 tests covering all seams (1063 total, all passing)
Live smoke exposed a self-pollution bug (pre-existing from PR #17): the post-build
denied-path check wrote its own _build_diff_z.bin / _build_status_z.bin into the
working tree, then its own `git status --untracked-files=all` flagged them as
out-of-scope writes — failing any run with a narrow declared_scope (the smoke's
docs/**). Write them to $RUNNER_TEMP instead (read via $_DIFF_Z/$_STATUS_Z), so
the check no longer sees its own temp files. The agent-team pytest artifacts were
already correctly gitignored; only the check's own files tripped it. Test harness
updated to pass the env paths. 1044 tests, ruff clean.
GitHub Actions only runs workflows under .github/workflows/, so the apply/verify
workflow at agent-team/ci/ was never registered (workflow_dispatch 404'd). Move it
to .github/workflows/agent-team-apply-verify.yml so it is a real, dispatchable
workflow. Its only trigger is workflow_dispatch + it is gated by the agent-apply
required-reviewer environment, so it never auto-runs and nothing privileged runs
unapproved. Updated the two workflow test files' path refs (parents[2]/.github/
workflows) and the ci/README pointer. Dispatcher push gains --no-verify: the apply
path is scanned CI-side (guard + the PR's checks), so it must not be blocked by the
operator's LOCAL human-commit pre-push dev hook (which flags pre-existing whole-repo
FPs like .env.example). 1044 tests, ruff clean.
* feat(agent-team): P3-flip Phase 1 — expand denylist vectors (§4.2) + runner-trust assertion (§4.1)
First controls of the P3-live-flip Phase-1 CI hardening (workflow stays INERT;
this only tightens the trust boundary). Whole Phase-1 surface is gated by
/sh-security-review + GPT-4.1 cross-review before any flip.
§4.2 — expand the trust-control denylist with direct code-execution / supply-chain
vectors, kept byte-identical across all three copies (ci_gate.DENYLIST_GLOBS + the
guard + post-build inline DENY_GLOBS), drift-guarded:
.gitmodules, .husky/**, .githooks/**, .gitattributes, .npmrc, and generated/build
artifacts (__generated__, *.generated.*, dist/**, build/**, *.min.js).
Deliberate: lockfiles are NOT wholesale denied — lockfile-postinstall RCE is already
contained by the credential-less egress-blocked build sandbox, and the Tier-3 dep-CVE
fixer rewrites lockfiles to produce its draft PRs; a blanket deny would make it
un-shippable. Flagged in-code for the security gate. Direct code-execution config
(hooks/filters/npmrc/submodules) is the actual §4.2 RCE surface.
§4.1 — runner-trust: assert no job (esp. the privileged gate-and-pr) can run on a
self-hosted/user-provided runner; all must be GitHub-hosted.
998 tests pass, ruff clean.
* feat(agent-team): P3-flip Phase 1 — gate-weakening detector (§4.5)
A diff that ADDS a lint/type/coverage/security suppression (noqa, type: ignore,
pragma: no cover, nosec, nosemgrep), a test skip/xfail, or a hook bypass
(--no-verify) could make CI pass falsely. The pure-code gate now flags these via
gate_weakening_violations() and BLOCKs in evaluate_ci_gate as a top-priority trust
violation (step 1b, alongside the denylist) — regardless of the authenticated CI
conclusion. A build cannot pass itself by disabling its own checks; flagged diffs
escalate to a human. Only ADDED lines are inspected (removing a suppression is fine).
1015 tests pass, ruff clean.
* feat(agent-team): P3-flip — diff transport (§4.3) + flip privileged apply path live
Completes the box->CI diff handoff and flips the apply/verify privileged job
live (gated behind the agent-apply environment's required reviewer).
Transport (§4.3): the read-only box (D2) emits a diff but holds no write token.
- New credential-less `materialize` job decodes the untrusted `diff_b64`
dispatch input via env (CWE-94), fail-closed re-hashes it against
`expected_diff_hash`, and uploads it as the named artifact so guard/build-test
download it same-run. guard now `needs: materialize`.
- New `dispatcher.py` (the trusted apply path, operator/Mac-side — never the
box): pushes the diff as a head branch then `gh workflow run`s the workflow.
Pure input-assembly (sha256 == sha256sum, b64 round-trip, head ref) is
unit-tested; git/gh are injected seams. Push-before-dispatch; fail-closed on
empty diff/scope, unsafe task_id/owner/repo.
Flip: gate-and-pr binds `environment: agent-apply` (required reviewer
amoussa1229) + grants exactly `pull-requests: write`; the App-token + draft-PR
steps run only on `steps.gate.outputs.gate == 'pass'` (no more if:false); the
draft PR opens with an explicit `--head`; task_id/head_branch charset-validated
(§4.6). Updated the hardening tests from inert-state to live-state assertions +
added transport tests. 1039 tests, ruff clean, workflow YAML valid.
NOTE: workflow only runs on manual workflow_dispatch and the privileged job is
held at the required-reviewer gate, so nothing privileged runs unapproved.
* fix(agent-team): P3-flip — address GPT-4.1 cross-review (size bound, ref-traversal guard)
- BLOCK: cap candidate diff at 40 KB in the dispatcher (the diff rides a base64
workflow_dispatch input; GitHub caps inputs at ~64 KB so an oversized diff
cannot dispatch at all) + a defense-in-depth decoded-size bound in materialize.
- FIX: harden the draft-PR HEAD_BRANCH guard to reject leading/trailing slash,
'..' segments, and '//' (CWE-88 git ref-traversal), not just bad charset.
- NIT: document the mandatory invariants on gate-and-pr (required-reviewer
environment must stay; runs-on must stay GitHub-hosted).
- QUESTION (lockfiles): answered in-code — the build-test sandbox is
credential-less + egress-blocked, so lockfile-postinstall RCE is contained.
Tests added for all guards. 1042 tests, ruff clean, YAML valid.
* fix(agent-team): P3-flip — resolve /sh-security-review findings (LOGIC-1/2/3)
High-recall fan-out (injection/logic/iac+secrets) + proof-or-kill on the LIVE
apply path found 3 real issues the cross-review missed; all fixed:
- LOGIC-2 (HIGH, was a live hole): build-test ran `ruff check . || echo` /
`pytest -q || echo`, swallowing failures so the job was always 'success' and
the gate would open draft PRs on RED builds. ruff/pytest now run
authoritatively under set -e (pytest exit 5 'no tests' is the only non-fatal
case); the exit code IS the build-test conclusion the gate keys on.
- LOGIC-1 (verified!=shipped): the dispatcher used `git apply` + `git add -A`,
staging stray untracked content into the pushed PR head. Now `git apply
--index` stages exactly the diff, so the head tree is precisely base+diff —
bound to the bytes CI hash-verified.
- LOGIC-3 (§4.5 on the live path): gate-weakening was enforced only box-side;
added a gate-weakening check to the guard job so the live PR-opening path
rejects a diff that adds suppressions/skips, even on a green build.
Injection / secrets / least-privilege / flip-correctness / no-untrusted-checkout
all came back clean. 1044 tests, ruff clean, YAML valid.
* feat(agent-team): durable GitHub-issue intake de-dup (schema v2)
The intake poller de-duped ingested issues in an in-memory set that does
not survive a process restart. A scheduled/cron intake (each run a fresh
process) would therefore re-ingest every still-open labeled issue on every
run and spawn duplicate pipeline tasks. Since the box is read-only (no write
token to remove the intake label), durable de-dup is the only correct guard.
- schema v2: new ingested_issues(source, issue_id, ingested_at) table +
issue_already_ingested / record_issue_ingested helpers; migrate() adds the
table to a legacy v1 DB and restamps; init_db creates it.
- github_intake: pluggable IngestStore seam (in-memory default preserved for
tests/one-off; durable build_ledger_ingest_store for production). Record is
after start_task succeeds, so a failed intake stays retryable.
- run-team.py intake-github wires the ledger store keyed by github:owner/repo,
making a scheduled timer idempotent across runs.
978 tests pass (ruff clean).
* fix(agent-team): harden github intake per /sh-security-review (CWE-918, idempotency)
Fixes from the high-recall detector fan-out on the durable-dedup change:
- INTAKE-LOGIC-01 (idempotency): switch the IngestStore seam from
check-then-record (seen/mark) to claim-then-do (claim/release). The id is
now reserved BEFORE the non-idempotent start_task side effect, so a crash in
that window cannot re-spawn a duplicate task on the next run; a raising
start_task releases the claim so transient failures stay retryable. Adds
delete_issue_ingested to the schema layer for the release path.
- INTAKE-SSRF-001 / INTAKE-PATHSPLICE-002 (CWE-918) in build_default_issue_client:
drop the caller-overridable api_root (hardcode GITHUB_API_ROOT) and validate
owner/repo against an anchored charset before splicing them into the
token-bearing API URL — mirrors the sibling ci_fetcher BLOCK-3/FIX-3 fixes.
Tests cover cross-process duplicate prevention, release-on-failure retry, and
the owner/repo + api_root rejection. 982 tests pass, ruff clean.
Follow-up (pre-existing, not introduced here): the label-only intake has no
author allowlist (cf. AGENT_TEAM_SLACK_OWNER_IDS on the Slack listener); the
Slack answer gate bounds the blast radius. Track as separate hardening.
Three robustness/hardening fixes surfaced by /sh-security-review on the
agent-team listener/coordinator surface. None alter AUTHZ-01 allowlist
behavior or the first-answer-wins compare-and-set semantics.
1. Respawn close() leak (CWE-772). The watchdog _supervise_slack_listener
respawned the inbound Slack listener without tearing down the dead one,
leaking a Socket Mode WebSocket / SDK thread set per flap. Now the dead
listener is closed before respawn (new _close_dead_listener, idempotent),
AND _run_listener has a finally that always closes the listener so a
crashed serve() releases its socket. SlackListener.close() is idempotent,
so the belt-and-braces close stays a safe no-op.
2. Broadened exception guard in handle_event (CWE-248). The submit block
only caught ValueError; the accept path (submit_answer ->
_question_turn/_question_thread) can raise KeyError on a concurrently
mutated row, and the CAS can raise sqlite3.Error. An uncaught exception
would escape into the Bolt dispatch. Added a separate `except Exception`
that logs at WARNING (not silent, not debug) and returns None. The
existing ValueError-as-debug behavior is unchanged; authorization still
runs first, so the trust boundary is not widened.
3. channel_ref partial-unique index (defense-in-depth). Added
uq_pending_questions_open_channel_ref — a PARTIAL UNIQUE index on
(channel_ref) WHERE channel_ref IS NOT NULL AND status='open' — so two
OPEN rows can never share a non-null channel_ref (a thread_ts can never
map to two open questions). Installed in init_db AND unconditionally in
migrate (idempotent IF NOT EXISTS) so existing v1 DBs gain it. NULLs and
closed rows are excluded; mirrored verbatim into schema.sql.
Tests: +8 (was 960, now 968). New: schema partial-unique reject/null/closed/
migrate cases; handle_event KeyError + sqlite3.Error swallow cases;
coordinator close-before-respawn + run_listener-closes-on-crash. Fixed the
operator-cli test fixture to use a per-question channel_ref (it previously
inserted multiple open rows sharing one ref, which the new index correctly
rejects).
* fix(agent-team): handle real slack_bolt event envelope + map thread replies to open questions
The Socket Mode inbound listener was unit-tested against a SYNTHETIC payload
shape that does not match what slack_bolt actually delivers, so the suite was
green while a real Slack thread reply was silently dropped (the clarifier
question stayed `open`). Real slack_bolt delivers an Events API message /
app_mention as `{"type":"event_callback","event":{"type":"message",...}}` and
a free-text thread reply carries NO callback_id/question_id/metadata.
Three breaks fixed (all on the free-text reply path):
1. Type gate — handle_event gated on the OUTER `type`, which is
"event_callback" for a real message/app_mention, so the event fell outside
_ANSWER_BEARING_TYPES and was dropped. Now collapsed to the discriminating
INNER `event.type` via _discriminating_type / _inner_event.
2. question_id recovery — a real reply has no callback_id/question_id/metadata
(the bot's metadata is on the QUESTION message, not the reply). When explicit
id recovery fails, the listener now resolves the question by the inner
event's `thread_ts` against the OPEN ledger row whose `channel_ref` equals it
(new schema helper find_open_question_by_channel_ref, constrained to
status='open' as anti-replay). Explicit id recovery still takes precedence.
3. answer extraction — a real message event carries its text at `event.text`,
not a top-level `answer`/`text`. The thread-reply path now takes the inner
`event.text` (stripped) as the answer value.
AUTHZ-01 is unchanged and still runs FIRST: authorization gates on the sender's
Slack user id (`event.user` for the Events API shape) and fails closed on an
empty/unknown allowlist or unrecoverable sender. The new mapping only resolves
WHICH question is answered, never WHO may answer. Answers stay opaque DATA
(parameterized SQL + json.dumps; never eval/exec/interpolate).
Tests: replaced the synthetic events-API fixtures with REAL Bolt envelopes and
added regression coverage — real thread reply maps via channel_ref and is
accepted, text is stripped, non-owner reply rejected (row stays open), thread_ts
matching no open row is a no-op, reply to an already-answered row is a no-op
(anti-replay), and app_mention is normalized identically. block_actions /
slash_command paths retained.
* fix(agent-team): bind subscription invoker in the start CLI
`run-team.py start` runs the clarifier graph to the first human gate IN the CLI
process, and the clarifier calls Claude (assess_confidence). The invoker is a
process-local binding that only `serve` set, so `start` failed with
"claude_invoke has no invoker bound". Bind the real subscription invoker here,
mirroring Coordinator.serve(). Found during the live R720 P1 bring-up.
* fix(agent-team): serve() starts the inbound Slack listener (D-1)
Coordinator.serve() now constructs and starts the SlackListener concurrently
with the tick/drain loop on a background daemon thread, but ONLY when the live
transport is a SlackTransport AND SLACK_APP_TOKEN is configured. When Slack is
not the transport or the app token is absent, serve() behaves exactly as before
(tick/recover only) — Slack is never made mandatory.
- New injectable build_listener seam + default_slack_listener_factory sharing
the coordinator's own transport, ledger db_path, and resume_queue put.
- AUTHZ-01 owner-allowlist + open-status CAS untouched: serve() sources
AGENT_TEAM_SLACK_OWNER_IDS in SlackListener.serve, which still fails closed.
- SlackListener.close() added for clean Socket Mode teardown on shutdown;
serve() stops the listener + joins the thread in a finally.
- Tests: start-when-Slack+app-token, no-start otherwise, clean shutdown,
idempotent start, serve start/stop around the loop, listener close().
* fix(agent-team): systemd unit loads ~/orchestrator/.env + uses venv python (D-2/D-7)
D-2: add EnvironmentFile=-/home/adam/orchestrator/.env (optional '-') so the P2
GPT-4.1 review loop's cross_reviewer sub-process can read the non-Claude provider
key once a task reaches REVIEW. Mirrors the sea-haven-secrev unit.
D-7: point ExecStart at the agent-team venv interpreter
(/home/adam/orchestrator/agent-team/.venv/bin/python) instead of
/usr/bin/env python3, which resolved the system interpreter without the
installed deps under systemd's PATH.
All hardening (NoNewPrivileges / ProtectSystem=full / ProtectHome=read-only /
ReadWritePaths) is retained unchanged (locked decision).
* docs(agent-team): land provisioning + operator runbooks under docs/provisioning
- PROVISIONING-RUNBOOK.md: merged final state (6 checkers, dep-bump fixer, P5
intake-checker loop), SLACK_CHANNEL_ID, the gated P3-live flip steps (GitHub
App + agent-apply env + gated_build_verify_wiring), and D-1/D-2/D-7 marked
FIXED so the demo can use the live Slack answer path.
- P1-DEMO-SCRIPT.md: live Slack answer path now available (D-1 fixed); both the
Slack and operator-CLI answer paths documented for all four exit criteria.
- DEPLOY-AUDIT.md: D-1/D-2/D-7 RESOLVED (this PR); D-4/D-5 dep pinning and the
operator-CLI divergence kept as provisioning notes.
- OPERATOR-RUNBOOK.md (new): incident handling for pipeline stalls, parked tasks,
failed HITL resumes, budget exhaustion, transport outages, and
COMPLACENCY/COVERAGE alarms — each grounded in real run-team.py verbs, plus the
re-alarm-backoff -> Jira-after-N-nights escalation ladder (design §5/§6.6).
* fix(agent-team): supervise the Slack listener thread — recurring ALARM + respawn
sh-security-review (logic) MEDIUM: a crashed listener thread was logged once,
then the daemon ran on 'deaf' — posting clarifier questions but receiving no
answers, every gate silently parking, process never exiting so systemd
Restart=on-failure never fired. serve() now calls _supervise_slack_listener()
each pass: when the listener is enabled but its thread is dead, it emits a
recurring ERROR ALARM and respawns via the idempotent starter (self-heal).
No-op when alive or disabled. +3 tests. (authz detector: wiring clean — AUTHZ-01
fail-closed allowlist + open-status CAS intact, dead listener fails SAFE.)
* feat(agent-team): Plane-1 Tier-3 fixer — dependency-cve finding -> patch + CI dispatch (opt-in/inert)
The fixer (design §4 fixer row, §7 Phase 5, §3.3.2) takes a CONFIRMED,
low-risk dependency-cve finding (the narrowest fix class) and produces:
* a fix SPEC (Claude, via the §3.1 billing seam), and
* a minimal bump PATCH (DeepSeek fast_coder, via the orchestrator run.py
path that builders_llm uses),
records the candidate diff + its content-hash, and emits the org-CI
workflow_dispatch inputs (task_id / diff_artifact_name / expected_diff_hash /
declared_scope) for the gate-passed P3-live apply/verify surface.
INERT / opt-in / fail-safe, mirroring build_verify_wiring:
* plan_fix dispatches NOTHING; dispatch_fix has NO default dispatcher
(the box holds no write token, D2) so an un-wired call can never fire a
workflow.
* no git/patch/subprocess/fs-write in executable code — the patch is emitted
as diff TEXT only; CI applies it and opens a DRAFT PR, the box never
applies/pushes/merges.
* untrusted-patch hygiene: the generated diff is confined box-side to the
single dependency manifest (declared_scope) and rejected via
ci_gate.denylist_violations if it escapes scope or touches the
trust-control surface — defense-in-depth with the CI guard.
* bad/ambiguous findings (wrong check/status/category, missing
package/fixed_version, ambiguous fixed_version, unparseable/empty diff)
yield a FAILED no-op plan, never a fabricated fix.
29 new pytest tests under agent-team/tests/test_fixer.py.
* feat(agent-team): run-team.py 'fix --dry-run' subcommand for the Plane-1 fixer
Adds the fixer front door to the operator CLI: load one confirmed
dependency-cve finding from a dependency-cve.json report (--report
--finding-id), plan the fix, and in --dry-run print the spec + patch + the
org-CI workflow_dispatch inputs WITHOUT dispatching anything.
Opt-in/inert: the command binds NO workflow dispatcher and holds no write
token, so even an ok plan only prints; live dispatch is provisioning-gated
(refuses to run without --dry-run). A non-fixable finding prints the
fail-safe reason and exits 1.
4 new pytest tests under agent-team/tests/test_run_team.py.
