Archived

This repository has been archived on 2026-08-04. You can view files and clone it, but cannot push or open issues or pull requests.

Adam Moussa dc2da449fd Plane 2 foundation: interfaces, SQLite schemas, state-store, billing seam

2026-06-17 15:16:12 -04:00

41 KiB

Raw Blame History

R720 Agent Team — Design (v2)

Status: DESIGN LOCKED — ready to build. Not built yet. v5 folded the final plan-review refinements (SQLite BEGIN IMMEDIATE for the compare-and-set §3.3.1; CI denylist defense-in-depth + authenticated-only pass/fail gate + compromised-box honesty §3.3.2; contention reserve + parked-task aging §6.6; backup integrity definition + post-restore reconciliation §6.7; tested rollbacks §7; per-role canaries §6.4; CLI audit/confirm §3.3.1). The design went through 3 GPT-4.1 sh-plan-review cycles (v1, v3, v4); the architecture was stable throughout and remaining grain is now build-time implementation detail captured in the P1/P3 exit gates. No further gate runs by decision; build may begin with Phase 0 / P1. Earlier status: Drafted 2026-06-17. v3 reframes around Adam's clarified north star: the R720 is not just scheduled checkers, it is a self-hosted, human-gated agentic SDLC pipeline (intake -> clarify -> plan -> review -> build -> verify), fed from the Mac harness and tickets, with the scheduled checkers as one task source. v2's roster work becomes "Plane 1"; the pipeline is "Plane 2" and the centerpiece. v4 folds in every v3 plan-review finding: B2 (§3.3.1) and B4 (§3.3.2) resolved with P1/P3 exit gates; B1 billing realism + contention (§6.6); B3/B5/F5 provisioning + rollback + cross-review gate (§7); F1 state durability + backup (§6.7); F2 canary update process (§6.4); F3 service-account lifecycle + F1 backups (§9); F4 runbook incident handling (Phase 6); Q3 escalation ladder (§5); Q1/Q2 in §3.3.1. Ready for a re-run of sh-plan-review. No build until that gate passes and Adam approves.

Extends the sh-secrev pattern (see security-review/DEPLOY-R720.md) from a single security sweep into a small roster of scheduled, unattended agents that check, plan, and (carefully) build, coordinated by a Claude claude -p brain and using GPT / Gemini / DeepSeek where each is the better fit.

This doc is the plan, not a runbook. It deliberately reuses the security agent's proven substrate.

0. Locked decisions (this revision)

#	Decision	Choice
D1	Headless `claude -p` under Max	Permitted for now (Anthropic pushed the disallowing ToS change to a later, unannounced date). Build the auth path swappable so the cutover is a config flip. See `reference_claude_subscription_billing`.
D2	Fixer write path	Option B: the always-on box stays read-only; it emits a patch + opens an issue, and a trusted org CI workflow (OIDC) applies the patch on a branch and opens the draft PR. No standing write token on the box.
D3	Checker/planner output mode	Report + ALARM-only to start. Clean nights post nothing; confirmed criticals alarm Slack; everything else lands in a mode-600 report. No auto-Jira/Notion writes until signal quality is trusted.
D4	Billing/auth resilience	Build a billing-mode abstraction now (subscription OAuth default, API-key / Bedrock fallback ready).
D5	aws-posture	Resident on the box via IAM Roles Anywhere, with a new step-ca internal CA for automated short-lived leaf-cert rotation (no long-lived AWS key on the box).
D6	Confluence write identity	Dedicated `confluence-bot` Atlassian service account, edit scoped to the IT space only (Confluence API tokens inherit the whole user's permissions, so a scoped service account is how we bound blast radius). Token in `~/secrev.env` (mode 600). Costs one Confluence seat.
D7	Confluence agent modes + Mermaid	Scheduled = read + recommend only (gaps/staleness into the report, per D3, never auto-writes). On-demand = SSH-invoked from Adam's Mac for an actual write. Mermaid map edits go through `~/.claude/scripts/confluence_mermaid.py` (ADF-only, dry-run-default, macro-count + revert-diff guarded).
D8	Two-plane architecture	Plane 1 = scheduled checkers (v2 roster), which also act as a task source. Plane 2 = the agentic SDLC task pipeline (the centerpiece). Shared substrate.
D9	Pipeline foundation	LangGraph (the open-source library, runs in-process, NOT SaaS) + a local SQLite checkpointer. Durable, resumable graph; `interrupt()` for the human gate. Reuses the existing stack.
D10	Human-in-the-loop transport	Pluggable, all three adapters (Slack Block Kit, GitHub/Jira ticket comments, Claude Code on the Mac); Adam picks the channel per task. A transport-agnostic notify/resume layer maps answers back to the task thread.
D11	Build/verify execution	Org CI is the primary sandbox. Builders emit candidate diffs; the Option-B OIDC workflow builds/tests/security-reviews; the verifier agent reads CI results. Keeps the 4GB box light + read-only. Dedicated builder VM only if fast local loops prove necessary.
D12	Observability (LangSmith)	Deprecate LangSmith (the SaaS tracer). Keep LangGraph (framework, local). Local JSONL (`telemetry.py`, already exists) is the default; self-hosted Phoenix optional later. No SaaS dependency.
D13	Task intake	Mac harness first (SSH-invoke enqueues onto the box), GitHub issues next (phased).

1. Goal and scope

Stand up a coordinated team of agents on the existing sh-secrev R720 VM that runs unattended on a schedule, operates across every Sea-Haven-Industries org repo automatically, stays cost-bounded, and reports through one alarm channel. Claude (subscription OAuth) coordinates and does deep reasoning; Gemini does broad scans; GPT does adversarial cross-checks; DeepSeek does mechanical code edits.

In scope: read-mostly checkers, a low-blast-radius planner, a resident AWS posture check (D5), and a CI-applied draft-PR fixer (D2). Out of scope: anything interactive or needing back-and-forth, and (per the secrev ethos) any standing write credential on the always-on box.

This does not replace the security review agent; it sits beside it and reuses its plumbing.

2. What we reuse vs. what is new

Reused from sh-secrev as-is: the VM, the systemd-timer model, the OAuth billing path, clean-clone auto-discovery into ~/repo-mirrors (read-only PAT, scrubbed post-fetch; the team scans the same mirrors, it does not re-clone), the budget primitives (per-call + total caps, fail-toward-over-reporting), ALARM-only Slack, mode-600 reports, the anti-complacency canary + coverage-rotation idea, the orchestrator/ rsync deploy, and the non-Claude provider keys in ~/orchestrator/.env.

New: a coordinator runner (agent-team/ dir; entry CLI run-team.py, importable modules stay snake_case per Python rules; the resource/dir name is kebab-case per handbook), per-role prompt/checklist modules each with its own canary, a billing-mode abstraction (D4), the fixer patch -> CI -> draft-PR path (D2), and the step-ca + Roles Anywhere setup for aws-posture (D5).

3. Architecture

systemd timer (shared with secrev — see §8) 
        │
        ▼
  mirror step (reuse sh-secrev discovery) ──► ~/repo-mirrors  (read-only)
        │
        ▼
  COORDINATOR  (claude -p via billing-mode abstraction; read-only tools + Bash to call orchestrator/gh)
        │  shared budget ledger + versioned rotation/coverage state
        ├──────────────┬───────────────┬────────────────┬──────────────┬───────────┐
        ▼              ▼               ▼                ▼              ▼           ▼
   drift checker   dep/CVE checker  doc-drift checker  aws-posture    planner   (each role
   (Gemini scan    (Claude +        (Gemini large-     (Sonnet via    (Claude)   has its own
    + Claude judge)  GPT tiebreak)   context)           Roles Anywhere)           canary)
        │              │               │                │              │
        └──────────────┴───────────────┴────────────────┴──────────────┘
                                   │ structured JSON per agent
                                   ▼
                          coordinator: dedup + prioritize + route
                                   │
        ┌──────────────────────────┼───────────────────────────┐
        ▼                          ▼                            ▼
   Slack ALARM            mode-600 report                 fix-spec queue (D2)
   (confirmed crit)       (everything else;                     │
                           no auto-Jira/Notion yet, D3)          ▼
                                                    FIXER: DeepSeek edit + Claude
                                                    spec + GPT review ──► patch + issue
                                                          │
                                                          ▼  org CI (OIDC) applies patch,
                                                             opens DRAFT PR, runs pre-push
                                                             hooks + CI gates + Claude Code App

Model assignment (matches how orchestrator already splits them): Claude (subscription) = coordinator, deep checks, fix-spec authoring; Gemini 2.5 Pro = broad whole-repo scans; GPT-4.1 = adversarial cross-check / tiebreak / PR review; DeepSeek = mechanical patch writing.

3.1 Billing-mode abstraction (D4)

A single claude_invoke(...) seam selects the Claude auth/billing path from config: subscription (OAuth token, default today), api (metered ANTHROPIC_API_KEY), or bedrock (cross-account Bedrock, already used by secrev for the rare cross-family tiebreak). Switching modes is a config flip, not a code change. The box still pops any stray ANTHROPIC_API_KEY in subscription mode so OAuth cannot be silently overridden.

3.2 Relationship to the LangSmith orchestrator (what stays vs what the R720 hosts)

Two orchestrators coexist after this plan; neither replaces the other. The split is trigger + Claude billing, not capability.

LangSmith orchestrator (Mac-hosted, unchanged). The existing orchestrator/ (LangGraph router + memory retriever + Composio connector + LangSmith tracing on the orchestration project) stays the on-demand, interactive delegation path: one task -> retrieve memory -> route -> one agent -> result, API-billed. This is the CLAUDE.md hybrid-delegation path Claude Code uses for cross-family review, large scans, fast coding, and connector actions. It stays one-shot and stateless (Q1: not refactored for persistence).
R720 orchestrator (the agent-team coordinator, new). The scheduled, unattended, multi-agent layer: cadence, the coordinator brain, shared budget ledger, versioned rotation/coverage state, the clean-clone mirror corpus, and per-role canaries. Claude work here runs headless under subscription OAuth (Agent SDK), billing-mode-swappable (§3.1).

Component	Today (LangSmith orchestrator, Mac)	After this plan
Trigger	On-demand from Claude Code / CLI	+ scheduled (systemd timer) and SSH-invoked on-demand, on the R720
Execution shape	One task -> one agent (stateless)	+ multi-agent coordination with shared budget + versioned state (R720)
Claude billing	Metered API key	Subscription OAuth on the R720 (swappable to api/bedrock per §3.1)
Non-Claude (GPT-4.1 / Gemini / DeepSeek)	`run.py` router, API-billed, LangSmith-traced	unchanged in shape — the R720 coordinator calls the local `~/orchestrator/run.py` (already rsync'd to the box) for these single-shot sub-tasks, so they keep API billing + LangSmith tracing
Memory retriever + embeddings cache	Mac	reused read-only by both (the box's rsync'd copy embeds the same memory store)
Composio connector (Slack/Notion/GitHub)	Mac	reused; the R720 routes Slack/Jira/Notion through it. Confluence stays OUT of the connector — native Atlassian MCP on the Mac for interactive edits, `confluence_mermaid.py` + REST for the box
Observability	LangSmith SaaS tracing (`orchestration` project)	LangSmith deprecated (D12) — local JSONL (`telemetry.py`) default, self-hosted Phoenix optional. LangGraph framework stays (it is not SaaS)
`models.py` factories + model-ID constants	Mac	shared code (rsync'd); single source of truth for both

What does NOT migrate (stays Mac / interactive): the daily-driver Claude Code sessions, the hybrid on-demand delegation, and interactive Confluence edits via the native Atlassian MCP.

What is genuinely NEW on the R720 (not a migration — these never existed in the LangSmith orchestrator): scheduling, the coordinator + shared state/budget, canary/coverage, and subscription-OAuth Claude.

Net: the LangSmith orchestrator keeps its job (on-demand routing, non-Claude execution, tracing, connector); the R720 becomes the host for everything scheduled, stateful, and subscription-billed, and it reuses the LangSmith orchestrator in place (the local rsync'd copy) for the non-Claude single-shots rather than re-implementing them.

3.3 Plane 2 — the agentic SDLC pipeline (the centerpiece)

A durable, human-gated task pipeline hosted on the R720. A task is a long-lived, resumable record; the coordinator drives it through stages, asking Adam for input when it is not confident and handing off to the org CI to actually build and verify.

INTAKE ─► CLARIFIER ─► PLANNER ─► REVIEW LOOP ─► BUILDERS ─► VERIFIERS ─► draft PR + report
  │          │            │            │              │            │
Mac harness  asks Adam    phased plan  GPT-4.1 +      Claude spec  org CI builds/tests/
(SSH-invoke) question-    (Claude)     multi-model    + DeepSeek   security-review;
GitHub issue sets until                adversarial;   edits ─►     verifier reads results;
checker      98%+, then                loops back     candidate    loops back to builders
finding      HUMAN GATE   	          to planner     diff         on failure

Stages and model per stage:

Intake — a task enters from the Mac harness (D13, first), a GitHub issue (next), or a Plane-1 checker finding. It is written as a new task record (LangGraph thread) with a unique thread_id.
Clarifier (Claude) — gathers context (repo, memory, handbook), then asks Adam question-sets until 98%+ confident. This is a LangGraph interrupt(): the task suspends and checkpoints, a question-set is delivered over the chosen transport (D10), and the task resumes via Command(resume=...) when the answer arrives. The human gate: no progression to build without the clarifier clearing the bar and Adam approving the plan.
Planner (Claude) — produces a phased plan (the format these design docs use).
Review loop (GPT-4.1 + optional multi-model) — adversarial plan review (the sh-plan-review / cross_reviewer discipline). Loops back to the planner on REQUEST CHANGES; escalates to Adam if it cannot converge.
Builders (Claude spec + DeepSeek edits) — turn the approved plan into a candidate diff. They do not write to repos; per D2/D11 they emit the diff for CI.
Verifiers (org CI + a Claude/GPT reader) — CI (Option-B OIDC) applies the diff on a branch, builds, runs tests + the security review + lint; the verifier agent reads the CI results and either loops back to builders or advances. Confirmed pass produces a draft PR plus a report to Adam.

Durable state (D9). LangGraph (local) + a SQLite checkpointer. Each stage transition is checkpointed, so a crash, a budget pause, or an overnight wait on a human answer all resume cleanly instead of restarting. The task record holds: status, current phase, the full Q&A history, the plan, review verdicts, the candidate diff, and CI results.

Human-in-the-loop (D10). A small transport-agnostic responder service owns the notify+resume seam: it posts the interrupt's question-set to the channel Adam chose for that task (Slack Block Kit / ticket comment / a Claude Code session) and maps his reply back to the right thread_id to resume it. Adapters are independent so one can ship first (Slack) and the others follow.

Stability + autonomy bounds (the "stable" requirement). Hard gates, not vibes: (1) no build before the clarifier hits 98% AND Adam approves the plan; (2) draft PRs only, never auto-merge; (3) the verifier must pass or the task loops/holds, never ships; (4) per-task budget cap inside the shared nightly cap (§6.1); (5) every stage checkpointed so failures resume, not restart; (6) a task that stalls (no human answer within a window, or N failed build loops) parks and ALARMs rather than spinning. Plane-1's canary/coverage discipline applies to the pipeline's agents too.

3.3.1 Durable human-in-the-loop suspend/resume (resolves B2)

LangGraph interrupt() + the SQLite checkpointer suspend and resume the graph, but the checkpoint alone does not track the human-interaction lifecycle (delivery, duplicate/late answers, expiry). So the pipeline adds one durable source of truth, a SQLite pending_questions table, and resolves every race with an atomic compare-and-set against it. This is the riskiest mechanic, so it is specified here and P1 must prove it.

Identity. Each task is a graph thread_id. Each question-set gets a question_id (uuid) and a monotonic turn within the task. The interrupt payload carries {thread_id, question_id, turn, question_set, transport, deadline}.
Ledger. pending_questions(question_id PK, thread_id, turn, status[open|answered|expired|superseded], transport, channel_ref, posted_at, deadline_at, answer_json, answered_at, answered_via). The LangGraph checkpoint holds graph state; this table holds the question lifecycle and is what delivery, the responder, and recovery read.
Delivery (and lost-post). On interrupt, write the row open first, then post to the chosen transport and store its channel_ref (Slack message ts / issue-comment id / Claude session id). The posted question embeds the question_id (Slack callback_id; a  marker in a GitHub comment). If the post fails, the row stays open with no ref and a reconcile loop retries idempotently.
Answer mapping + idempotency (first-answer-wins). Each transport's inbound adapter normalizes an answer to (question_id, answer, via). The responder then runs one atomic statement: UPDATE pending_questions SET status='answered', answer_json=?, answered_via=? WHERE question_id=? AND status='open'. rowcount 1 = first valid answer, enqueue a resume job; rowcount 0 = the question was not open (already answered/expired/superseded), so the answer is a duplicate or late and is ignored with a "already closed" reply. This single compare-and-set makes duplicate clicks, transport redelivery, answers via two channels, and answer-after-timeout all safe. The statement runs inside a BEGIN IMMEDIATE transaction (SQLite's default deferred isolation does not serialize concurrent responders, so the check-and-set must take the write lock up front).
Resume (single-flight, turn-guarded). A resume worker serializes per thread_id and calls graph.invoke(Command(resume=answer), {configurable:{thread_id}}). Before resuming it checks the live checkpoint is still interrupted on this turn; if the graph already advanced (stale/redelivered job) it marks the question superseded and skips. A resume can never double-apply.
Deadline / no-answer. Each open question has deadline_at. A timer loop flips overdue open rows to expired (same compare-and-set) and applies the task policy: park + ALARM Adam, or apply a defined default answer. An answer arriving for an already-expired question loses the compare-and-set and is ignored. Timeout vs answer is a deterministic race on flipping open.
Restart recovery. All state is durable (both SQLite stores), so a reboot converges via a startup sweep: retry delivery for open rows lacking a ref; re-enqueue resume for answered rows whose graph is still interrupted on that turn (idempotent via the turn guard); run the deadline policy for overdue open rows. No in-memory-only state.
Parallel tasks + isolation (Q1). Each task is its own thread_id with its own checkpoint and ledger rows; the resume worker serializes per thread but runs different threads concurrently within the budget cap.
Manual path. A small CLI over the ledger lets an operator list open/parked questions, re-deliver, force-expire, or answer on a task's behalf; a stuck task parks rather than spins. Destructive CLI actions (force-expire, answer-on-behalf, force-resume) are audit-logged and require an explicit confirmation flag.
Transport seam (Q2 fallback). A Transport interface (post_question(...) -> channel_ref, parse_answer(raw) -> (question_id, answer, via)) with Slack / GitHub / Claude Code adapters; the ledger + resume logic are transport-independent. Adam picks the channel per task at intake. If the chosen transport is unreachable, reconcile retries and, after N failures, falls back to a Slack ALARM pointing at the task.

3.3.2 CI-as-verifier trust boundary (resolves B4)

The builders are semi-trusted at best: an LLM that read repo content can be wrong or prompt-injected, so the candidate diff is treated as untrusted code. The threat is that executing it in CI with org credentials lets a bad diff exfiltrate secrets, assume the deploy role, or tamper with other repos. Five boundaries bound it:

Split CI: untrusted execution is credential-less; privileged steps never see the patch. The job that checks out and runs the diff (install/build/test) runs with permissions: contents: read, no secrets, no OIDC, no write token, and restricted network egress. The patch executes only here, where there is nothing to steal and nothing to assume. Any privileged action (the OIDC role, authoritative status, opening the PR) runs in a separate job that does not check out or execute patch-controlled code; it consumes the build/test report as data only. This is the standard untrusted-code-in-CI ("pwn request") mitigation, so the workflow must NOT use pull_request_target with a checkout of the head ref.
The patch may not touch the trust-control surface. A box-side check and a CI guard both reject any candidate diff that modifies .github/workflows/**, IAM/policy/permission IaC (CDK/SAM), branch-protection / CODEOWNERS / Dependabot config, or files outside the task's declared scope. Such a diff is escalated to mandatory human review + GPT cross-review, never auto-built (those files are the mandatory-cross-review surface regardless). The match is not naive: enforcement is a CI-side hard fail (not only the box check), it resolves symlinks and canonicalizes paths, and it rejects renames into denied paths and build steps that generate files into them, so a path match cannot be bypassed by indirection.
Diff integrity, box to CI. The builder hashes the candidate diff and records the hash in the task ledger. The box has no write token (D2), so the diff reaches CI as a signed artifact / via a short-lived single-purpose token that can only create the agent branch; CI verifies the diff hash matches the ledger before applying. A tampered or substituted diff fails the hash check.
Pass/fail is a pure-code gate over authenticated CI results, not the LLM verifier. Mirroring secrev's "one pure-code script owns the block decision," a deterministic gate reads the CI run conclusion via the GitHub Checks/Actions API (authenticated as the box read-only PAT), keyed to the specific run id + diff hash. It consumes only that authenticated, patch-independent conclusion; it never trusts a success/failure file or artifact the patch could have written. The verifier agent only reads failures to propose the next fix; it cannot declare success.
Branch protection is the final enforcement. The draft PR targets a protected branch; the locked-down build/test/security-review are required status checks; merge needs them green plus the security-review, the Claude Code App review, and human approval. The agent never auto-merges (D2).

The CI apply/verify workflow is itself IaC + untrusted-input handling, so before it ships it goes through /sh-security-review and the mandatory GPT-4.1 cross-review (in P3), and its actions are SHA-pinned per the handbook Pinning Principle. Provenance (diff hash, CI run id, checks consumed, gate decision) is recorded per task; a diff that tried to touch the denylist, a build job that attempted egress, or a checks/hash mismatch is an ALARM. Egress restriction in the untrusted job is enforced at the runner level (a harden-runner-style egress policy or a self-hosted-runner network deny), but the primary mitigation is that the job holds no secrets, so a successful exfil yields nothing of value.

Threat-model honesty (the diff-hash does not cover a compromised box). The hash protects the box->CI transit (tamper/substitution), not a box that is itself compromised: the box is in the trusted computing base for what it proposes. The real backstops against a bad box are the credential-less build job, the trust-control-surface denylist, and above all the human review of the draft PR plus required checks (nothing merges without Adam). A signed-artifact/attestation step hardens transit further but is not claimed to defend a compromised box.

4. Roster

Tier 1 — checkers (read-only)

Agent	Models	Cadence	Output / gate
compliance-drift	Gemini scan + Claude judge	nightly	Drift vs engineering-handbook (naming, secrets placement, CI/CD present, Dependabot, branch protection). Report + Slack ALARM on violations (no auto-Jira yet, D3)
dependency-cve	Claude + GPT tiebreak	nightly	Cross-ref lockfiles vs advisories org-wide; report + feed fixer. Complements Dependabot
doc-drift	Gemini (large context)	weekly	Flags repos whose architecture moved but Confluence/README did not

Tier 2 — aws-posture (resident, D5) + planner

Agent	Models	Cadence	Output
aws-posture	Sonnet collectors + judge	weekly	Idle/anomalous spend (≈$330/mo flagged) + reasoning layer over baseline findings. Auths via Roles Anywhere (short-lived leaf certs, auto-rotated by step-ca). Complements existing GuardDuty/Security Hub/Config, does not replace them
plan-groomer	Claude	weekly	Drafts a groomed weekly plan into the mode-600 report for now (D3); auto-write to Notion/Jira is a later toggle once trusted
confluence-doc	Gemini scan + Claude judge	weekly (scheduled) + on-demand	Scheduled: diffs repos + AWS inventory + the page-ID map (`project_confluence_migration`) against Confluence, reports doc gaps / stale pages / missing runbooks (recommend-only, D3). On-demand (SSH-invoked): performs an actual update, including Mermaid map edits via `confluence_mermaid.py`. Writes as the IT-space-scoped `confluence-bot` (D6). Overlaps the existing `sh-confluence-audit`/`sh-confluence` skills; the box adds unattended cross-repo scope + the tested Mermaid script

Tier 3 — fixer (D2)

Agent	Models	Trigger	Output
fixer	DeepSeek edit + Claude spec + GPT review	on a confirmed, low-risk finding	Emits a patch + opens an issue; org CI applies it and opens a draft PR. Never auto-merges; pre-push hooks + CI + Claude Code App gate it (F3)

5. Coordination model

Nightly, after the mirror refresh, the coordinator: (1) loads the shared budget ledger and the versioned rotation/coverage state (F1); (2) runs the canary suite first, one planted-fault corpus per role, a miss is a COMPLACENCY ALARM and that role is skipped; (3) fans out the scheduled agents over the mirrors, each with a per-call cap, all drawing from one shared total cap (critical for the Claude subscription draw, §6.1), with budget-exhausted roles deferred via the rotation pointer (never dropped) and a COVERAGE ALARM if a role slips past MAX_CYCLE_NIGHTS; (4) collects each agent's structured JSON, dedups across agents, prioritizes; (5) routes per D3: Slack ALARM for confirmed criticals, everything else to the mode-600 report, and fix-specs to the fixer queue if Tier 3 is enabled; (6) a fully clean night posts nothing.

Escalation ladder (resolves Q3). A confirmed critical that stays unaddressed escalates beyond a one-shot Slack ALARM: it re-alarms on a backoff each night it persists, and after N nights (default 3) the coordinator opens a tracking Jira ticket (INFRA) so it cannot quietly linger. The same ladder applies to a COMPLACENCY or COVERAGE alarm that does not clear. Escalation stays ALARM-only in spirit (nothing posts on a clean state).

The coordinator holds its own state; it does not rely on the orchestrator (one-shot, stateless). It may call orchestrator/run.py for GPT/Gemini/DeepSeek single-shot sub-tasks, or call those providers directly (Q1: the team implements its own coordination; the orchestrator is not refactored for persistence).

6. Constraints and how this revision answers them

6.1 Subscription billing draw — shared cap (B1 resolved by D1/D4)

Headless under Max is permitted for now (D1). Every Claude SDK call still draws from the same Max pool as interactive Claude Code, so the team runs under one shared nightly cap across all agents, and pushes volume to Gemini/GPT/DeepSeek (own-account billing) where quality allows. The billing-mode abstraction (D4) lets us flip to api/bedrock when the ToS cutover lands. ANTHROPIC_API_KEY stays unset in subscription mode.

6.2 Fixer write path (B2 resolved by D2)

Box stays read-only; CI applies the patch and opens the draft PR. The CI workflow + its OIDC role is new IAM and goes through the mandatory GPT-4.1 cross-review before it is built (§7, B3).

6.3 step-ca + Roles Anywhere (D5)

New internal CA (step-ca) issues short-lived leaf certs auto-renewed by a systemd timer; the Roles Anywhere trust anchor + the read-only AWS role are new IAM and go through the mandatory cross-review before build (B3). The leaf is short-lived (self-expiring), which is stronger than the box's long-lived GitHub PAT.

6.4 Anti-complacency per role

Each agent ships with its own canary corpus, versioned in the repo. A role with a failing/stale canary is skipped with a COMPLACENCY ALARM, never run silently degraded. Canary update process (resolves F2): canary corpora are version-controlled per agent/role (no shared global corpus, to avoid cross-role confusion); a change to any canary goes through a PR + review with a revert point, and because the canary runs every night, a canary edit that silently weakens recall is itself caught on the next run.

6.5 Host capacity

4GB / 2 vCPU / 40GB. Work is I/O-bound, but more report history + step-ca may pressure disk. Re-check headroom after Phase 1; size up the Hyper-V VM (snapshot first per feedback_ec2_replacement_snapshot discipline) if needed rather than risking the secrev workload.

6.6 Claude budget realism + contention (resolves B1)

A multi-stage pipeline draws far more Claude than a single sweep (clarifier loop + planner + verifier-read per task), all on the same Max pool as Adam's interactive Claude Code. Three controls:

Pre-build measurement (gate before P3 builds anything). Using the existing telemetry.py token capture, measure the per-stage Claude token draw on a representative task plus the worst-case clarifier loop, then project per-task and daily aggregate at expected task volume. P1/P2 must emit these numbers before P3 proceeds; the design measures, it does not assume.
One shared daily Claude cap across ALL R720 Claude work (pipeline + Plane-1 sweeps), tracked in the persistent budget ledger. Non-Claude stages are pushed to GPT/Gemini/DeepSeek (own-account billing) to keep the Claude draw down.
Interactive-first contention rule. Adam's interactive Claude Code is never blocked. The box keeps a reserve headroom; before starting a stage it checks remaining headroom, and if below the reserve it parks new pipeline tasks and ALARMs rather than competing for the pool. A task already mid-flight checkpoints and pauses at the next stage boundary (never killed). A clarifier is capped at N turns per task, then escalates/parks, so an ambiguous task cannot loop-drain the pool.
Fairness + no starvation. The reserve is a fixed configured fraction of the daily cap (not guessed at runtime). Parked tasks are FIFO-aged with a MAX_PARK window; a task that exceeds it escalates (ALARM, and a Jira ticket per §5) rather than starving silently, and an operator can force-resume or re-prioritize it via the CLI. A clarifier parked at the turn cap is resumable the same way: Adam adds context and re-opens it, so it is never an indefinite stall.

6.7 State durability + backup (resolves F1)

All durable state (the LangGraph SQLite checkpoint, the pending_questions ledger, the budget ledger, the Plane-1 rotation/coverage pointer) is written atomically (write-temp-then-rename), integrity-checked on load, and included in the nightly offsite backup (mode 600). On corruption the coordinator refuses to proceed silently: the rotation pointer is rebuildable from the report history, and a corrupt task checkpoint parks that task with an ALARM rather than restarting it blindly. "Integrity-checked" is concrete: schema-version match + a stored content hash + a logical-consistency check (e.g. no answered question whose graph is already past that turn). After a restore, a reconciliation step re-syncs against external state (in-flight CI runs, current GitHub PR status) before any task resumes, so a restored backup cannot act on stale external assumptions.

7. Phased rollout (re-sequenced for provisioning order, rollback, cross-review, and docs-as-you-go)

Phase 0 — substrate factoring (with rollback, B4). Back up nightly_sweep.sh (tag a revert point); extract discovery/mirror/budget-ledger/rotation/Slack/canary into a shared module used by both secrev and the team. Gate: secrev passes its canary + existing behavior after the refactor, else revert. No team behavior yet.
Phase 1 — one checker end to end. Build compliance-drift + its canary + the mode-600 report path + a routing dry-run (F4) for the Slack alarm. Dry-run on the mirrors. Proves the substrate generalizes. Create the project_r720_agent_team memory now (B5/docs-as-you-go).
Phase 2 — coordinator + second checker. Add the coordinator (shared budget, dedup, versioned state F1) and dependency-cve. Run a forced budget-squeeze dry-run to prove deferral-not-drop + COVERAGE ALARM (F2).
Phase 3 — doc-drift + step-ca/Roles Anywhere + aws-posture. Stand up step-ca and the Roles Anywhere trust anchor + read-only AWS role; cross-review the IAM before building the agent (B3). Wire aws-posture. Add doc-drift.
Phase 4 — planner + confluence-doc. plan-groomer writing into the report only (D3). For confluence-doc: create the confluence-bot service account with IT-space-only edit rights (D6), put its token in ~/secrev.env; ship the scheduled gap-detection (read-only, recommend) first, then wire the on-demand SSH-invoked write path. The Mermaid script (~/.claude/scripts/confluence_mermaid.py, already written and offline-tested) must pass a live dry-run against page 1540098 (verify it lists all 16 weweave macros and that a no-op set is clean) before any --apply. Notion/Jira auto-write stays a later toggle.
Phase 5 — fixer (D2). Build the org CI apply-and-open-draft-PR workflow; cross-review its OIDC IAM before building (B3). Confirm fixer PRs hit pre-push hooks + CI + Claude Code App (F3). Start with the narrowest fix class (dep bumps). Draft PRs only.
Phase 6 — document as standing infra. Update Confluence (the team and the still-undocumented secrev host) in the IT host/LAN inventory; write the operator runbook. The runbook must cover incident handling (resolves F4): pipeline stalls, stuck/parked tasks, failed human-in-the-loop resumes, budget exhaustion mid-pipeline, transport outages, and COMPLACENCY/COVERAGE alarms, each with the manual CLI recovery steps (§3.3.1) and the escalation ladder (§5).

Provisioning + rollback + cross-review gate (resolves B3/B5/F5), applied to every phase:

Cross-review is a hard gate, not a note. Any new IAM role/policy, trust anchor, OIDC role, or permission change is provisioned AND passes the mandatory GPT-4.1 cross-review (plus /sh-security-review where it touches the CI / untrusted-input surface) before any code that depends on it is built. A phase cannot start its dependent work until that review is recorded.
Every stateful phase has a revert point. Not just Phase 0: before standing up step-ca, the Roles Anywhere trust anchor + AWS role, or the CI apply-workflow, capture a documented rollback (remove the role/CA, restore the prior workflow, revert the cert config) and gate the phase on a successful dry-run. The rollback is exercised (sandbox or simulated teardown/re-provision), not merely written, before the phase is accepted. VM changes snapshot first per feedback_ec2_replacement_snapshot.
Docs land with the change, enforced as definition-of-done. A phase is not "done" until its memory entry and the relevant Confluence page are updated; that update is a checklist item in the phase, not deferred (Phase 6 is only the final standing-infra writeup).

7.1 Pipeline track (Plane 2) — depends only on Phase 0 substrate

This track is largely independent of the Plane-1 checker phases (1-6); both build on the Phase 0 substrate. Given the north star, Adam may prioritize this track first. Sequencing within it:

Phase P1 — skeleton + the human gate. LangGraph graph + SQLite checkpointer on the box; one trivial task type; Mac SSH-invoke intake; the clarifier with interrupt()/resume over one transport (Slack first). Stops at an approved plan, no build yet. This proves durable suspend/resume across a real human answer (the riskiest mechanic) before anything else. Exit criteria (must demonstrate §3.3.1): (a) kill the box mid-wait and have the task resume after restart; (b) submit a duplicate answer and confirm it no-ops; (c) submit an answer after the deadline expired and confirm it is rejected and the task parked; (d) two tasks suspended concurrently resume independently to the correct thread.
Phase P2 — planner + review loop. Wire the planner and the GPT-4.1 review loop (reuse cross_reviewer), including loop-back and the escalate-to-Adam path.
Phase P3 — builders + verifier via org CI. Builders emit a candidate diff; the Option-B OIDC workflow builds/tests/security-reviews; the verifier reads CI results and produces a draft PR. Start with the narrowest task class (e.g. a dependency bump or a single-file fix), draft PRs only. Build the §3.3.2 trust boundary: split untrusted/privileged CI jobs, diff-hash integrity, the trust-control-surface denylist, and the pure-code pass/fail gate. The CI apply/verify workflow + its OIDC role go through /sh-security-review AND the mandatory GPT-4.1 cross-review before this phase ships (it is IaC/IAM + untrusted-input handling).
Phase P4 — more transports + GitHub intake. Add the ticket-comment and Claude-Code responder adapters (D10) and GitHub-issue intake (D13).
Phase P5 — checker findings as a task source. Let a confirmed Plane-1 finding open a pipeline task, closing the loop between the two planes.

Observability for both planes (D12): LangSmith stays off; the existing local JSONL (telemetry.py) covers the LangChain/LangGraph path, the Agent-SDK path keeps its own run logs, and self-hosted Phoenix is an optional later add if per-run trace UI is wanted.

8. Open items folded in (no longer blocking)

Shared vs separate timer: shared with secrev (one discovery/mirror pass, one shared budget); error isolation handled by per-role try/skip + canary, documented in the runbook (N2).
Read-only PAT sufficiency (Q2): confirm the existing fine-grained PAT covers all mirrors before Phase 1; it already clones every non-archived org repo for secrev, so this is a verification step, not a change.

9. Obligations on build (per global instructions)

Memory: project_r720_agent_team created in Phase 1; cross-link project_security_review_agent, project_orchestration_migration, reference_claude_subscription_billing, feedback_cloudwatch_alarms.
Confluence: document the team (and secrev) as standing infra (always-on VM holding read-only org PAT + now a Roles Anywhere AWS identity).
Handbook/naming: kebab-case dirs/resources (agent-team), snake_case importable Python modules; secrets in .env/Secrets Manager/~/secrev.env (mode 600), never committed; CI/CD for the fixer apply-workflow.
Cross-review: the fixer CI OIDC role and the Roles Anywhere trust anchor + AWS read role each go through the mandatory GPT-4.1 cross-review before their phase builds.
Service-account lifecycle (F3): the confluence-bot token is rotated on a schedule (90 days, calendared like the GitHub PAT), has a documented revocation step, keeps its edits attributable in Confluence page history, and is decommissioned if the agent is retired.
Backups (F1): the durable state stores (checkpoint, ledgers, rotation pointer) are included in the nightly offsite backup.

41 KiB Raw Blame History