orchestrator/docs/provisioning/P3-LIVE-FLIP-PLAN.md

P3-LIVE-FLIP PLAN — agent-team build → verify → draft-PR

Formal phased plan to take the agent-team Plane-2 pipeline from clarify+plan only to producing reviewable draft PRs, while keeping the always-on R720 box read-only and the apply path zero-AWS.

Status as of 2026-06-23: PARTIALLY LIVE. The split-CI apply/verify workflow and its provisioning (the scoped GitHub App, the agent-apply environment + Actions secrets, and the dispatched runs) are LIVE since 2026-06-22 — i.e. the workflow-authoring + provisioning phases below (Phases 1/1b/2) are DONE. The remaining work is the box-side build → dispatch → verify integration on feat/agent-team-p3-box-integration (BUILD → DISPATCH → VERIFY wiring; operator-initiated dispatch; run-name correlation for run_id capture; async CI-watch resume-on-complete; fail-safe serve default) — see docs/P3-PHASE0-DESIGN.md for the recorded design decisions. The two remaining human gates are: (C1) re-run /sh-security-review + the GPT-4.1 cross-family review against the enabled workflow

• the bound box-side wiring (Phase 3 / B5), and (D) deploy → smoke-test → merge (Phase 4). This plan was the input to /sh-plan-review before the build began; that loop is complete.

Prerequisite reading: docs/r720-agent-team-design.md §3.3.2 (CI-as-verifier trust boundary, "B4"), PROVISIONING-RUNBOOK.md (the P3-live-flip section), and memory project_r720_agent_team (locked decisions).

---

1. Objective & current state

Live infrastructure (since 2026-06-22): the split-CI apply/verify workflow (agent-team/ci/agent-team-apply-verify.yml) is authored, enabled, and provisioned — the scoped GitHub App (pull-requests:write + minimal contents), the agent-apply GitHub Actions Environment (Adam as required reviewer + branch protection), and the App credentials as Actions secrets all exist, and dispatched runs have executed against it. Org CI can build/test/security-review an untrusted diff in a sandbox, a pure-code gate confirms green from authenticated Checks-API results, and the privileged job opens a draft PR — all without the box ever holding a write token.

Not yet wired (the remaining build, on feat/agent-team-p3-box-integration): the box-side integration that makes a task flow BUILD → DISPATCH (trigger that live CI, capture the run_id) → VERIFY (read its conclusion) automatically. As-built, serve binds the fail-safe gated P3 wiring (degrading to the INERT path if the dispatch target / CI-read token is unset — Phase-0 Decision 5), dispatch is operator-initiated (branch push + gh workflow run on operator-host credentials; the box holds no standing write token), run_id capture is run-name / correlation-tag based, and the CI wait is async resume-on-complete via a tick()-driven CI-watcher (not a blocking poll). The Tier-3 fixer remains --dry-run only until the Phase-4 smoke test passes.

After the box-side integration + the remaining human gates: the pipeline emits a diff end-to-end into a reviewable draft PR for human review. The box never gains a write token.

2. Locked decisions (carried in — do not relitigate here)

• D-OIDC: apply path uses a GitHub App pull-requests:write token, ZERO AWS, no OIDC.
• D2: box stays read-only / no standing write token; org CI does the applying.
• B4: the LLM-proposed diff is untrusted code; the CI trust boundary (below) is mandatory.
• Output is draft PRs only — nothing auto-merges; human approval is the merge gate.
• (Separate, not part of this plan: aws-posture resident access via step-ca + IAM Roles Anywhere — that IAM is already cross-review-approved and is its own sub-task.)

3. Hard gates (must clear before the flip — these block everything)

Gate	Why	Owner	Status
/sh-plan-review on THIS plan	adversarial plan audit before build	me → GPT-4.1	✅ DONE (Round 1 + 2, 2026-06-22)
/sh-security-review on the apply/verify CI surface	auth + untrusted-input + CI trust boundary	me	⏳ RE-RUN on the enabled workflow + bound box-side wiring (C1 / Phase 3 / B5)
GPT-4.1 cross-family review on the apply/verify CI + any permission change	mandatory for the trust-boundary / permissions surface	orchestrator	⏳ RE-RUN on the enabled workflow + bound box-side wiring (C1 / Phase 3 / B5)
GH_TOKEN→GITHUB_TOKEN resolved ✅ DONE 2026-06-22	transport reads GITHUB_TOKEN; box now has a GITHUB_TOKEN alias of GH_TOKEN in ~/secrev.env	me	✅ DONE

The apply/verify CI workflow + its provisioning are already LIVE (2026-06-22). The two /sh-security-review + GPT-4.1 cross-review gates were satisfied against the authored workflow during build; per B5 they are RE-RUN against the ACTUAL enabled workflow AND the bound box-side build→dispatch→verify wiring before the box-side integration is deployed/merged (Phase 3). The box-side integration does NOT deploy/merge until that re-run passes — hard stop (see Phase 3).

Plan-review disposition (GPT-4.1 cross-family, 2026-06-22 — REQUEST CHANGES). Findings folded into §4 and Phases 1/1b: expanded denylist vectors, runner-trust, concrete diff-transport + threat model, gate-weakening detection, PR-metadata sanitization, ledger anti-tamper, recovery for a merged-privileged change, required-check-name discovery, deploy-before-merge enforcement, no-write-token audit, draft-PR rate monitoring + stale-PR cleanup. Several items the reviewer marked BLOCK are already implemented in PR #17's CI (canonicalization, egress, SHA-pin, empty-hash fail-closed, Checks-API) — Phase 1 verifies them rather than rebuilding. Open QUESTIONs to answer when building: how human reviewers are notified of new draft PRs, and how "every deployable repo has CI" is enforced for targets.

Plan-review disposition — ROUND 2 (GPT-4.1 cross-family, 2026-06-22 eve — REQUEST CHANGES). Re-review after the durable-intake work merged (#32) and the coordinator went live. Six BLOCK items folded into the phases below: (B1) deploy-before-merge made a CONCRETE CI-enforced gate, not prose (Phase 1 + Phase 3); (B2) rollback added for a prematurely-flipped privileged job that RUNS — not only an accidental merge (Phase 1b); (B3) rollback for ALL privileged surfaces is a tested script, exercised, not one-time manual (Phase 1b); (B4) Phase 3 is explicitly gated on Phase 2 being fully provisioned + verified (Phase 3 precondition); (B5) /sh-security-review + GPT-4.1 cross-review are RE-RUN on the actual enabled workflow before the flip (Phase 3); (B6) docs/memory updated INCREMENTALLY at each privileged step, not deferred to Phase 6. FIX/NIT also folded: periodic review of gate-weakening patterns, a check-name-discovery test across targets, a memory update when denylist vectors change, snapshot retention in Phase 0, and the draft-PR notification QUESTION assigned in Phase 4. (Findings-to-verify, not gospel: the reviewer under-read §3 — cross-review IS already a hard gate; B5 is the genuine delta = re-run on the enabled file.)

4. The CI trust boundary (design B4 — what the workflow must enforce)

Already implemented in the merged P3-live CI hardening (PR #17) — Phase 1 VERIFIES, does not rebuild: denylist path canonicalization + symlink/rename/traversal resistance, egress restriction on the untrusted job, SHA-pinned actions, empty/missing-hash fail-closed, and authenticated Checks-API result consumption. The GPT-4.1 plan-review (2026-06-22) flagged these as "missing" because the plan under-restated them; confirm each against the actual agent-team/ci/agent-team-apply-verify.yml + ci_fetcher.py/ci_gate.py rather than re-authoring.

1. Split CI. An untrusted build/test job: contents: read only, no secrets / no OIDC / no write token, egress-restricted (verify/audit the restriction, don't just assert it). A separate privileged job that never checks out the patch code (no pull_request_target + head checkout) opens the draft PR. Privileged jobs MUST run only on GitHub-hosted runners — assert no self-hosted/user-provided runner can pick them up.
2. Denylist (reject/escalate, never auto-build). Beyond .github/workflows/**, IAM/permission IaC, branch-protection / CODEOWNERS / Dependabot config, and out-of-scope files, the denylist MUST also cover these RCE/priv-esc vectors: .gitmodules / submodule changes, git hooks (.git/hooks, core.hooksPath, .husky/**), .gitattributes (filter/clean-smudge process), lockfiles + package-manager postinstall/preinstall hooks, and generated/build-artifact files (codegen output is not reviewable as source). Path matching is canonicalized (PR #17) so symlinks/renames/traversal can't slip a denied path past.
3. Diff-transport integrity (concretely specified, threat-modeled). The box records the diff content hash in its ledger; CI verifies the hash before apply. The diff reaches CI as a content-addressed signed artifact (HMAC/keyed digest the box and the privileged job share via an Actions secret) OR a short-lived, branch-only token scoped to a single ref — the box never holds a write token. Threat model the path: tamper-in-transit (defeated by hash+signature verify), replay of an old diff (defeated by per-task nonce + the status='open'/one-shot ledger state), and a hostile artifact name.
4. Pure-code green gate. Pass/fail is owned by a pure-code gate reading authenticated Checks-API results (run id + diff hash). The LLM verifier may propose fixes but can never declare a build green.
5. Gate-weakening detection. A diff that lowers a gate — adds # noqa, # type: ignore, test skip/xfail, coverage/lint excludes, --no-verify, or edits the gate config itself — is flagged and escalated (a build can't make itself pass by disabling the checks).
6. PR-metadata sanitization. The draft-PR title / body / comments are sanitized so a hostile diff or LLM output can't exfiltrate secrets/env or inject content into the PR text.
7. Ledger anti-tamper. The diff-hash ledger entry is integrity-protected (the existing atomic-write + integrity-check substrate; verify the hash row can't be silently rewritten between record and apply).
8. Merge gate. Draft PR + required checks + /sh-security-review + Claude Code App review + human approval.

5. Phases

Phase 0 — Plan review & pre-reqs 🤖/🧑 — ✅ DONE

DONE. Plan review complete; pre-reqs resolved. (Task label: C0.)

• Run /sh-plan-review on this doc; fold BLOCK/FIX items in. (Round 1 + Round 2 done; this doc is the result.)
• Confirm a clean revert point (git tag main; Hyper-V snapshot of sh-secrev). Snapshot retention: keep the pre-P3 snapshot until P3 has run clean for one full cycle (Phase 4 DoD), then prune — recorded here so it is not an open-ended snapshot.
• Resolve GH_TOKEN→GITHUB_TOKEN (box ~/secrev.env now has a GITHUB_TOKEN alias).
• Rollback: none (no state changed).

Phase 1 — Author/verify the split-CI apply/verify workflow 🤖 (review-gated) — ✅ DONE (2026-06-22)

DONE. The split-CI apply/verify workflow is authored, enabled, and provisioned. The deliverables below were built and shipped; they are retained for the record and for the Phase-3 re-run gates to verify against. (Task label: this is part of C2.)

• Reconcile agent-team/ci/agent-team-apply-verify.yml with §4. First confirm the PR-#17 controls are present (canonicalized denylist, egress restriction, SHA-pins, empty-hash fail-closed, Checks-API consumption); only then add the new §4 items.
• Add denylist vectors (§4.2): submodules/.gitmodules, git hooks/core.hooksPath/.husky, .gitattributes filters, lockfile postinstall/preinstall, generated/build artifacts. Add a test suite proving canonicalization resists symlink/rename/traversal.
• Runner-trust assertion (§4.1): test that privileged jobs cannot run on a self-hosted/user-provided runner.
• Concretize + threat-model the diff transport (§4.3): pick content-addressed signed artifact (shared HMAC secret) or short-lived branch-only token; add per-task nonce anti-replay; document and test it.
• Gate-weakening detector (§4.5): CI step that fails on a diff adding noqa/type: ignore/skip/xfail/excludes/--no-verify or editing the gate config. The detector's pattern list is reviewed/expanded each time a new bypass vector is found (FIX) — record the list in code with a comment pointing here, and update it + memory when a vector is added (same discipline as the denylist below).
• PR-metadata sanitization (§4.6) and ledger anti-tamper (§4.7) implemented + tested.
• agent_team/ci_fetcher.py (read-only Checks-API fetcher; fails closed) + ci_gate.py (pure-code green). Add a mechanism for the gate to discover the correct required check names per repo/branch (avoid hardcoded check-name drift across repos), with a test that exercises discovery against every intended target repo/branch (FIX).
• Memory/doc update when denylist vectors change (FIX): adding a denylist vector (here or later) updates project_r720_agent_team memory + the Confluence host page in the SAME change — the denied set is operational/security-critical, not tribal knowledge.
• Deploy-before-merge enforcement — CONCRETE, CI-enforced (B1). REQUIRED Phase-1 deliverable; the flip does not proceed without it (no manual fallback). "Deploy" of this change = the privileged path is proven on the live box before the workflow PR merges. Build, in this phase: - (a) the box exercises a dispatch dry-run of the apply/verify workflow (privileged steps still if:${{ false }}) that emits a signed "exercised-on-box" artifact/status. Dry-run faithfulness (NIT): it runs the SAME workflow file and the SAME untrusted build/verify + pure-code-gate jobs as the live path — ONLY the privileged if: differs — so the dry-run is a faithful proof of the path, not a separate mock. - (b) a required status check on the workflow-file PR consumes that proof, so the PR is un-mergeable until the box has run it. - (c) branch-protection set so the required checks cannot be bypassed by admins ("do not allow bypassing the above settings" / include-administrators) — no --admin merge of the privileged flip. Applied in Phase 2; no ordering window because the flip (Phase 3) is gated on Phase 2 completion (the B4 precondition), so admin-bypass is already disabled before any flip is possible. The Phase-1 required-status-check (a/b) and the Phase-2 branch-protection (c) together are the gate; the flip cannot happen until BOTH are in place. This is the gate; until it is built+green, the flip is blocked. (Owner: 🤖 build; verified in the Phase-1 /sh-security-review + cross-review hard stop. The check's implementation is itself a Phase-1 build task — that it is not yet physically built is expected for a pre-build plan; what matters is it is non-optional and flip-blocking, enforced at the Phase-1 hard stop.)
• Concrete "no write token on the box" audit (B-QUESTION → a real check): a test/script asserting the box env + coordinator config hold no pull-requests:write / contents-write token (grep the live env names + assert the App token is only an Actions secret), runnable on the box and in CI. Not a prose claim.
• /sh-security-review + GPT-4.1 cross-review on this surface (against the authored workflow during build). Hard stop until both pass. (These are RE-RUN on the enabled workflow + the bound box-side wiring in Phase 3 — the C1 human gate, see B5 there. That re-run is the genuine outstanding gate; the in-build pass is satisfied.)
• Rollback: the apply/verify workflow is now LIVE; rollback is no longer "delete an inert file" — use the Phase-1b tested rollback script (revert the workflow SHA → inert, restore the environment + branch protection from baseline). See Phase 1b / Phase 3 rollback.

Phase 1b — Recovery for an accidentally-merged/applied privileged change 🤖/🧑 — ✅ DONE (2026-06-22)

DONE. The privileged surfaces are LIVE, so their recovery tooling shipped with them. The tested rollback script + the runaway/stale-PR monitoring below are built and in place; the Phase-3 rollback exercises the script against the live state. (Task label: part of C2.)

• Rollback as a TESTED SCRIPT covering ALL privileged surfaces (B3) — not a one-time manual exercise. One scripted, re-runnable rollback that, per surface, restores from a recorded baseline: (1) the apply/verify workflow (revert the SHA → inert), (2) the agent-apply environment (required-reviewer + protection rules), (3) the GitHub App permissions/installation (rotate token, reduce/uninstall), (4) branch protection. The script asserts the post-restore state matches the baseline. Exercised in Phase 3 rollback AND re-runnable on demand.
• Premature-flip-that-RAN rollback (B2). Distinct from an accidental merge: cover the case where if:${{ false }} is flipped early (or the environment gate is misconfigured) and the privileged job actually runs — incident steps: rotate the GitHub App token immediately, close/revert any draft PR (or branch) it opened, confirm via the Checks/PR audit trail exactly what ran in the window, restore the environment + branch protection from baseline, and file the incident. This is the "it executed" path, not just "it merged."
• Add light monitoring on draft-PR creation rate (runaway-volume alarm) and an orphaned/stale draft-PR cleanup step.

Phase 2 — Provision the GitHub App + environment 🧑 OPERATOR (browser/admin) — ✅ DONE (2026-06-22)

DONE. Provisioning is LIVE: the scoped GitHub App, the agent-apply environment, and the App-credential Actions secrets all exist, and dispatched runs have executed against them. (Task label: C0 complete.) No write token landed on the box — the App token lives only as an Actions secret; this is enforced by scripts/assert_no_write_token.py.

• Create a dedicated GitHub App with pull-requests:write (+ minimal contents to open a branch/PR); install on the org. Token lives in CI, never on the box.
• Create the agent-apply GitHub Actions Environment with a required reviewer (Adam) + branch-protection so the privileged job cannot run unreviewed.
• Store the App credentials as repo/org Actions secrets (not on the box).
• Rollback: uninstall the App; delete the environment + secrets (via the Phase-1b script).

Phase 3 — Bind the box-side build→dispatch→verify wiring 🤖 — IN PROGRESS (the remaining build)

Workflow flip already DONE (2026-06-22): permissions: pull-requests: write and environment: agent-apply are uncommented and the two if: ${{ false }} are enabled — the workflow is LIVE. The PRECONDITION (B4) below is satisfied (Phase 2 provisioning is complete + verified). The remaining Phase-3 work is the box-side integration built on feat/agent-team-p3-box-integration (BUILD → DISPATCH → VERIFY; operator-initiated dispatch; run-name correlation; async CI-watch; fail-safe serve default — see docs/P3-PHASE0-DESIGN.md) plus the C1 re-run gates and the box env wiring. C1 (the re-run gates) and D (deploy/smoke/ merge, Phase 4) remain the outstanding HUMAN gates.

• PRECONDITION (B4): Phase 2 fully complete + verified before ANY flip. ✅ SATISFIED — the GitHub App exists with pull-requests:write (+ minimal contents) and is installed, the agent-apply environment exists with Adam as required reviewer + branch protection, and the App credentials are stored as Actions secrets (NOT on the box).
• In the workflow: uncomment permissions: pull-requests: write and environment: agent-apply; flip the two if: ${{ false }} → enabled. ✅ DONE 2026-06-22.
• C1 — RE-RUN BOTH GATES ON THE ENABLED WORKFLOW + BOUND WIRING (B5). 🧑 OUTSTANDING HUMAN GATE. /sh-security-review + the GPT-4.1 cross-family review are run again against the actual enabled agent-team-apply-verify.yml (permissions live, if: true) and the bound box-side gated_build_verify_wiring (BUILD → DISPATCH → VERIFY) — NOT only the inert Phase-1 version. Hard stop until both pass. (Permissions are live + the dispatch/verify wiring is new → the mandatory cross-family review is independently required here against the real diff.)
• Bind agent_team.coordinator.gated_build_verify_wiring(...) (real diff builder + dispatch with run-name correlation run_id capture + read-only CI-result fetcher) as the fail-safe serve default (Decision 5: degrade to the INERT P3 path if the dispatch target / CI-read token is unset, never crash-loop). The leaf path is BUILD → DISPATCH (trigger the live CI, suspend) → [CI-watcher resumes on terminal conclusion] → VERIFY (pure-code gate). (Built on feat/agent-team-p3-box-integration.)
• Set the box-side apply env vars the live path reads (read-only CI-result token + dispatch target: AGENT_TEAM_REPO_OWNER/_NAME/_BASE_BRANCH/_CI_READ_TOKEN). Confirm no write token lands on the box (run the no-write-token audit, scripts/assert_no_write_token.py).
• Incremental docs (B6): update OPERATOR-RUNBOOK.md + memory as the box-side flip lands (do not wait for Phase 6) — what the apply path can/can't do, the denylist, the rollback. (The P3 box env wiring is already documented in docs/provisioning/OPERATOR-RUNBOOK.md.)
• Rollback: run the Phase-1b tested rollback script (re-set if: ${{ false }}, re-comment environment:, set build_verify_wiring=None, restart the coordinator). Exercise it once here to prove it works before relying on it.

Phase 4 — Deploy, smoke test to a first draft PR, merge 🧑/🤖 — D (OUTSTANDING HUMAN GATE)

D — the remaining HUMAN gate. After C1 passes, deploy the box-side integration to the live coordinator (sh-secrev, via /sh-deploy-r720), drive the smoke test below, then merge. Gated on Phase 3 (box-side wiring bound + C1 re-run gates green).

• Drive one trivial, in-scope task end-to-end → confirm: untrusted job builds/tests with no secrets, denylist rejects an out-of-scope diff, pure-code gate gates on real Checks results, privileged job opens a draft PR with required checks attached, nothing merged.
• Flip the Tier-3 fixer off --dry-run only after the smoke test passes; verify a dependency-CVE bump produces a draft PR.
• Draft-PR reviewer notification (QUESTION resolved/assigned): decide + wire how a new draft PR pings the human reviewer — default = the existing Slack ALARM path posts a #agent-team notice with the PR link (Adam owns this decision; recorded so it is not left open).
• Rollback: close the draft PR; Phase-3 rollback.

Phase 5 — Enable the cross-plane loop 🤖

• Allow confirmed Plane-1 checker findings (intake-checker) to flow into pipeline tasks that end in draft-PR fixes (start conservative: highest-severity, one at a time).
• Conservative-rollout controls (QUESTION resolved): enforce "one at a time, highest severity" concretely — a cap on concurrently-open cross-plane draft PRs (start = 1) read by the intake-checker leaf, the severity floor in config, and a #agent-team log line per promotion so the rate is observable. Not just prose.
• Rollback: revert intake-checker wiring to report-only.

Phase 6 — Final docs & memory consolidation 🤖

B6: docs/memory are updated INCREMENTALLY at each privileged step above (Phase 3 runbook+memory on flip; denylist/gate-weakening changes update memory in the same change). Phase 6 is the FINAL reconciliation pass, not the first time docs are touched.

• Reconcile OPERATOR-RUNBOOK.md, the Confluence host page (20054017) + runbook (22609921), and project_r720_agent_team memory: P3 is LIVE, what the apply path can/can't do, the denylist, the merge gate, the rollback — confirming the incremental updates are complete + consistent.

6. What changes (and what does NOT)

Changes:

• The pipeline produces draft PRs for well-scoped work (dep bumps, small fixes, docs, checker-surfaced issues). Your role shifts from author → reviewer/approver for that class.
• Tier-3 fixer goes live (dep-CVE bumps → draft PRs).
• Plane-1 findings can become draft-PR fixes (cross-plane loop).

Does NOT change:

• Box stays read-only; CI applies, not the box. No AWS / no OIDC in the apply path.
• Denylisted, high-blast-radius changes (workflows, IAM, branch protection, CODEOWNERS, Dependabot) are never auto-built — always escalated.
• Human gates stay: the clarifier question-set AND the PR approval are both yours.
• Drafts, not merges — you remain the merge authority.

7. Risks & mitigations

Risk	Mitigation
Untrusted diff exfiltrates secrets / escalates in CI	split CI: build job has no secrets/OIDC/write, egress-restricted; privileged job never checks out patch code
Malicious diff edits CI/IAM/branch-protection to self-escalate	denylist → reject/escalate, never auto-build
LLM "declares" a broken build green	pure-code gate reads authenticated Checks-API only; LLM can't set status
Diff tampered between box and CI	diff-hash recorded in ledger, verified before apply
Standing write capability on the always-on box	there is none — App token lives in CI; box holds only a read-only CI-result token
Runaway PR volume	start with Tier-3 only + one finding at a time; required-reviewer environment gates each

8. Definition of done

• /sh-plan-review passed; /sh-security-review + GPT-4.1 cross-review passed on the CI surface during build. ⏳ C1 outstanding: both are RE-RUN against the enabled workflow + the bound box-side wiring before the box-side integration deploys/merges (Phase 3 / B5).
• D: Phase-4 deploy + smoke test produced a draft PR; nothing auto-merged; rollback exercised once.
• No write token on the box (verified via scripts/assert_no_write_token.py); apply path is zero-AWS. (Re-confirm after the box-side env vars are set in Phase 3.)
• Docs + Confluence + memory updated (this plan + OPERATOR-RUNBOOK.md reflect the live infra; Confluence + memory final reconciliation is Phase 6).
• Snapshot retained until P3 runs clean for one cycle, then pruned.