This repository has been archived on 2026-08-04. You can view files and clone it, but cannot push or open issues or pull requests.
orchestrator/security-review/DEPLOY-R720.md
Adam Moussa dbe2f8c186 Raise nightly agentic budget $20 -> $120 for full per-night coverage
First-run data (2026-06-17) showed the $20 ceiling covered only the
canary + 5 of ~22 scannable repos before pausing the rotation, leaving
16 repos un-deep-scanned that night. Raise TOTAL_BUDGET_USD default to
$120 so every repo gets a deep agentic pass each night (~22 x ~$5 +
canary, with headroom). Spend draws on the Max subscription pool; the
per-target cap ($12) and round-robin rotation are unchanged, so this is
a ceiling raise, not a per-repo cost change. Updates README, DEPLOY, and
the systemd Environment example to match.
2026-06-17 13:18:46 -04:00

8.2 KiB

Phase 3 — Path B deployment (R720 VM)

Status: host built; headless runner built + validated; two-tier auto-discovery nightly sweep built. Pending: provision the read-only GH_TOKEN and run one live VM dry-run to validate the clone-mirror path end-to-end. CI was removed by design — the git hooks + this nightly sweep are the backstop. See memory project-security-review-agent.

Path B is the unattended backstop that shares one pure-code gate (review.sh) with the interactive Path A (/sh-security-review). This file is the operator runbook for the box that runs it.

Host

  • Hypervisor: R720 at 10.10.60.40 (Windows Server 2022, Hyper-V role).
  • VM: sh-secrev, always-on Ubuntu 24.04 (kernel 6.8), Gen2, 4GB / 2 vCPU / 40GB dynamic vhdx.
  • Reach it: ssh -i ~/.ssh/r720_seahaven adam@10.10.60.120 (key-only, NOPASSWD sudo).

Operate on the VM, not from the Mac against the host by hand.

What is installed on the VM

  • Deterministic scanners: semgrep, gitleaks, checkov, pip-audit, cfn-lint. Node 18 (npm audit).
  • claude CLI (Node) — the subscription-auth path for Path B.
  • Python 3.12 venv at ~/orchestrator/.venv with claude-agent-sdk.
  • Repo: ~/orchestrator/ (rsync from the Mac, .env excluded — NOT a git clone). After editing the sweep locally, re-sync: rsync -av --exclude .env --exclude .venv ~/Documents/repositories/orchestrator/ adam@10.10.60.120:orchestrator/.
  • Testbed corpus: ~/security-review-testbed (also rsync'd; includes the Node + .NET fixtures).
  • No gh CLI required — discovery uses the GitHub REST API via curl. run_headless.py is self-contained (detector/verifier prompts are inline), so the VM needs no ~/.claude assets to run.

Auth, billing, and the read-only GitHub token

Claude (subscription OAuth)

  • Token from claude setup-token, stored in ~/secrev.env as CLAUDE_CODE_OAUTH_TOKEN (mode 600, NOT in git).
  • The 2026-06-15 SDK-billing split was deferred, so automated SDK usage draws from the Max 20x subscription's normal usage limits — the same pool as interactive Claude Code. The two-tier sweep below is what keeps that draw bounded. See memory reference-claude-subscription-billing.
  • CRITICAL: a raw ANTHROPIC_API_KEY would silently win and meter to API rates — it must NOT be set on this host. run_headless.py pops it defensively and refuses to run without CLAUDE_CODE_OAUTH_TOKEN.

GitHub (GH_TOKEN, read-only — REQUIRED for auto-discovery)

The nightly sweep enumerates and clones org repos with a fine-grained, read-only PAT. Never give this always-on box a write-capable token.

  1. github.com → Settings → Developer settings → Fine-grained personal access tokens → Generate new.
  2. Resource owner: Sea-Haven-Industries. Repository access: All repositories.
  3. Permissions: Repository → Contents: Read-only, Metadata: Read-only (auto). Nothing else.
  4. Set an expiry (e.g. 90 days; calendar a rotation). Generate and copy the github_pat_... value.
  5. On the VM, append it to ~/secrev.env and lock the file down:
    echo 'GH_TOKEN=github_pat_xxxxxxxx' >> ~/secrev.env && chmod 600 ~/secrev.env
    
  6. Verify (should print repo names, not a 401):
    set -a; . ~/secrev.env; set +a
    curl -fsS -H "Authorization: Bearer $GH_TOKEN" \
      "https://api.github.com/orgs/Sea-Haven-Industries/repos?per_page=3" | jq '.[].full_name'
    

Non-Claude provider keys

The GPT-4.1 critical tiebreak (optional) uses keys in ~/orchestrator/.env (mode 600, gitignored, auto-loaded by run.py). They bill to their own provider accounts — keep them out of ~/secrev.env.

The headless runner: run_headless.py

Runs the 6 fresh-context detectors + proof-or-kill verifier unattended via the Agent SDK; emits the finding-schema JSON that review.sh --agent-findings consumes. Read-only tools, hermetic (setting_sources=[]), fails toward over-reporting. CLI:

CLAUDE_CODE_OAUTH_TOKEN=... python3 run_headless.py TARGET_DIR \
  [--scope "src infra web"] [--out findings.json] [--model claude-...] \
  [--detectors injection,authz,...] [--concurrency 3] [--max-turns 40] \
  [--detector-budget-usd 2.0] [--total-budget-usd 12.0]

When the total budget is exhausted the verifier is skipped and remaining candidates stay unverified — never silently dropped. Manual single-repo run:

cd ~/orchestrator
set -a; . ~/secrev.env; set +a
.venv/bin/python security-review/run_headless.py ~/security-review-testbed --out /tmp/agent.json
security-review/review.sh --agent-findings /tmp/agent.json ~/security-review-testbed

Nightly two-tier, clean-clone auto-discovery sweep

nightly_sweep.sh needs no per-repo wiring. Each night it:

  1. Discovers every non-archived Sea-Haven-Industries repo via the REST API (curl + GH_TOKEN) and mirrors each as a shallow clean clone (git clone --depth=1, default branch from the API default_branch) into ~/repo-mirrors. The token is injected only for the fetch and scrubbed from the on-disk remote afterward. Clean clones contain no developer-local gitignored .env, so live secrets stay out of scope by construction.
  2. Canary first: scans ~/security-review-testbed agentically (anti-complacency) — must block and meet the recall floor, else COMPLACENCY ALARM.
  3. Tier 1 (every repo, $0 Claude): review.sh --scanners-only over every mirror.
  4. Tier 2 (bounded agentic): run_headless.py over a deterministic round-robin rotation that fits TOTAL_BUDGET_USD, with a persistent cycle pointer (~/sweep-reports/.rotation-state.json) so every repo gets a deep pass within MAX_CYCLE_NIGHTS; a COVERAGE ALARM fires if it falls behind.

ALARM-only (a clean night posts nothing). Secret-shaped values are redacted from the Slack string; reports under ~/sweep-reports/<UTC-date>/ are mode 600.

Config (env / systemd Environment=)

GH_ORG (Sea-Haven-Industries) · MIRROR_DIR (/repo-mirrors) · TOTAL_BUDGET_USD (120) · PER_TARGET_BUDGET_USD (12) · CANARY_FLOOR (10) · MAX_CYCLE_NIGHTS (4) · MAX_AGENTIC_PER_NIGHT (0 = unlimited) · CENTRAL_SKIP_FILE (/.secrev-skip.txt) · ENABLE_XMODEL_HOOK (0) · TARGETS (manual override — scan explicit paths, no discovery).

Skip a repo

Commit a .security-review-skip at its root, or add its name to ~/.secrev-skip.txt. Marker-skips are logged in the report (a sensitive repo cannot silently self-exclude).

Manual dry-run (do this once after provisioning GH_TOKEN)

cd ~/orchestrator
set -a; . ~/secrev.env; set +a
./security-review/nightly_sweep.sh
# Watch: discovery count, mirrors, canary block+recall, tier1 over all repos, tier2 rotation, clean exit.
# Then re-tune CANARY_FLOOR to the reported recall, and confirm the ALARM path with a forced failure.

Install the timer

sudo cp security-review/systemd/sea-haven-secrev.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now sea-haven-secrev.timer   # the timer drives it; do not enable the .service
systemctl list-timers sea-haven-secrev.timer

Fires nightly ~02:00 local (Persistent=true catches missed runs). TimeoutStartSec=21600 (6h) bounds a hang without killing a healthy long night; spend is capped by TOTAL_BUDGET_USD.

Anti-complacency reinforcements

  • Canary: the testbed (now Python/IaC/React + Node + .NET planted vulns) is scanned every night; a recall drop or non-block is a COMPLACENCY ALARM.
  • Coverage: the rotation pointer + MAX_CYCLE_NIGHTS guarantee every repo gets a deep pass on a cadence, with a COVERAGE ALARM if it slips — no silent incomplete coverage.
  • Two-model disagreement (optional): ENABLE_XMODEL_HOOK=1 re-checks confirmed criticals with GPT-4.1.

Remaining (deferred by design)

  • Persistent budget/telemetry ledger: cross-run spend tracking beyond the per-run + nightly caps (optional).
  • Phase 4 roster growth (compliance/drift sweep, CVE agent, optional auto-fixer) — only per a real job.
  • Phase 5 remediation: harden findings as real repos surface them (payments-dashboard first).
  • Confluence: document sh-secrev as standing infrastructure (always-on VM holding a read-only org PAT, pulling all org repos nightly) in the IT host/LAN inventory.