This repository has been archived on 2026-08-04. You can view files and clone it, but cannot push or open issues or pull requests.
orchestrator/agent-team/scripts/p3_rollback.sh
Adam Moussa 00c51192c8 fix(agent-team): remediate C1 security-review BLOCK (2 HIGH + MED/LOW)
High-recall /sh-security-review fan-out + proof-or-kill verifier found two
confirmed HIGH; both now closed (verified empirically against the working tree):

- LOGIC-RACE-01 (HIGH, CWE-835): the build-loop budget was structurally dead
  (verifier read a shared wiring-time VerifierConfig.build_loops, always 0, so
  the max_build_loops park never fired -> a perpetually-failing task looped
  BUILD->DISPATCH->VERIFY forever, force-pushing + firing a CI run each round).
  Threaded build_loops through durable PipelineState/TaskRecord; verifier reads
  state.get('build_loops',0), writes the incremented count back on each FAIL, and
  PARKS at max_build_loops. Parks after exactly N failures, never unbounded.
- SEC-01 (HIGH, CWE-532) + SEC-02 (MED, CWE-214): p3_rollback.sh echoed the live
  App JWT to stdout in default dry-run and passed it as a gh argv literal. Added
  redact_secrets (Bearer/Authorization/ghX_/PEM masking) through run_or_plan; the
  App uninstall now uses curl -H @<0600 tempfile> (JWT never on argv), shredded
  after. Empirical: app/incident/all dry-runs leak 0 JWT occurrences.
- SEC-03 (MED, CWE-798): assert_no_write_token now applies the PEM regex + the
  configured App-ID to env/config VALUES (not just files) — an App private key
  under a benign env name is caught.
- SEC-04 (LOW) + P3-IAC-08 (LOW): tightened the box GITHUB_TOKEN fallback /
  value-scan; staged-only WARN on the live workflow revert.

Suite: 1382 passed, ruff clean. Branch only; not merged/deployed.
NOTE: re-verifier flagged SEC-01 as open by grepping COMMITTED blobs (the fix was
uncommitted working-tree state); independently confirmed closed empirically.
2026-06-23 19:52:04 -04:00

1011 lines
48 KiB
Bash
Executable file

#!/usr/bin/env bash
# p3_rollback.sh — restore EVERY privileged P3 surface to a recorded baseline.
#
# The P3 apply/verify CI workflow is ALREADY LIVE (flipped + provisioned
# 2026-06-22): the `agent-apply` environment with its required reviewer, the
# GitHub App installation (pull-requests: write), the run-name/permissions edits
# on the workflow, and branch protection on `main`. This script is the inverse:
# it takes a recorded baseline of those surfaces and restores them, then asserts
# post-restore == baseline. It is what you run when the flip must be unwound, or
# when a premature flip RAN and opened a draft PR that should not exist.
#
# DESTRUCTIVE-SAFE BY DEFAULT. Every gh/git mutation is held behind a --dry-run
# default that only PRINTS the plan. --apply performs the mutations. Nothing
# destructive happens without --apply on the command line.
#
# KNOWN LIMITATIONS — VALIDATE LIVE BEFORE RELYING ON --apply (C1 gate). These
# GitHub REST calls are exercised only against a stubbed `gh` in tests; their
# exact live shapes MUST be confirmed with a real `--dry-run` against the target
# repo/App during the mandatory C1 /sh-security-review + GPT-4.1 cross-review
# (this script touches IAM-adjacent surfaces):
# * Branch-protection RESTORE: the recorded baseline is the verbatim GET object,
# but GitHub's branch-protection GET and PUT schemas are NOT symmetric
# (GET enforce_admins is {enabled,url} vs PUT's bare bool; GET carries
# read-only url/checks fields PUT rejects; PUT requires a `restrictions` key
# GET may omit). A real PUT of the raw GET object can 422 — transform GET->PUT
# (enforce_admins->bool, required_status_checks->{strict,contexts},
# required_pull_request_reviews->writable subkeys, restrictions->null if
# absent) and confirm against the live API before --apply.
# * App neutralise is uninstall (DELETE /app/installations/{id}) which needs an
# App JWT ($AGENT_APPLY_APP_JWT), NOT operator gh; --apply hard-fails without
# it rather than pretending operator gh can do it.
# * environment.deployment_branch_policy is recorded from the live env GET; if a
# custom branch-name policy is in use, confirm the restore preserves it.
# The post-restore protection assert IS real (normalize_protection reads argv,
# fails loud on parse error, and a divergent live state exits non-zero — see
# tests/test_rollback.py::test_apply_protection_DETECTS_divergent_live_restore).
#
# Surfaces restored (design: docs/P3-PHASE0-DESIGN.md; the workflow:
# .github/workflows/agent-team-apply-verify.yml):
# 1. workflow — the apply/verify flip.
# PRE-merge: close the flip PR + delete its branch.
# POST-merge: git revert the flip commit + push + re-run CI.
# LIVE: the run-name/permissions edits are already on
# the branch, so also revert them to the recorded
# baseline SHA (`workflow_baseline_sha`).
# 2. environment — the `agent-apply` environment: required reviewer(s) +
# deployment branch policy. Reviewers are restored BY NUMERIC
# USER ID (logins are resolved to ids when the baseline only
# recorded logins), via a proper JSON reviewers array.
# 3. app — neutralise the GitHub App. There is NO "reduce App
# permissions" REST endpoint (PATCH .../permissions is a 404),
# and an installation token cannot be revoked by operator gh
# (DELETE /installation/token revokes the token you
# authenticate WITH — operator host gh is not that token).
# The only programmatic neutralise is UNINSTALL
# (DELETE /app/installations/{id}), which requires an APP JWT
# (NOT operator gh). The fallback is an OUT-OF-BAND operator
# action (remove the install in the org UI / rotate the App
# private key). This script PLANS those steps and honours
# --apply only for the JWT-authenticated uninstall.
# 4. protection — branch protection on `main`; restored from the FULL recorded
# protection baseline (not just enforce_admins) and asserted.
# Baseline asserts include_administrators == ON.
# 5. incident — "premature flip that RAN": neutralise the App (uninstall via
# App JWT, or out-of-band — see surface 3), revert any draft
# PR/branch the App opened, audit the Checks trail, restore
# environment + protection, and file an incident note.
#
# Usage:
# scripts/p3_rollback.sh <surface> [--apply] [--baseline FILE] [options]
#
# <surface> ∈ { workflow | environment | app | protection | all | incident }
#
# Options:
# --record-baseline capture the CURRENT live state into the baseline JSON
# (workflow SHA, env reviewer ids, full branch protection,
# App installation id) instead of restoring. Honours
# --apply (default --dry-run just prints what it would
# capture).
# --apply perform the mutations (default is --dry-run: print only)
# --dry-run print the plan only (default; explicit for clarity)
# --baseline FILE path to the recorded baseline JSON (default:
# $P3_ROLLBACK_BASELINE or .security-review/p3-baseline.json)
# --repo OWNER/NAME target repo (default: $AGENT_TEAM_REPO_OWNER/_NAME)
# --env-name NAME environment name for --record-baseline (default agent-apply)
# --app-installation-id N App installation id to record for --record-baseline
# --flip-pr N the flip PR number (pre-merge close path)
# --flip-branch REF the flip PR head branch (pre-merge delete path)
# --merged the flip is already merged (post-merge revert path)
# --flip-commit SHA the merged flip commit to revert (post-merge path)
# -h | --help show this help and exit
#
# Baseline JSON (recorded BEFORE the flip; the restore target). Keys consumed:
# {
# "repo": "Sea-Haven-Industries/orchestrator",
# "default_branch": "main",
# "workflow_path": ".github/workflows/agent-team-apply-verify.yml",
# "workflow_baseline_sha": "<git sha of the pre-flip workflow blob/commit>",
# "environment": {
# "name": "agent-apply",
# // Prefer NUMERIC ids (restore is exact + assertable). Logins are also
# // accepted and resolved to ids at restore via `gh api users/{login}`.
# "required_reviewer_ids": [1234567],
# "required_reviewers": ["amoussa1229"],
# "deployment_branch_policy": "protected"
# },
# "app": {
# "slug": "agent-apply",
# "installation_id": 0,
# // The ONLY programmatic neutralise is "uninstall" (App JWT). There is no
# // permission-reduction REST endpoint, so "reduce" is not an apply action —
# // it degrades to an out-of-band operator instruction.
# "action": "uninstall" // "uninstall" | "out-of-band"
# },
# "protection": {
# "branch": "main",
# "include_administrators": true,
# // The FULL protection payload recorded pre-flip (the exact body to PUT
# // back to repos/{repo}/branches/{branch}/protection). Captured verbatim
# // by --record-baseline. enforce_admins is also asserted on its own.
# "full": { ... },
# "required_status_checks": ["guard", "build-test"]
# }
# }
#
# Record a baseline FIRST (before the flip) with:
# scripts/p3_rollback.sh --record-baseline [--baseline FILE] [--repo OWNER/NAME] \
# [--env-name agent-apply] [--app-installation-id N]
# which captures the live workflow SHA, env reviewer ids, full branch protection,
# and App installation id into the baseline JSON (creating .security-review/ if
# absent). The restore surfaces then read from it.
#
# Exit non-zero on any failure (set -euo pipefail). Requires bash + gh + (git for
# the post-merge revert path). No standing box token is used — run from the Mac /
# operator host where `gh` is authenticated. The App UNINSTALL step additionally
# needs an APP JWT (NOT operator gh) — see surface 3.
set -euo pipefail
# ── defaults ─────────────────────────────────────────────────────────────────
APPLY=0
RECORD_BASELINE=0
BASELINE="${P3_ROLLBACK_BASELINE:-.security-review/p3-baseline.json}"
REPO_DEFAULT=""
if [ -n "${AGENT_TEAM_REPO_OWNER:-}" ] && [ -n "${AGENT_TEAM_REPO_NAME:-}" ]; then
REPO_DEFAULT="${AGENT_TEAM_REPO_OWNER}/${AGENT_TEAM_REPO_NAME}"
fi
REPO="${REPO_DEFAULT}"
SURFACE=""
ENV_NAME="agent-apply"
APP_INSTALLATION_ID=""
FLIP_PR=""
FLIP_BRANCH=""
FLIP_COMMIT=""
MERGED=0
PROG="$(basename "$0")"
say() { printf '\n\033[1;36m== %s\033[0m\n' "$*"; }
plan() { printf ' \033[1;33m[PLAN]\033[0m %s\n' "$*"; }
do_() { printf ' \033[1;32m[APPLY]\033[0m %s\n' "$*"; }
err() { printf '\033[1;31m!! %s\033[0m\n' "$*" >&2; }
usage() {
# Print the leading comment block (everything up to the first blank line after
# `set -euo pipefail`) as help. Kept simple: emit a concise synopsis.
cat <<USAGE
${PROG} — restore the privileged P3 surfaces to a recorded baseline.
Usage:
${PROG} <surface> [--apply] [--baseline FILE] [options]
Surfaces:
workflow restore the apply/verify flip (pre/post-merge + live revert)
environment restore the agent-apply environment (required reviewer ids + policy)
app neutralise the GitHub App (uninstall via App JWT, or out-of-band)
protection restore the FULL branch protection baseline (include_administrators ON)
all run workflow + environment + app + protection in order
incident premature-flip-that-RAN path (neutralise App, revert PR, audit, note)
Options:
--record-baseline capture current live state into the baseline JSON instead
of restoring (honours --apply)
--apply perform mutations (default: --dry-run, print plan only)
--dry-run print plan only (default)
--baseline FILE recorded baseline JSON (default: \$P3_ROLLBACK_BASELINE
or .security-review/p3-baseline.json)
--repo OWNER/NAME target repo (default: \$AGENT_TEAM_REPO_OWNER/_NAME)
--env-name NAME environment name for --record-baseline (default agent-apply)
--app-installation-id N App installation id to record for --record-baseline
--flip-pr N flip PR number (pre-merge close path)
--flip-branch REF flip PR head branch (pre-merge delete path)
--merged flip already merged (post-merge revert path)
--flip-commit SHA merged flip commit to revert (post-merge path)
-h, --help show this help
Operator ACK env vars (fail-closed by default; a partial/manual step must be
acknowledged so a restore is never silently weaker than the baseline):
AGENT_APPLY_APP_JWT=<jwt> App JWT for the automated App uninstall.
P3_ROLLBACK_OOB_ACK=1 acknowledge you will neutralise the App by hand
when no APP JWT is available (lets the other
surfaces restore instead of aborting).
P3_ROLLBACK_ALLOW_PARTIAL=1 accept a DEGRADED branch-protection restore
(enforce_admins only) when the baseline has no
protection.full — otherwise this is refused.
USAGE
}
# ── arg parsing ──────────────────────────────────────────────────────────────
# A no-op-safe parser: a single positional surface, then flags. Unknown flags
# are a hard error (fail closed — never silently ignore an option that would
# change destructive behaviour).
parse_args() {
while [ "$#" -gt 0 ]; do
case "$1" in
workflow|environment|app|protection|all|incident)
if [ -n "${SURFACE}" ]; then
err "surface already set to '${SURFACE}'; got extra '$1'"; return 2
fi
SURFACE="$1" ;;
--record-baseline) RECORD_BASELINE=1 ;;
--apply) APPLY=1 ;;
--dry-run) APPLY=0 ;;
--env-name) shift; ENV_NAME="${1:-}"; [ -n "${ENV_NAME}" ] || { err "--env-name needs a value"; return 2; } ;;
--env-name=*) ENV_NAME="${1#*=}" ;;
--app-installation-id) shift; APP_INSTALLATION_ID="${1:-}"; [ -n "${APP_INSTALLATION_ID}" ] || { err "--app-installation-id needs a value"; return 2; } ;;
--app-installation-id=*) APP_INSTALLATION_ID="${1#*=}" ;;
--baseline) shift; BASELINE="${1:-}"; [ -n "${BASELINE}" ] || { err "--baseline needs a value"; return 2; } ;;
--baseline=*) BASELINE="${1#*=}" ;;
--repo) shift; REPO="${1:-}"; [ -n "${REPO}" ] || { err "--repo needs a value"; return 2; } ;;
--repo=*) REPO="${1#*=}" ;;
--flip-pr) shift; FLIP_PR="${1:-}"; [ -n "${FLIP_PR}" ] || { err "--flip-pr needs a value"; return 2; } ;;
--flip-pr=*) FLIP_PR="${1#*=}" ;;
--flip-branch) shift; FLIP_BRANCH="${1:-}"; [ -n "${FLIP_BRANCH}" ] || { err "--flip-branch needs a value"; return 2; } ;;
--flip-branch=*) FLIP_BRANCH="${1#*=}" ;;
--flip-commit) shift; FLIP_COMMIT="${1:-}"; [ -n "${FLIP_COMMIT}" ] || { err "--flip-commit needs a value"; return 2; } ;;
--flip-commit=*) FLIP_COMMIT="${1#*=}" ;;
--merged) MERGED=1 ;;
-h|--help) usage; exit 0 ;;
*) err "unknown argument: $1"; usage >&2; return 2 ;;
esac
shift
done
# --record-baseline does not take a surface; restore modes require one.
if [ "${RECORD_BASELINE}" = "1" ]; then
if [ -n "${SURFACE}" ]; then
err "--record-baseline does not take a surface (got '${SURFACE}')"; return 2
fi
return 0
fi
if [ -z "${SURFACE}" ]; then
err "a surface is required (workflow|environment|app|protection|all|incident)"
usage >&2
return 2
fi
}
# ── helpers ──────────────────────────────────────────────────────────────────
# jget KEY — read a value from the baseline JSON via gh's bundled jq-less reader.
# We use `gh` only for API; for JSON parsing we prefer python3 (stdlib) so there
# is no jq dependency and behaviour is identical in tests.
jget() {
local expr="$1"
python3 - "$BASELINE" "$expr" <<'PY'
import json, sys
path, expr = sys.argv[1], sys.argv[2]
try:
with open(path, encoding="utf-8") as fh:
data = json.load(fh)
except FileNotFoundError:
print("", end="")
sys.exit(0)
cur = data
for part in expr.split("."):
if part == "":
continue
if isinstance(cur, dict) and part in cur:
cur = cur[part]
else:
cur = None
break
if cur is None:
print("", end="")
elif isinstance(cur, bool):
print("true" if cur else "false", end="")
elif isinstance(cur, (list, dict)):
print(json.dumps(cur), end="")
else:
print(cur, end="")
PY
}
require_baseline() {
if [ ! -f "${BASELINE}" ]; then
err "baseline file not found: ${BASELINE} (record it BEFORE the flip)"
return 1
fi
local repo_from_baseline
repo_from_baseline="$(jget repo)"
if [ -z "${REPO}" ]; then
REPO="${repo_from_baseline}"
fi
if [ -z "${REPO}" ]; then
err "no target repo: pass --repo OWNER/NAME or set repo in the baseline"
return 1
fi
# If both are present they must agree — refuse to restore against a repo the
# baseline was not recorded for (fail closed: a mismatched restore is worse
# than no restore).
if [ -n "${repo_from_baseline}" ] && [ "${repo_from_baseline}" != "${REPO}" ]; then
err "repo mismatch: baseline=${repo_from_baseline} requested=${REPO}"
return 1
fi
}
# require_keys KEY... — fail HARD if any baseline key is absent (MEDIUM-1). A
# partial baseline must never yield a partial/weaker restore (e.g. restoring only
# enforce_admins while silently dropping status checks): each restore_* asserts
# the keys IT needs up front, BEFORE any mutation, so a missing key aborts the
# whole surface instead of half-restoring it.
require_keys() {
local missing="" k
for k in "$@"; do
if [ -z "$(jget "${k}")" ]; then
missing="${missing} ${k}"
fi
done
if [ -n "${missing}" ]; then
err "baseline ${BASELINE} is missing required key(s):${missing}"
err " refusing a PARTIAL restore — record a complete baseline first (--record-baseline)."
return 1
fi
}
warn() { printf ' \033[1;35m[WARN]\033[0m %s\n' "$*" >&2; }
# redact_secrets — read text on stdin and mask anything that looks like a secret
# (App JWTs, GitHub tokens, Authorization headers, PEM blocks) to a placeholder.
# SEC-01 (CWE-532): the DEFAULT --dry-run mode echoes the gh/git argv into the
# plan output, and restore_app passes `-H "Authorization: Bearer ${JWT}"`, so the
# real App JWT would otherwise reach stdout/CI logs. This is applied CENTRALLY in
# run_or_plan (and anywhere else that echoes argv) so no secret can be printed.
redact_secrets() {
# sed: each pattern collapses the secret to a stable placeholder. Order matters
# (Authorization/Bearer first so the bare-token rules don't double-process it).
# * Bearer <token> -> Bearer <REDACTED>
# * Authorization: <anything> -> Authorization: <REDACTED>
# * ghp_/gho_/ghu_/ghs_/ghr_/github_pat_ tokens -> <REDACTED-TOKEN>
# * PEM private-key bodies -> <REDACTED-PEM>
sed -E \
-e 's/(Bearer)[[:space:]]+[A-Za-z0-9._~+/=-]+/\1 <REDACTED>/g' \
-e 's/([Aa]uthorization:)[[:space:]]*[^"'"'"']+/\1 <REDACTED>/g' \
-e 's/(gh[pousr]_|github_pat_)[A-Za-z0-9_]+/<REDACTED-TOKEN>/g' \
-e 's/-----BEGIN [A-Z ]*PRIVATE KEY-----[^-]*-----END [A-Z ]*PRIVATE KEY-----/<REDACTED-PEM>/g'
}
# redacted — emit "$*" with any secret masked. Use whenever argv is echoed.
redacted() {
printf '%s' "$*" | redact_secrets
}
# run_or_plan "<human description>" gh ... — print the plan; only execute on --apply.
# The plan branch echoes the FULL argv, so it is passed through redact_secrets
# first (SEC-01) — a Bearer JWT or token in "$@" never reaches stdout.
run_or_plan() {
local desc="$1"; shift
if [ "${APPLY}" = "1" ]; then
do_ "$(redacted "${desc}")"
"$@"
else
plan "$(redacted "${desc}: $*")"
fi
}
# assert_equal EXPECTED ACTUAL CONTEXT — fail closed on a post-restore mismatch.
assert_equal() {
local expected="$1" actual="$2" ctx="$3"
if [ "${expected}" != "${actual}" ]; then
err "POST-RESTORE ASSERT FAILED [${ctx}]: expected '${expected}', got '${actual}'"
return 1
fi
printf ' \033[1;32m[OK]\033[0m %s == %s (%s)\n' "${expected}" "${actual}" "${ctx}"
}
# ── surface 1: workflow flip ─────────────────────────────────────────────────
restore_workflow() {
say "1. Restore the apply/verify workflow flip"
require_keys workflow_path workflow_baseline_sha || return 1
local wf_path baseline_sha
wf_path="$(jget workflow_path)"
[ -n "${wf_path}" ] || wf_path=".github/workflows/agent-team-apply-verify.yml"
baseline_sha="$(jget workflow_baseline_sha)"
if [ "${MERGED}" = "1" ]; then
# POST-merge: revert the flip commit, push, re-run CI.
if [ -z "${FLIP_COMMIT}" ]; then
err "post-merge path needs --flip-commit SHA"; return 1
fi
run_or_plan "git revert the merged flip commit ${FLIP_COMMIT}" \
git revert --no-edit "${FLIP_COMMIT}"
run_or_plan "push the revert to ${REPO}" \
git push origin HEAD
run_or_plan "re-run CI on the revert for ${REPO}" \
gh workflow run --repo "${REPO}" "$(basename "${wf_path}")"
else
# PRE-merge: close the flip PR + delete its branch.
if [ -z "${FLIP_PR}" ]; then
err "pre-merge path needs --flip-pr N"; return 1
fi
run_or_plan "close the flip PR #${FLIP_PR} on ${REPO}" \
gh pr close "${FLIP_PR}" --repo "${REPO}" \
--comment "Reverting the P3 apply/verify flip to baseline." --delete-branch
if [ -n "${FLIP_BRANCH}" ]; then
run_or_plan "delete the flip head branch ${FLIP_BRANCH}" \
gh api -X DELETE "repos/${REPO}/git/refs/heads/${FLIP_BRANCH}"
fi
fi
# LIVE: the run-name/permissions edits are already on the branch — revert the
# workflow file to the recorded baseline SHA so the live YAML matches baseline.
if [ -n "${baseline_sha}" ]; then
run_or_plan "revert ${wf_path} to baseline SHA ${baseline_sha}" \
git checkout "${baseline_sha}" -- "${wf_path}"
# P3-IAC-08 (CWE-665): `git checkout <sha> -- <path>` only STAGES the revert
# in the local worktree/index — unlike the post-merge path it does NOT commit
# or push, so the remote live YAML is unchanged until a human lands it. Warn
# loudly so the restore is never assumed complete on the remote.
warn "LIVE-YAML revert of ${wf_path} is STAGED-ONLY (local worktree/index)."
warn " It is NOT committed or pushed — the remote workflow is UNCHANGED until you"
warn " manually commit + push (e.g. git commit -m 'revert P3 flip workflow' && git push)."
else
plan "(no workflow_baseline_sha recorded — skipping live YAML revert)"
fi
}
# resolve_reviewer_ids — emit a sorted, space-separated list of NUMERIC user ids
# for the environment's required reviewers. Prefers baseline-recorded ids
# (environment.required_reviewer_ids); for any baseline that only recorded logins
# (environment.required_reviewers), resolve login->id via `gh api users/{login}`.
# Sorting makes the post-restore compare order-independent.
resolve_reviewer_ids() {
local ids logins login id
ids="$(jget environment.required_reviewer_ids)" # JSON array of ints, or ""
logins="$(jget environment.required_reviewers)" # JSON array of strings, or ""
python3 - "$ids" "$logins" <<'PY'
import json, sys
ids_raw, logins_raw = sys.argv[1], sys.argv[2]
out = []
if ids_raw:
out = [str(int(x)) for x in json.loads(ids_raw)]
elif logins_raw:
# Marker: logins need resolving. Print each login prefixed so the caller
# can resolve via gh (we can't shell out from inside python here).
for login in json.loads(logins_raw):
print("LOGIN:" + str(login))
sys.exit(0)
print(" ".join(sorted(out, key=int)))
PY
}
# reviewers_put_args ID... — echo the repeatable -F args for the reviewers array
# in the exact field form gh expects: -F 'reviewers[][type]=User' -F 'reviewers[][id]=N'
reviewers_put_args() {
local id
for id in "$@"; do
printf -- '-F\nreviewers[][type]=User\n-F\nreviewers[][id]=%s\n' "${id}"
done
}
# ── surface 2: agent-apply environment ───────────────────────────────────────
restore_environment() {
say "2. Restore the agent-apply environment (required reviewer ids + policy)"
require_keys environment.name || return 1
# The reviewer can be recorded as numeric ids OR logins (the login->id resolve
# is an EQUIVALENT, not weaker, restore) — but at least one source is required,
# else there is nothing to restore the required reviewer FROM (MEDIUM-1).
if [ -z "$(jget environment.required_reviewer_ids)" ] \
&& [ -z "$(jget environment.required_reviewers)" ]; then
err "baseline has neither environment.required_reviewer_ids nor environment.required_reviewers"
err " — nothing to restore the required reviewer from; record a complete baseline."
return 1
fi
local env_name policy raw line id ids
env_name="$(jget environment.name)"
[ -n "${env_name}" ] || env_name="agent-apply"
policy="$(jget environment.deployment_branch_policy)"
# Resolve the baseline reviewer set to NUMERIC ids. If the baseline only
# recorded logins, resolve each login->id via `gh api users/{login}`.
raw="$(resolve_reviewer_ids)"
ids=""
if printf '%s' "${raw}" | grep -q '^LOGIN:'; then
while IFS= read -r line; do
[ -n "${line}" ] || continue
local login="${line#LOGIN:}"
if [ "${APPLY}" = "1" ]; then
id="$(gh api "users/${login}" --jq '.id' 2>/dev/null || true)"
if [ -z "${id}" ]; then
err "could not resolve reviewer login '${login}' to a numeric id"; return 1
fi
else
# Dry-run: we don't hit the network. Show the resolution we WOULD do.
plan "resolve reviewer login '${login}' -> id via 'gh api users/${login} --jq .id'"
id="<id:${login}>"
fi
ids="${ids:+${ids} }${id}"
done <<EOF
${raw}
EOF
else
ids="${raw}"
fi
# Sort numeric ids for an order-independent post-restore compare (only when we
# have real ids — dry-run placeholders are left as-is for display).
if [ "${APPLY}" = "1" ] && [ -n "${ids}" ]; then
ids="$(printf '%s\n' ${ids} | sort -n | tr '\n' ' ')"
ids="${ids% }"
fi
# Build the reviewers array in the exact -F field form gh requires, then PUT.
local -a put_args=()
if [ -n "${ids}" ] && [ "${APPLY}" = "1" ]; then
while IFS= read -r line; do
[ -n "${line}" ] || continue
put_args+=("${line}")
done < <(reviewers_put_args ${ids})
fi
run_or_plan "restore environment '${env_name}' required reviewer ids [${ids}], policy ${policy:-protected}" \
gh api -X PUT "repos/${REPO}/environments/${env_name}" \
"${put_args[@]}" -F "deployment_branch_policy=${policy:-protected}"
if [ "${APPLY}" = "1" ]; then
# Post-restore assert: compare the LIVE reviewer-id set to the baseline set.
# A failed restore (missing/extra reviewer) is now actually detected — not
# merely "the env name still exists".
local got_ids
got_ids="$(gh api "repos/${REPO}/environments/${env_name}" \
--jq '[.protection_rules[]? | select(.type=="required_reviewers")
| .reviewers[]? | select(.type=="User") | .reviewer.id] | sort | join(" ")' \
2>/dev/null || true)"
assert_equal "${ids}" "${got_ids}" "environment.required_reviewer_ids"
else
plan "(post-restore: assert LIVE env '${env_name}' reviewer ids == baseline [${ids}])"
fi
}
# ── surface 3: GitHub App installation ───────────────────────────────────────
# Honest model of what's actually possible against the GitHub REST API:
#
# * You CANNOT "rotate" the App's installation token from operator gh.
# DELETE /installation/token revokes the token you authenticate WITH — i.e.
# the installation token itself, not some other installation's token. The
# operator host `gh` (user OAuth / PAT) is NOT that token, so it returns 403.
# A leaked installation token is short-lived (<=1h) and self-expires; to kill
# it sooner you must either run that DELETE *with that token* (inside the
# runner) or neutralise the source of new tokens (below).
#
# * There is NO permission-reduction REST endpoint.
# PATCH /app/installations/{id}/permissions does NOT exist (404). Installation
# permissions are reduced only by editing the App in the UI/settings API and
# re-accepting, which is an out-of-band operator action.
#
# * The only programmatic NEUTRALISE is UNINSTALL:
# DELETE /app/installations/{id} — which requires an APP JWT (NOT operator gh,
# NOT an installation token, NOT a fine-grained PAT). Provide the JWT via
# $AGENT_APPLY_APP_JWT; this surface only --apply's the uninstall when it is
# set. Without it, the step degrades to an out-of-band instruction.
# Out-of-band ACK gate (MEDIUM-2): when neutralising the App needs a MANUAL
# operator action (no APP JWT for the uninstall, or action=out-of-band), --apply
# must not silently skip it. Require an explicit acknowledgement
# (P3_ROLLBACK_OOB_ACK=1) that the operator WILL perform the manual step, so the
# App is never left un-neutralised without a conscious sign-off. WITH the ack the
# surface returns 0 so the other surfaces still restore (the App step is operator-
# owed, loudly warned); WITHOUT it, --apply aborts this surface.
require_oob_ack() {
local what="$1"
if [ "${P3_ROLLBACK_OOB_ACK:-}" = "1" ]; then
warn "OUT-OF-BAND ACK accepted (P3_ROLLBACK_OOB_ACK=1): ${what}"
warn " the App is NOT neutralised by this script — you MUST do it by hand NOW."
return 0
fi
err "${what}"
err " set P3_ROLLBACK_OOB_ACK=1 to acknowledge you will perform the manual step"
err " (lets the other surfaces restore), or set \$AGENT_APPLY_APP_JWT for the automated uninstall."
return 1
}
# uninstall_app_via_curl SLUG INSTALLATION_ID — issue the App-JWT-authenticated
# DELETE /app/installations/{id} WITHOUT the JWT ever appearing on a command line
# (SEC-02 / CWE-214). The Authorization header is written to a 0600 temp file and
# passed to `curl -H @file`; argv carries only the filename. The file is shredded
# (or rm'd) on return via a trap. Honours --apply (dry-run only prints the plan,
# with the JWT redacted by run_or_plan-style masking).
uninstall_app_via_curl() {
local slug="$1" inst="$2"
if [ "${APPLY}" != "1" ]; then
# Dry-run: never write the header file; describe the action with NO secret.
plan "$(redacted "UNINSTALL App '${slug}' installation ${inst} via curl -H @<0600 hdrfile> (Authorization: Bearer ${AGENT_APPLY_APP_JWT})")"
plan " (JWT is written to a 0600 temp header file, never to the process argv)"
return 0
fi
do_ "UNINSTALL App '${slug}' installation ${inst} (App JWT via 0600 header file; not on argv)"
local hdr_file rc=0
hdr_file="$(mktemp "${TMPDIR:-/tmp}/p3-app-hdr.XXXXXX")"
# Tighten perms BEFORE writing the secret.
chmod 600 "${hdr_file}"
printf 'Authorization: Bearer %s\n' "${AGENT_APPLY_APP_JWT}" > "${hdr_file}"
curl -fsS -X DELETE \
-H @"${hdr_file}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"https://api.github.com/app/installations/${inst}" || rc=$?
# Always shred/remove the header file so the JWT does not linger on disk.
shred -u "${hdr_file}" 2>/dev/null || rm -f "${hdr_file}"
return "${rc}"
}
restore_app() {
say "3. Neutralise the GitHub App installation"
require_keys app.action || return 1
local action installation_id slug
action="$(jget app.action)"
[ -n "${action}" ] || action="uninstall"
installation_id="$(jget app.installation_id)"
slug="$(jget app.slug)"
# Be explicit that operator gh cannot revoke the installation token.
plan "(installation token: cannot be revoked by operator gh — DELETE /installation/token"
plan " revokes the token you AUTHENTICATE WITH; the short-lived install token self-expires."
plan " To kill it sooner, run that DELETE inside the runner with that token, or uninstall below.)"
case "${action}" in
uninstall)
if [ -z "${installation_id}" ] || [ "${installation_id}" = "0" ]; then
err "uninstall requested but no app.installation_id in baseline"; return 1
fi
# DELETE /app/installations/{id} requires an APP JWT — NOT operator gh.
if [ -n "${AGENT_APPLY_APP_JWT:-}" ]; then
# SEC-02 (CWE-214): keep the JWT OUT of the process argv (visible in
# ps/proc to any local user). `gh api` has no header-from-file mechanism
# and -H puts the value on the command line, so we issue the DELETE with
# `curl -H @headerfile` instead — the Authorization header (with the JWT)
# lives only in a 0600 temp file that is shredded on return. The argv
# carries only the file reference, never the secret.
uninstall_app_via_curl "${slug}" "${installation_id}"
else
# No JWT available: never pretend operator gh can do this. Plan only.
plan "UNINSTALL App '${slug}' installation ${installation_id} requires an APP JWT"
plan " (set \$AGENT_APPLY_APP_JWT). Without it: do it OUT-OF-BAND —"
plan " remove the installation in the org UI, or run:"
plan " gh api -X DELETE app/installations/${installation_id} -H 'Authorization: Bearer <APP_JWT>'"
if [ "${APPLY}" = "1" ]; then
require_oob_ack \
"uninstall needs an APP JWT (\$AGENT_APPLY_APP_JWT); operator gh cannot do this" || return 1
fi
fi
;;
out-of-band)
# Honest: no API path. Tell the operator exactly what to do by hand.
plan "OUT-OF-BAND App neutralise for '${slug}' (no REST endpoint reduces App permissions):"
plan " - remove the installation in the org UI (Settings > GitHub Apps > Configure > Uninstall), AND/OR"
plan " - rotate the App private key out-of-band (App settings > Generate a new private key,"
plan " update the AGENT_APPLY_APP_PRIVATE_KEY Actions secret), which invalidates new-token minting."
if [ "${APPLY}" = "1" ]; then
require_oob_ack \
"app.action 'out-of-band' has no API path — neutralise the App by hand" || return 1
fi
;;
*)
err "unknown app.action '${action}' (expected uninstall|out-of-band)"; return 1 ;;
esac
}
# ── surface 4: branch protection ─────────────────────────────────────────────
restore_protection() {
say "4. Restore the FULL branch protection baseline (include_administrators ON)"
require_keys protection.branch protection.include_administrators || return 1
# protection.full is the COMPLETE payload (status checks, PR reviews, ...).
# Restoring only enforce_admins when it is absent leaves a WEAKER posture than
# baseline (MEDIUM-1). Refuse that silent degrade by default; the operator must
# explicitly accept the enforce_admins-only restore via P3_ROLLBACK_ALLOW_PARTIAL=1.
if [ -z "$(jget protection.full)" ]; then
if [ "${P3_ROLLBACK_ALLOW_PARTIAL:-}" != "1" ]; then
err "baseline has no protection.full — an enforce_admins-only restore would leave a WEAKER"
err " posture (status checks / PR reviews NOT restored). Set P3_ROLLBACK_ALLOW_PARTIAL=1 to"
err " accept the degraded restore, or record a complete baseline (--record-baseline)."
return 1
fi
warn "DEGRADED protection restore (P3_ROLLBACK_ALLOW_PARTIAL=1): enforce_admins ONLY;"
warn " status checks / PR reviews are NOT restored — restore them by hand or re-record."
fi
local branch include_admins full
branch="$(jget protection.branch)"
[ -n "${branch}" ] || branch="$(jget default_branch)"
[ -n "${branch}" ] || branch="main"
include_admins="$(jget protection.include_administrators)"
full="$(jget protection.full)" # the full protection payload, captured verbatim
# The baseline MUST require admins to be included — a rollback that left admins
# exempt would be a weaker posture than baseline. Fail closed otherwise.
if [ "${include_admins}" != "true" ]; then
err "baseline protection.include_administrators is not true (got '${include_admins}'); refusing"
return 1
fi
# Restore the FULL protection object (not only enforce_admins): one PUT to
# repos/{repo}/branches/{branch}/protection with the recorded body. This rebuilds
# required_status_checks, required_pull_request_reviews, restrictions, etc.
if [ -n "${full}" ] && [ "${full}" != "null" ]; then
if [ "${APPLY}" = "1" ]; then
do_ "PUT full branch protection on ${branch} from baseline protection.full"
printf '%s' "${full}" | gh api -X PUT \
"repos/${REPO}/branches/${branch}/protection" --input -
else
plan "PUT full branch protection on ${branch} from baseline protection.full: ${full}"
fi
else
# No full payload recorded: at minimum re-enable enforce_admins so admins are
# not left exempt. Be explicit that this is a partial restore.
plan "(no protection.full recorded — partial restore: enforce_admins only)"
run_or_plan "enforce_admins ON for ${branch} (partial; record protection.full for a full restore)" \
gh api -X PUT "repos/${REPO}/branches/${branch}/protection/enforce_admins"
fi
if [ "${APPLY}" = "1" ]; then
# Assert enforce_admins is ON.
local got
got="$(gh api "repos/${REPO}/branches/${branch}/protection/enforce_admins" --jq '.enabled' 2>/dev/null || true)"
assert_equal "true" "${got}" "protection.include_administrators"
# If a full baseline was recorded, assert the LIVE protection matches it.
if [ -n "${full}" ] && [ "${full}" != "null" ]; then
local live_norm base_norm
live_norm="$(normalize_protection "$(gh api "repos/${REPO}/branches/${branch}/protection" 2>/dev/null)")"
base_norm="$(normalize_protection "${full}")"
assert_equal "${base_norm}" "${live_norm}" "protection.full"
fi
else
plan "(post-restore: assert enforce_admins.enabled == true on ${branch})"
plan "(post-restore: assert LIVE protection == baseline protection.full)"
fi
}
# normalize_protection — read a branch-protection JSON object on stdin and emit a
# canonical, comparable projection of the fields we restore. GitHub's GET adds
# url/metadata fields that a PUT body never carries, so we project only the
# semantically meaningful settings and sort, making the baseline-vs-live compare
# robust to representational noise.
normalize_protection() {
# The JSON is passed as $1 (NOT piped): a `python3 - <<'PY'` heredoc occupies
# stdin, so json.load(sys.stdin) would read the program text, not the data —
# which silently yielded "" for every input and made the post-restore assert
# vacuous. Read argv[1] instead (mirrors the other heredoc helpers here).
python3 - "${1-}" <<'PY'
import json, sys
raw = sys.argv[1] if len(sys.argv) > 1 else ""
if not raw.strip():
# Empty input (e.g. the live GET failed) — emit a distinct sentinel so the
# baseline-vs-live compare MISMATCHES and the restore is flagged, never a
# silent pass.
print("__NORMALIZE_EMPTY__")
sys.exit(0)
try:
d = json.loads(raw)
except Exception as exc:
print(f"__NORMALIZE_ERROR__:{exc}")
sys.exit(0)
def boolish(node, *path, key="enabled"):
cur = node
for p in path:
if not isinstance(cur, dict):
return None
cur = cur.get(p)
if isinstance(cur, dict):
return bool(cur.get(key))
if isinstance(cur, bool):
return cur
return None
proj = {
"enforce_admins": boolish(d, "enforce_admins"),
"required_status_checks_strict": (
(d.get("required_status_checks") or {}).get("strict")
if isinstance(d.get("required_status_checks"), dict) else None
),
"required_status_checks_contexts": sorted(
(d.get("required_status_checks") or {}).get("contexts", []) or []
) if isinstance(d.get("required_status_checks"), dict) else None,
"required_pull_request_reviews": (
(lambda r: {
"required_approving_review_count": r.get("required_approving_review_count"),
"dismiss_stale_reviews": r.get("dismiss_stale_reviews"),
"require_code_owner_reviews": r.get("require_code_owner_reviews"),
})(d["required_pull_request_reviews"])
if isinstance(d.get("required_pull_request_reviews"), dict) else None
),
"required_linear_history": boolish(d, "required_linear_history"),
"allow_force_pushes": boolish(d, "allow_force_pushes"),
"allow_deletions": boolish(d, "allow_deletions"),
}
print(json.dumps(proj, sort_keys=True, separators=(",", ":")))
PY
}
# ── incident: premature flip that RAN ────────────────────────────────────────
incident_premature_flip() {
say "INCIDENT — premature flip that RAN (rotate / revert / audit / restore / note)"
local branch note_path
branch="$(jget protection.branch)"
[ -n "${branch}" ] || branch="$(jget default_branch)"
[ -n "${branch}" ] || branch="main"
# (a) Neutralise the App FIRST — anything the premature run minted is suspect.
# NB: operator gh CANNOT revoke the installation token (DELETE
# /installation/token revokes the token you authenticate WITH). The minted
# token is short-lived and self-expires; the durable neutralise is to
# uninstall the App (App JWT) or rotate its private key out-of-band.
say "a. Neutralise the GitHub App (uninstall via App JWT, or out-of-band)"
restore_app
# (b) Revert any draft PR / branch the App opened. The draft PR head is the
# dispatcher's namespaced ref (agent-team/apply/<task_id>); close + delete.
say "b. Revert the draft PR / branch the App opened"
if [ -n "${FLIP_PR}" ]; then
run_or_plan "close the premature draft PR #${FLIP_PR}" \
gh pr close "${FLIP_PR}" --repo "${REPO}" \
--comment "Closing premature agent-apply draft PR (incident rollback)." \
--delete-branch
else
plan "(no --flip-pr given: list open agent-team/apply/* PRs to triage)"
run_or_plan "list open agent-apply draft PRs for triage" \
gh pr list --repo "${REPO}" --draft --search "head:agent-team/apply/" --json number,headRefName
fi
if [ -n "${FLIP_BRANCH}" ]; then
run_or_plan "delete the premature head branch ${FLIP_BRANCH}" \
gh api -X DELETE "repos/${REPO}/git/refs/heads/${FLIP_BRANCH}"
fi
# (c) Audit the Checks trail of the premature run (read-only — always run).
say "c. Audit the Checks trail (read-only)"
run_or_plan "audit recent apply/verify runs (Checks trail)" \
gh run list --repo "${REPO}" --workflow agent-team-apply-verify.yml \
--limit 20 --json databaseId,headBranch,event,conclusion,createdAt
# (d) Restore environment + branch protection to baseline.
say "d. Restore environment + protection to baseline"
restore_environment
restore_protection
# (e) File an incident note.
say "e. File an incident note"
note_path=".security-review/incidents/p3-premature-flip-$(date +%Y%m%dT%H%M%SZ).md"
if [ "${APPLY}" = "1" ]; then
do_ "write incident note ${note_path}"
mkdir -p "$(dirname "${note_path}")"
{
printf '# Incident: premature P3 apply/verify flip that RAN\n\n'
printf -- '- Repo: %s\n' "${REPO}"
printf -- '- Baseline: %s\n' "${BASELINE}"
printf -- '- Flip PR: %s\n' "${FLIP_PR:-<unknown>}"
printf -- '- Flip branch: %s\n' "${FLIP_BRANCH:-<unknown>}"
printf -- '- Recorded: %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
printf '\n## Actions taken\n'
printf -- '1. Neutralised the GitHub App (uninstall via App JWT, or out-of-band).\n'
printf -- ' NB: the minted installation token cannot be revoked by operator gh;\n'
printf -- ' it is short-lived and self-expires.\n'
printf -- '2. Closed the premature draft PR + deleted its branch.\n'
printf -- '3. Audited the apply/verify Checks trail.\n'
printf -- '4. Restored the agent-apply environment + branch protection.\n'
} > "${note_path}"
else
plan "write incident note to ${note_path} (repo/flip/baseline + actions taken)"
fi
}
# ── --record-baseline ────────────────────────────────────────────────────────
# Capture the CURRENT live state into the baseline JSON: the live workflow SHA,
# the env required-reviewer NUMERIC ids, the FULL branch protection object, and
# the App installation id. Honours --apply (default --dry-run just prints what it
# would capture). Creates .security-review/ if absent.
record_baseline() {
say "RECORD BASELINE — capture current live state into ${BASELINE}"
# Resolve a repo: --repo, or env default. The baseline file may not exist yet,
# so we do NOT call require_baseline here.
if [ -z "${REPO}" ]; then
err "no target repo: pass --repo OWNER/NAME (baseline does not exist yet to read it from)"
return 1
fi
local branch wf_path
branch="main"
wf_path=".github/workflows/agent-team-apply-verify.yml"
if [ "${APPLY}" != "1" ]; then
plan "capture live workflow SHA for ${wf_path} via 'gh api repos/${REPO}/contents/${wf_path} --jq .sha'"
plan "capture env '${ENV_NAME}' required reviewer NUMERIC ids"
plan "capture FULL branch protection for ${branch}"
plan "capture App installation id (${APP_INSTALLATION_ID:-<--app-installation-id N>})"
plan "write baseline JSON to ${BASELINE} (creating $(dirname "${BASELINE}") if absent)"
return 0
fi
do_ "capture live state into ${BASELINE}"
mkdir -p "$(dirname "${BASELINE}")"
local wf_sha reviewer_ids protection_full
wf_sha="$(gh api "repos/${REPO}/contents/${wf_path}" --jq '.sha' 2>/dev/null || true)"
reviewer_ids="$(gh api "repos/${REPO}/environments/${ENV_NAME}" \
--jq '[.protection_rules[]? | select(.type=="required_reviewers")
| .reviewers[]? | select(.type=="User") | .reviewer.id] | sort' \
2>/dev/null || echo '[]')"
[ -n "${reviewer_ids}" ] || reviewer_ids='[]'
protection_full="$(gh api "repos/${REPO}/branches/${branch}/protection" 2>/dev/null || echo 'null')"
[ -n "${protection_full}" ] || protection_full='null'
# Assemble the baseline JSON deterministically with python3 (stdlib).
REPO="${REPO}" BRANCH="${branch}" WF_PATH="${wf_path}" WF_SHA="${wf_sha}" \
ENV_NAME="${ENV_NAME}" REVIEWER_IDS="${reviewer_ids}" \
APP_INSTALLATION_ID="${APP_INSTALLATION_ID}" PROTECTION_FULL="${protection_full}" \
python3 - "${BASELINE}" <<'PY'
import json, os, sys
out_path = sys.argv[1]
try:
reviewer_ids = json.loads(os.environ.get("REVIEWER_IDS") or "[]")
except Exception:
reviewer_ids = []
try:
protection_full = json.loads(os.environ.get("PROTECTION_FULL") or "null")
except Exception:
protection_full = None
include_admins = bool(
isinstance(protection_full, dict)
and isinstance(protection_full.get("enforce_admins"), dict)
and protection_full["enforce_admins"].get("enabled")
)
inst = os.environ.get("APP_INSTALLATION_ID") or ""
baseline = {
"repo": os.environ["REPO"],
"default_branch": os.environ["BRANCH"],
"workflow_path": os.environ["WF_PATH"],
"workflow_baseline_sha": os.environ.get("WF_SHA") or "",
"environment": {
"name": os.environ["ENV_NAME"],
"required_reviewer_ids": reviewer_ids,
"deployment_branch_policy": "protected",
},
"app": {
"slug": os.environ["ENV_NAME"],
"installation_id": int(inst) if inst.isdigit() else 0,
"action": "uninstall",
},
"protection": {
"branch": os.environ["BRANCH"],
"include_administrators": include_admins,
"full": protection_full,
},
}
with open(out_path, "w", encoding="utf-8") as fh:
json.dump(baseline, fh, indent=2, sort_keys=True)
fh.write("\n")
print(f" wrote {out_path}")
PY
printf ' \033[1;32m[OK]\033[0m baseline recorded to %s\n' "${BASELINE}"
}
# ── main ─────────────────────────────────────────────────────────────────────
main() {
parse_args "$@"
if [ "${RECORD_BASELINE}" = "1" ]; then
if [ "${APPLY}" = "1" ]; then
say "MODE: --record-baseline --apply (will CAPTURE live state) on repo ${REPO}"
else
say "MODE: --record-baseline --dry-run (plan only) on repo ${REPO}"
fi
record_baseline
say "Done (record-baseline, $([ "${APPLY}" = "1" ] && echo apply || echo dry-run))."
return 0
fi
require_baseline
if [ "${APPLY}" = "1" ]; then
say "MODE: --apply (mutations WILL be performed) on repo ${REPO}"
else
say "MODE: --dry-run (plan only; no mutations) on repo ${REPO}"
fi
case "${SURFACE}" in
workflow) restore_workflow ;;
environment) restore_environment ;;
app) restore_app ;;
protection) restore_protection ;;
all)
restore_workflow
restore_environment
restore_app
restore_protection
;;
incident) incident_premature_flip ;;
*) err "unhandled surface: ${SURFACE}"; return 2 ;;
esac
say "Done (${SURFACE}, $([ "${APPLY}" = "1" ] && echo apply || echo dry-run))."
}
main "$@"