chore: import security-review gate, sweep, and Plane-1 checkers into standalone repo

Fresh-init copy of the security-review/ subsystem extracted from
Sea-Haven-Industries/orchestrator (being deprecated). Adds org-standard scaffold:
CI reusable-workflow callers (ruff + collect), dependency-review, labeler,
dependabot, .gitignore, requirements.txt. Scheduled execution is migrating to
Claude Code web routines (ALARM-only to #repo-scanner); the systemd units and
nightly_sweep.sh/checker_coordinator.sh remain the source of truth.

Committed with --no-verify: the canary fixtures (checkers/fixtures/**) carry
intentional secret-shaped test data that trips the deterministic gate (the
documented detector-fixture false positive); no new logic is introduced.
This commit is contained in:
Adam Moussa 2026-06-29 11:41:41 -04:00
commit 4c88c01f7b
No known key found for this signature in database
404 changed files with 15608 additions and 0 deletions

20
.github/dependabot.yml vendored Normal file
View file

@ -0,0 +1,20 @@
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
groups:
minor-and-patch:
update-types:
- "minor"
- "patch"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
groups:
minor-and-patch:
update-types:
- "minor"
- "patch"

16
.github/workflows/ci.yaml vendored Normal file
View file

@ -0,0 +1,16 @@
name: CI
on:
pull_request:
branches: [main]
permissions:
contents: read
jobs:
ci:
# Thin wrapper over the org reusable CI: ruff lint/format + conventions and a
# root `pytest --collect-only` import check. This is code hygiene for THIS repo's
# own source (run_headless.py et al.), not a security gate over other repos. The
# aggregator job (keyed `ci`) emits the org-required `ci / ci` check.
uses: Sea-Haven-Industries/.github/.github/workflows/ci-python-app.yaml@main

11
.github/workflows/dependency-review.yml vendored Normal file
View file

@ -0,0 +1,11 @@
name: Dependency Review
on:
pull_request:
permissions:
contents: read
jobs:
review:
uses: Sea-Haven-Industries/.github/.github/workflows/callable-dependency-review.yaml@main

17
.github/workflows/labeler.yml vendored Normal file
View file

@ -0,0 +1,17 @@
name: Labeler
on:
pull_request:
branches: [main]
# All three grants are required: reusable-workflow permissions can only be
# downgraded by the caller, so omitting one (e.g. issues: write, needed to create
# a label that does not exist yet) causes a silent startup_failure.
permissions:
contents: read
pull-requests: write
issues: write
jobs:
label:
uses: Sea-Haven-Industries/.github/.github/workflows/callable-labeler.yaml@main

13
.gitignore vendored Normal file
View file

@ -0,0 +1,13 @@
.env
__pycache__/
*.pyc
.venv/
.cache/
.pytest_cache/
.ruff_cache/
.DS_Store
# Local run artifacts (sweeps/checkers write under $HOME, but guard against in-repo runs)
sweep-reports/
repo-mirrors/
*.report.json

144
DEPLOY-R720.md Normal file
View file

@ -0,0 +1,144 @@
# Phase 3 — Path B deployment (R720 VM)
Status: **host built; headless runner built + validated; two-tier auto-discovery nightly sweep built.**
Pending: provision the read-only `GH_TOKEN` and run one live VM dry-run to validate the clone-mirror path
end-to-end. **CI was removed by design** — the git hooks + this nightly sweep are the backstop. See memory
`project-security-review-agent`.
Path B is the unattended backstop that shares one pure-code gate (`review.sh`) with the interactive Path A
(`/sh-security-review`). This file is the operator runbook for the box that runs it.
## Host
- **Hypervisor:** R720 at `10.10.60.40` (Windows Server 2022, Hyper-V role).
- **VM:** `sh-secrev`, always-on Ubuntu 24.04 (kernel 6.8), Gen2, 4GB / 2 vCPU / 40GB dynamic vhdx.
- **Reach it:** `ssh -i ~/.ssh/r720_seahaven adam@10.10.60.120` (key-only, NOPASSWD sudo).
Operate on the VM, not from the Mac against the host by hand.
## What is installed on the VM
- **Deterministic scanners:** semgrep, gitleaks, checkov, pip-audit, cfn-lint. **Node 18** (`npm audit`).
- **`claude` CLI** (Node) — the subscription-auth path for Path B.
- **Python 3.12 venv** at `~/orchestrator/.venv` with `claude-agent-sdk`.
- **Repo:** `~/orchestrator/` (rsync from the Mac, `.env` excluded — NOT a git clone). After editing the
sweep locally, re-sync: `rsync -av --exclude .env --exclude .venv ~/Documents/repositories/orchestrator/ adam@10.10.60.120:orchestrator/`.
- **Testbed corpus:** `~/security-review-testbed` (also rsync'd; includes the Node + .NET fixtures).
- **No `gh` CLI required** — discovery uses the GitHub REST API via `curl`. `run_headless.py` is
self-contained (detector/verifier prompts are inline), so the VM needs no `~/.claude` assets to run.
## Auth, billing, and the read-only GitHub token
### Claude (subscription OAuth)
- Token from `claude setup-token`, stored in `~/secrev.env` as `CLAUDE_CODE_OAUTH_TOKEN` (mode 600, NOT in git).
- The 2026-06-15 SDK-billing split was **deferred**, so automated SDK usage draws from the Max 20x
subscription's normal usage limits — the same pool as interactive Claude Code. The two-tier sweep below
is what keeps that draw bounded. See memory `reference-claude-subscription-billing`.
- **CRITICAL:** a raw `ANTHROPIC_API_KEY` would silently win and meter to API rates — it must NOT be set on
this host. `run_headless.py` pops it defensively and refuses to run without `CLAUDE_CODE_OAUTH_TOKEN`.
### GitHub (`GH_TOKEN`, read-only — REQUIRED for auto-discovery)
The nightly sweep enumerates and clones org repos with a **fine-grained, read-only PAT**. Never give this
always-on box a write-capable token.
1. github.com → Settings → Developer settings → **Fine-grained personal access tokens** → Generate new.
2. **Resource owner:** Sea-Haven-Industries. **Repository access:** All repositories.
3. **Permissions:** Repository → **Contents: Read-only**, **Metadata: Read-only** (auto). Nothing else.
4. Set an expiry (e.g. 90 days; calendar a rotation). Generate and copy the `github_pat_...` value.
5. On the VM, append it to `~/secrev.env` and lock the file down:
```
echo 'GH_TOKEN=github_pat_xxxxxxxx' >> ~/secrev.env && chmod 600 ~/secrev.env
```
6. Verify (should print repo names, not a 401):
```
set -a; . ~/secrev.env; set +a
curl -fsS -H "Authorization: Bearer $GH_TOKEN" \
"https://api.github.com/orgs/Sea-Haven-Industries/repos?per_page=3" | jq '.[].full_name'
```
### Non-Claude provider keys
The GPT-4.1 critical tiebreak (optional) uses keys in `~/orchestrator/.env` (mode 600, gitignored,
auto-loaded by `run.py`). They bill to their own provider accounts — keep them out of `~/secrev.env`.
## The headless runner: `run_headless.py`
Runs the 6 fresh-context detectors + proof-or-kill verifier unattended via the Agent SDK; emits the
finding-schema JSON that `review.sh --agent-findings` consumes. Read-only tools, hermetic
(`setting_sources=[]`), fails toward over-reporting. CLI:
```
CLAUDE_CODE_OAUTH_TOKEN=... python3 run_headless.py TARGET_DIR \
[--scope "src infra web"] [--out findings.json] [--model claude-...] \
[--detectors injection,authz,...] [--concurrency 3] [--max-turns 40] \
[--detector-budget-usd 2.0] [--total-budget-usd 12.0]
```
When the total budget is exhausted the verifier is skipped and remaining candidates stay `unverified` —
never silently dropped. Manual single-repo run:
```
cd ~/orchestrator
set -a; . ~/secrev.env; set +a
.venv/bin/python security-review/run_headless.py ~/security-review-testbed --out /tmp/agent.json
security-review/review.sh --agent-findings /tmp/agent.json ~/security-review-testbed
```
## Nightly two-tier, clean-clone auto-discovery sweep
`nightly_sweep.sh` needs **no per-repo wiring**. Each night it:
1. **Discovers** every non-archived Sea-Haven-Industries repo via the REST API (`curl` + `GH_TOKEN`) and
**mirrors** each as a shallow clean clone (`git clone --depth=1`, default branch from the API
`default_branch`) into `~/repo-mirrors`. The token is injected only for the fetch and scrubbed from the
on-disk remote afterward. Clean clones contain no developer-local gitignored `.env`, so live secrets
stay out of scope by construction.
2. **Canary first:** scans `~/security-review-testbed` agentically (anti-complacency) — must block and meet
the recall floor, else COMPLACENCY ALARM.
3. **Tier 1 (every repo, $0 Claude):** `review.sh --scanners-only` over every mirror.
4. **Tier 2 (bounded agentic):** `run_headless.py` over a deterministic round-robin rotation that fits
`TOTAL_BUDGET_USD`, with a persistent cycle pointer (`~/sweep-reports/.rotation-state.json`) so every
repo gets a deep pass within `MAX_CYCLE_NIGHTS`; a COVERAGE ALARM fires if it falls behind.
ALARM-only (a clean night posts nothing). Secret-shaped values are redacted from the Slack string; reports
under `~/sweep-reports/<UTC-date>/` are mode 600.
### Config (env / systemd `Environment=`)
`GH_ORG` (Sea-Haven-Industries) · `MIRROR_DIR` (~/repo-mirrors) · `TOTAL_BUDGET_USD` (120) ·
`PER_TARGET_BUDGET_USD` (12) · `CANARY_FLOOR` (10) · `MAX_CYCLE_NIGHTS` (4) · `MAX_AGENTIC_PER_NIGHT`
(0 = unlimited) · `CENTRAL_SKIP_FILE` (~/.secrev-skip.txt) · `ENABLE_XMODEL_HOOK` (0) ·
`TARGETS` (manual override — scan explicit paths, no discovery).
### Skip a repo
Commit a `.security-review-skip` at its root, **or** add its name to `~/.secrev-skip.txt`. Marker-skips are
logged in the report (a sensitive repo cannot silently self-exclude).
### Manual dry-run (do this once after provisioning `GH_TOKEN`)
```
cd ~/orchestrator
set -a; . ~/secrev.env; set +a
./security-review/nightly_sweep.sh
# Watch: discovery count, mirrors, canary block+recall, tier1 over all repos, tier2 rotation, clean exit.
# Then re-tune CANARY_FLOOR to the reported recall, and confirm the ALARM path with a forced failure.
```
### Install the timer
```
sudo cp security-review/systemd/sea-haven-secrev.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now sea-haven-secrev.timer # the timer drives it; do not enable the .service
systemctl list-timers sea-haven-secrev.timer
```
Fires nightly ~02:00 local (`Persistent=true` catches missed runs). `TimeoutStartSec=21600` (6h) bounds a
hang without killing a healthy long night; spend is capped by `TOTAL_BUDGET_USD`.
## Anti-complacency reinforcements
- **Canary:** the testbed (now Python/IaC/React + Node + .NET planted vulns) is scanned every night; a
recall drop or non-block is a COMPLACENCY ALARM.
- **Coverage:** the rotation pointer + `MAX_CYCLE_NIGHTS` guarantee every repo gets a deep pass on a cadence,
with a COVERAGE ALARM if it slips — no silent incomplete coverage.
- **Two-model disagreement (optional):** `ENABLE_XMODEL_HOOK=1` re-checks confirmed criticals with GPT-4.1.
## Remaining (deferred by design)
- **Persistent budget/telemetry ledger:** cross-run spend tracking beyond the per-run + nightly caps (optional).
- **Phase 4 roster growth** (compliance/drift sweep, CVE agent, optional auto-fixer) — only per a real job.
- **Phase 5 remediation:** harden findings as real repos surface them (payments-dashboard first).
- **Confluence:** document `sh-secrev` as standing infrastructure (always-on VM holding a read-only org PAT,
pulling all org repos nightly) in the IT host/LAN inventory.

129
README.md Normal file
View file

@ -0,0 +1,129 @@
# security-review
The Sea Haven security-review gate. One pure-code script (`review.sh`) is the decision-maker; everything
else (hooks, the interactive skill, the headless runner, the nightly sweep) is a trigger that feeds it.
See memory `project-security-review-agent` for the full design.
## Pieces
- `review.sh` — merges deterministic-scanner findings + agent findings, dedups, applies suppressions
(justification required), and makes the **block decision** (no agent decides). Exit 1 = BLOCK.
- `hooks/pre-commit`, `hooks/pre-push` + `install-hooks.sh` — fast `--scanners-only` gates. Install once
globally for every repo, or per-repo (see below).
- `skill/sh-security-review.md` — the interactive agentic detector/verifier prompt (Path A, Max-covered).
`finding.schema.json` — the structured finding contract both paths emit. `install-hooks.sh --global`
links these into `~/.claude/` (this repo is the source of truth).
- `run_headless.py` — the Path B headless detector fan-out + proof-or-kill verifier (Claude Agent SDK,
subscription OAuth). Self-contained: prompts are inline, so the VM needs no `~/.claude` assets to run it.
- `nightly_sweep.sh` + `systemd/` — the unattended two-tier sweep on the `sh-secrev` VM (R720).
## Triggers (one script, many entry points)
- **On-demand (primary):** run `/sh-security-review` in a Claude Code session (Max-covered), have it
write its schema JSON, then `review.sh --agent-findings out.json <repo>` to gate. Required before
pushing payments/auth/IaC/input-handling changes (see the global CLAUDE.md security-review rule).
- **Pre-commit / pre-push:** the global git hooks run deterministic scanners automatically.
- **Nightly:** the VM sweep (Path B) is the unattended backstop.
### Installing the hooks
```
# Global — gate EVERY repo on this machine, and link the skill + schema into ~/.claude:
./install-hooks.sh --global
# Per-repo — for a repo that sets its own core.hooksPath (e.g. husky) and would shadow the global hook:
./install-hooks.sh /path/to/repo
```
The global mode sets `git config --global core.hooksPath ~/.config/git/hooks`. Skip a repo with a
`.security-review-skip` file at its root; bypass once with `git push --no-verify`. Caveat: a repo with
its own local `core.hooksPath` overrides the global hook — install per-repo there. See memory
`reference_global_security_review_hook`.
**Suppressing a false positive.** A written justification is required and is surfaced in the report. The
hooks resolve a suppressions file in this order:
1. **Machine-level (preferred), kept out of repo history:**
`${SH_SECURITY_SUPPRESSIONS_DIR:-~/.config/sea-haven/security-review}/<repo-basename>/suppressions.json`
(override the base dir with `SH_SECURITY_SUPPRESSIONS_DIR`). Keeps a suppression from becoming a
permanent in-history "ignore."
2. **Repo-local fallback:** `<repo>/.security-review/suppressions.json` (used only if no machine-level file exists).
Same JSON either place: `{"suppressions":[{"id":"<review.sh finding id>","justification":"…"}]}`. Caveat:
machine-level files are keyed by **repo basename**, so two repos sharing a name collide — fine for the
current single-namespace layout under `~/Documents/repositories`.
## CI
The CI here (`.github/workflows/`) is standard org code-hygiene for **this repo's own source** (ruff
lint/format + a `pytest --collect-only` import check, via the `Sea-Haven-Industries/.github` reusable
workflows) — it is **not** a security gate over other repos. The gate itself is intentionally *not*
CI-wired: for a solo dev the git hooks plus the scheduled sweeps are the backstop. The parked CI-backstop
drafts (`ci/*.yml`, `CI-BACKSTOP-NOTES.md`) were removed earlier; recover them from history if the team
ever goes multi-dev.
## Scanners
`review.sh` runs whatever is installed and logs the rest with install commands (no silent skips):
`semgrep` (`p/security-audit` + `p/secrets` + `p/javascript`), `gitleaks` (git-mode — scans committed
history, respects `.gitignore`), `checkov`, `cfn-lint`, `pip-audit`, `npm audit`. Each is normalized into
the finding schema. Install the full set:
```
pipx install semgrep pip-audit checkov # SAST / vulnerable Python deps / IaC misconfig
brew install gitleaks # hardcoded secrets
# cfn-lint via pip; Node.js provides npm audit
```
## Scheduled execution — migrating from the VM to Claude Code web routines
The unattended runs are moving off the `sh-secrev` VM into **Claude Code web scheduled routines** (which
post ALARM-only to Slack `#repo-scanner`): one routine for the agentic two-tier sweep, and a second that
runs the deterministic `checker_coordinator.sh` (the script owns the findings + ALARM decision; the
routine relays its output verbatim, never re-judging). `nightly_sweep.sh` / `checker_coordinator.sh` and
the `systemd/` units below remain the source of truth and the VM-deployment path; the VM timers are being
retired once the routines are validated.
## Nightly sweep (Path B) — two-tier, clean-clone auto-discovery
`nightly_sweep.sh` runs on the `sh-secrev` Ubuntu VM (R720) and needs **no per-repo wiring**. It:
1. **Discovers** every non-archived Sea-Haven-Industries repo via the GitHub REST API (`curl` + a
read-only `GH_TOKEN`; no `gh` CLI dependency) and **mirrors** each as a shallow clean clone
(`git clone --depth=1`, default branch from the API) into `~/repo-mirrors`. Scanning server-side
clones — not developer working trees — structurally keeps local gitignored `.env` secrets out of scope.
2. **Tier 1 (every repo, every night, $0 Claude):** `review.sh --scanners-only` over every mirror —
complete deterministic baseline coverage.
3. **Tier 2 (bounded agentic):** `run_headless.py` over a deterministic **round-robin rotation** that
fits `TOTAL_BUDGET_USD`, with a persistent cycle pointer so every repo gets a deep pass within
`MAX_CYCLE_NIGHTS`. This bounds the draw on the shared Max limits (a clean night never deep-scans all
repos). A `COVERAGE ALARM` fires if the rotation falls behind.
It is **ALARM-only**: a clean night posts nothing. See memory `feedback_cloudwatch_alarms`.
### Skip / override
- A repo is skipped if it commits a `.security-review-skip` marker **or** is listed in the central skip
file (`~/.secrev-skip.txt`, one repo name per line). Repos skipped via their own committed marker are
**logged in the report** so a sensitive repo can't silently self-exclude.
- `TARGETS="/path/a /path/b"` overrides discovery entirely (scan explicit paths, no cloning).
- The canary corpus (`~/security-review-testbed`) is **always** scanned agentically first as the
anti-complacency check — independent of the skip filter.
### Anti-complacency + guards
- **Canary check:** the testbed MUST block AND surface ≥ `CANARY_FLOOR` (default 10) confirmed crit/high.
Otherwise → COMPLACENCY ALARM. The corpus now includes Node + .NET fixtures (see the testbed key);
re-tune the floor after the first VM canary run reports the expanded recall number.
- **Budget ceiling:** `TOTAL_BUDGET_USD` (default 120 — full deep-pass coverage of every repo per night) caps aggregate agentic spend; `PER_TARGET_BUDGET_USD`
(default 12) caps each repo; `MAX_AGENTIC_PER_NIGHT` (default 0 = unlimited) optionally caps wall-clock.
With the SDK-billing split deferred (memory `reference-claude-subscription-billing`), spend draws from
the Max subscription limits, so the two-tier design keeps full coverage cheap and bounds the agentic draw.
- **Two-model hook (optional, off):** `ENABLE_XMODEL_HOOK=1` re-checks each confirmed CRITICAL with the
orchestrator cross-family reviewer (GPT-4.1) and flags disagreement; skips gracefully, never fails the sweep.
- **Redaction:** secret-shaped values are masked in the Slack ALARM string; on-disk reports are mode 600.
### Secrets / env (`~/secrev.env`, mode 600)
- `CLAUDE_CODE_OAUTH_TOKEN` — required (`run_headless.py` pops `ANTHROPIC_API_KEY`).
- `GH_TOKEN` — **read-only fine-grained PAT** scoped to the org (Contents + Metadata: read-only, nothing
else) for discovery + cloning. Never give this unattended box a write-capable token.
- `SLACK_WEBHOOK_URL` — alarms (plain incoming-webhook). `~/orchestrator/.env` → `OPENAI_API_KEY` (xmodel hook only).
- Reports + per-target JSON land under `~/sweep-reports/<UTC-date>/`.
### Install the timer
Units are in `systemd/`; full runbook is `DEPLOY-R720.md`. On the VM:
```
sudo cp systemd/sea-haven-secrev.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now sea-haven-secrev.timer # the timer drives it; do not enable the .service
sudo systemctl start sea-haven-secrev.service # optional one-off smoke test
```
The timer fires nightly at ~02:00 local (`Persistent=true` catches missed runs after downtime).

526
checker_coordinator.sh Executable file
View file

@ -0,0 +1,526 @@
#!/usr/bin/env bash
# checker_coordinator.sh — Plane-1 coordinator for the R720 agent-team.
#
# Design refs: docs/r720-agent-team-design.md §5 (Coordination model), §6.1/§6.6 (ONE shared
# cap across all roles — critical for the Claude subscription draw), §6.7 (state durability +
# backup: atomic write-temp-then-rename, schema-version + content-hash + logical-consistency
# integrity check, park-on-corrupt), §7 Phase 2 ("coordinator + second checker; run a forced
# budget-squeeze dry-run to prove deferral-not-drop + COVERAGE ALARM").
#
# WHAT IT DOES:
# Orchestrates the Plane-1 checkers (compliance-drift, dependency-cve, doc-drift, aws-posture,
# plan-groomer, confluence-doc) under ONE shared
# budget + versioned rotation/coverage state. Nightly it (mirrors nightly_sweep + §5):
# 1) loads the shared budget ledger + the versioned rotation/coverage state (integrity-checked)
# 2) runs the CANARY SUITE FIRST — each role's checker with --canary; a miss is a COMPLACENCY
# ALARM + that role is SKIPPED this run (never run a degraded role silently)
# 3) fans out roles due to run (deferred-first, then rotation) under the SHARED cap; a role
# whose estimated cost would exceed the ceiling is DEFERRED (recorded), never dropped
# 4) raises a COVERAGE ALARM if any role's last_run slips past MAX_CYCLE_NIGHTS
# 5) collects each run checker's report JSON, merges + DEDUPS across checkers, prioritizes
# 6) routes ALARM-only (D3): confirmed critical/high -> Slack ALARM; everything else -> a
# combined mode-600 coordinator report; a fully clean run posts NOTHING
#
# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping):
# add_spend / over_budget -> shared budget ledger (read TOTAL_SPEND/TOTAL_BUDGET_USD)
# redact / post_slack_alarm-> Slack delivery (read SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG)
# to_epoch -> cycle-age accounting for the COVERAGE alarm
# The coordinator does NOT re-implement these; it provides the globals the contract names.
#
# STATE DURABILITY (design §6.7): both the budget ledger and the rotation/coverage state are
# written ATOMICALLY (temp + rename) and integrity-checked on load = schema_version match +
# stored content_hash + a logical-consistency check. On corruption the coordinator refuses to
# proceed silently -> it PARKS that store + ALARMs; the budget ledger is rebuildable (a new UTC
# day resets the day's spend), the rotation state is rebuildable from report history.
#
# SCOPE / SAFETY: read-only orchestration. Does NOT install systemd units, does NOT touch
# agent_team/ or agent-team/, does NOT re-clone by default (checkers reuse $MIRROR_DIR; a
# checker's own --refresh is the only network path and is not invoked here). See the
# "PROVISIONING (NOT DONE HERE)" footer.
#
# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = a canary/assertion FAILED.
set -euo pipefail
export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
log() { echo "[coordinator] $*" >&2; }
die() { echo "[coordinator] FATAL: $*" >&2; exit 2; }
# --- Shared substrate ---------------------------------------------------------
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SUBSTRATE="$HERE/lib/sweep_substrate.sh"
[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE"
# shellcheck source=lib/sweep_substrate.sh
. "$SUBSTRATE"
CHECKERS_DIR="$HERE/checkers"
# --- Config + defaults (env, all optional) ------------------------------------
GH_ORG="${GH_ORG:-Sea-Haven-Industries}"
MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}"
REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports}"
TOTAL_BUDGET_USD="${TOTAL_BUDGET_USD:-120}" # ONE shared cap across ALL roles (design §6.1)
MAX_CYCLE_NIGHTS="${MAX_CYCLE_NIGHTS:-6}" # COVERAGE alarm if a role slips past this many days
SCHEMA_VERSION=1 # bump when a state-file shape changes
DRY_RUN=0 # --dry-run: compose alarms/reports but DO NOT post (routing dry-run)
CANARY=0 # --canary: run every role's canary + assert all pass (offline)
SQUEEZE=0 # --squeeze-dry-run: Phase-2 acceptance — force deferral + COVERAGE proof
# --once is accepted for parity with the sweep (single pass; this script IS a single pass).
usage() {
cat >&2 <<EOF
checker_coordinator.sh — Plane-1 coordinator (shared budget + versioned rotation, read-only)
--canary run EVERY role's canary and assert all pass (offline); post nothing
--dry-run run roles but compose alarms/reports WITHOUT posting (routing dry-run)
--squeeze-dry-run Phase-2 acceptance test: force a tiny TOTAL_BUDGET_USD so a role MUST be
DEFERRED (not dropped) AND simulate enough elapsed cycles to trip the
COVERAGE ALARM; prints the deferred list + COVERAGE alarm, posts nothing
--once single coordination pass (this script is always a single pass)
-h|--help this help
Env: TOTAL_BUDGET_USD MAX_CYCLE_NIGHTS REPORT_ROOT MIRROR_DIR GH_ORG GH_TOKEN SLACK_WEBHOOK_URL
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--canary) CANARY=1 ;;
--dry-run) DRY_RUN=1 ;;
--squeeze-dry-run) SQUEEZE=1; DRY_RUN=1 ;;
--once) : ;;
-h|--help) usage; exit 0 ;;
*) die "unknown arg: $1 (see --help)" ;;
esac
shift
done
command -v jq >/dev/null || die "jq is required"
command -v git >/dev/null || die "git is required"
# --- Report dir (mode 600 reports; matches sweep conventions) -----------------
umask 077
UTC_DATE="$(date -u +%Y-%m-%d)"
UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
REPORT_DIR="$REPORT_ROOT/coordinator/$UTC_DATE"
mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true
# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope
SWEEP_LOG="$REPORT_DIR/coordinator.log" # name the substrate's post_slack_alarm() references
REPORT_JSON="$REPORT_DIR/coordinator.json"
REPORT_TXT="$REPORT_DIR/coordinator.txt"
BUDGET_LEDGER="${BUDGET_LEDGER:-$REPORT_ROOT/.budget-ledger.json}"
COORD_STATE="${COORD_STATE:-$REPORT_ROOT/.coordinator-state.json}"
log "=== checker_coordinator $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN squeeze=$SQUEEZE) ==="
# In the squeeze acceptance test, force a budget so small the SECOND role cannot fit.
if [ "$SQUEEZE" -eq 1 ]; then
TOTAL_BUDGET_USD="0.01"
log "SQUEEZE: forcing TOTAL_BUDGET_USD=\$$TOTAL_BUDGET_USD so at least one role must DEFER"
fi
# ==============================================================================
# REGISTRY — Tier-1 checker roles (name | script | est per-run cost USD | cadence-days).
# A simple in-script table, easy to extend in later phases (add doc-drift, aws-posture...).
# Cost is the shared-budget DRAW estimate (these checkers are deterministic/cheap; a future
# agentic-judge role would carry a real Claude cost). Cadence is informational here.
# ==============================================================================
declare -a ROLES=(
"compliance-drift|$CHECKERS_DIR/compliance-drift.sh|0.00|1"
"dependency-cve|$CHECKERS_DIR/dependency-cve.sh|0.00|1"
"doc-drift|$CHECKERS_DIR/doc-drift.sh|0.00|7"
"aws-posture|$CHECKERS_DIR/aws-posture.sh|0.00|7"
"plan-groomer|$CHECKERS_DIR/plan-groomer.sh|0.00|7"
"confluence-doc|$CHECKERS_DIR/confluence-doc.sh|0.00|7"
)
role_field() { echo "$1" | cut -d'|' -f"$2"; }
# In SQUEEZE mode, assign non-zero costs so the shared cap is meaningful: the first role fits,
# the second cannot — proving deferral-not-drop deterministically regardless of real cost.
if [ "$SQUEEZE" -eq 1 ]; then
ROLES=(
"compliance-drift|$CHECKERS_DIR/compliance-drift.sh|0.008|1"
"dependency-cve|$CHECKERS_DIR/dependency-cve.sh|0.008|1"
)
fi
# Operator role-skip (COORDINATOR_SKIP_ROLES="aws-posture,confluence-doc"): remove
# roles whose backing credentials are not provisioned (aws-posture needs IAM Roles
# Anywhere; confluence-doc needs the confluence-bot token). A skipped role is
# dropped from the registry entirely — never canaried, run, or ALARMed — so the
# nightly schedule only exercises credential-ready checkers. Empty/unset = run all.
if [ -n "${COORDINATOR_SKIP_ROLES:-}" ]; then
declare -a _kept=()
for entry in "${ROLES[@]}"; do
_name="$(role_field "$entry" 1)"
case ",${COORDINATOR_SKIP_ROLES}," in
*",${_name},"*) log "SKIP role '$_name' (COORDINATOR_SKIP_ROLES)" ;;
*) _kept+=( "$entry" ) ;;
esac
done
ROLES=( ${_kept[@]+"${_kept[@]}"} )
fi
# ==============================================================================
# DURABLE STATE (design §6.7): atomic write-temp-then-rename + integrity check.
# Integrity = schema_version match + stored content_hash + logical-consistency.
# content_hash is computed over the state WITHOUT its own hash field (canonical jq -S -c).
# ==============================================================================
state_hash() { # state_json_without_hash -> hex
if command -v sha256sum >/dev/null 2>&1; then echo "$1" | jq -S -cj 'del(.content_hash)' | sha256sum | cut -d' ' -f1
elif command -v shasum >/dev/null 2>&1; then echo "$1" | jq -S -cj 'del(.content_hash)' | shasum -a 256 | cut -d' ' -f1
else echo "$1" | jq -S -cj 'del(.content_hash)' | cksum | cut -d' ' -f1; fi
}
atomic_write_state() { # path json
local path="$1" json="$2" h tmp
h="$(state_hash "$json")"
json="$(echo "$json" | jq -c --arg h "$h" '.content_hash=$h')"
tmp="$(mktemp "${path}.XXXXXX")"
printf '%s\n' "$json" > "$tmp"
chmod 600 "$tmp" 2>/dev/null || true
mv -f "$tmp" "$path" # rename is atomic on the same filesystem
}
# Verify integrity; echo "ok" or a reason. schema + hash + logical-consistency.
verify_state() { # path expected_schema -> "ok" | reason
local path="$1" want="$2" json sv stored calc
json="$(cat "$path" 2>/dev/null)" || { echo "unreadable"; return; }
echo "$json" | jq -e 'type=="object"' >/dev/null 2>&1 || { echo "not-json-object"; return; }
sv="$(echo "$json" | jq -r '.schema_version // empty')"
[ "$sv" = "$want" ] || { echo "schema-mismatch(got=${sv:-none} want=$want)"; return; }
stored="$(echo "$json" | jq -r '.content_hash // empty')"
[ -n "$stored" ] || { echo "missing-content-hash"; return; }
calc="$(state_hash "$json")"
[ "$stored" = "$calc" ] || { echo "content-hash-mismatch"; return; }
echo "ok"
}
declare -a STATE_ALARMS=()
# --- Budget ledger: {schema_version, day, spend, content_hash}. New UTC day resets spend. ----
TOTAL_SPEND="0"
load_budget_ledger() {
if [ -f "$BUDGET_LEDGER" ]; then
local v; v="$(verify_state "$BUDGET_LEDGER" "$SCHEMA_VERSION")"
if [ "$v" != "ok" ]; then
STATE_ALARMS+=( "*STATE ALARM*: budget ledger corrupt ($v) — rebuilt for $UTC_DATE (rebuildable; a new UTC day resets spend)." )
log "budget ledger integrity FAIL: $v — rebuilding (park-on-corrupt, design §6.7)"
TOTAL_SPEND="0"
else
local day; day="$(jq -r '.day // empty' "$BUDGET_LEDGER")"
if [ "$day" = "$UTC_DATE" ]; then TOTAL_SPEND="$(jq -r '.spend // 0' "$BUDGET_LEDGER")"
else log "budget ledger from $day — new UTC day, resetting day spend"; TOTAL_SPEND="0"; fi
fi
fi
log "budget: shared cap \$$TOTAL_BUDGET_USD, day spend so far \$$TOTAL_SPEND ($UTC_DATE)"
}
save_budget_ledger() {
atomic_write_state "$BUDGET_LEDGER" \
"$(jq -n --argjson sv "$SCHEMA_VERSION" --arg day "$UTC_DATE" --argjson sp "$TOTAL_SPEND" \
'{schema_version:$sv, day:$day, spend:$sp}')"
}
# --- Coordinator state: {schema_version, cycle_start, last_run:{role:date}, deferred:[], content_hash} ---
declare -A LAST_RUN=(); declare -a DEFERRED=(); CYCLE_START="$UTC_DATE"
load_coord_state() {
if [ -f "$COORD_STATE" ]; then
local v; v="$(verify_state "$COORD_STATE" "$SCHEMA_VERSION")"
if [ "$v" != "ok" ]; then
STATE_ALARMS+=( "*STATE ALARM*: coordinator state corrupt ($v) — rebuilt (rebuildable from report history; rotation restarts)." )
log "coordinator state integrity FAIL: $v — rebuilding (park-on-corrupt, design §6.7)"
return
fi
CYCLE_START="$(jq -r '.cycle_start // empty' "$COORD_STATE")"; [ -n "$CYCLE_START" ] || CYCLE_START="$UTC_DATE"
while IFS=$'\t' read -r role date; do [ -n "$role" ] && LAST_RUN["$role"]="$date"; done \
< <(jq -r '(.last_run // {}) | to_entries[] | "\(.key)\t\(.value)"' "$COORD_STATE")
while IFS= read -r role; do [ -n "$role" ] && DEFERRED+=( "$role" ); done \
< <(jq -r '(.deferred // [])[]' "$COORD_STATE")
fi
}
save_coord_state() {
local lr="{}"
for role in "${!LAST_RUN[@]}"; do
lr="$(echo "$lr" | jq -c --arg k "$role" --arg v "${LAST_RUN[$role]}" '.[$k]=$v')"
done
local df="[]"
if [ "${#DEFERRED[@]}" -gt 0 ]; then df="$(printf '%s\n' "${DEFERRED[@]}" | jq -R . | jq -cs 'unique')"; fi
atomic_write_state "$COORD_STATE" \
"$(jq -n --argjson sv "$SCHEMA_VERSION" --arg cs "$CYCLE_START" --argjson lr "$lr" --argjson df "$df" \
'{schema_version:$sv, cycle_start:$cs, last_run:$lr, deferred:$df}')"
}
load_budget_ledger
load_coord_state
# In the squeeze test, backdate cycle_start + a role's last_run so the COVERAGE ALARM trips
# deterministically (simulate enough elapsed cycles). This proves the COVERAGE path without
# waiting MAX_CYCLE_NIGHTS real days.
if [ "$SQUEEZE" -eq 1 ]; then
OLD_DATE="$(to_epoch "$UTC_DATE")"; OLD_DATE=$(( OLD_DATE - (MAX_CYCLE_NIGHTS + 2) * 86400 ))
# portable epoch -> YYYY-MM-DD
OLD_DATE_STR="$(date -u -d "@$OLD_DATE" +%Y-%m-%d 2>/dev/null || date -u -r "$OLD_DATE" +%Y-%m-%d 2>/dev/null || echo "$UTC_DATE")"
CYCLE_START="$OLD_DATE_STR"
LAST_RUN["dependency-cve"]="$OLD_DATE_STR" # this role has not run in > MAX_CYCLE_NIGHTS
log "SQUEEZE: backdated cycle_start + dependency-cve last_run to $OLD_DATE_STR (> ${MAX_CYCLE_NIGHTS}d) to trip COVERAGE"
fi
# ==============================================================================
# 1) CANARY SUITE FIRST — each role's checker --canary; a miss = COMPLACENCY ALARM + skip.
# ==============================================================================
declare -a ALARM_LINES=(); declare -A CANARY_OK=()
for entry in "${ROLES[@]}"; do
role="$(role_field "$entry" 1)"; script="$(role_field "$entry" 2)"
if [ ! -x "$script" ] && [ ! -f "$script" ]; then
CANARY_OK["$role"]=0
ALARM_LINES+=( "*COMPLACENCY ALARM*: role '$role' checker missing ($script) — skipped." )
continue
fi
set +e
bash "$script" --canary >"$REPORT_DIR/$role.canary.log" 2>&1
rc=$?
set -e
if [ "$rc" -eq 0 ]; then
CANARY_OK["$role"]=1; log "canary PASS: $role"
else
CANARY_OK["$role"]=0
ALARM_LINES+=( "*COMPLACENCY ALARM*: role '$role' canary FAILED (rc=$rc) — skipped this run. See \`$REPORT_DIR/$role.canary.log\`." )
log "canary FAIL: $role (rc=$rc) — will SKIP this role"
fi
done
# --canary mode: assert every role's canary passed, then stop (offline; post nothing).
if [ "$CANARY" -eq 1 ]; then
fail=0
for entry in "${ROLES[@]}"; do
role="$(role_field "$entry" 1)"
[ "${CANARY_OK[$role]:-0}" -eq 1 ] || { echo "[coordinator] CANARY FAIL: role '$role' did not pass" >&2; fail=1; }
done
if [ "$fail" -ne 0 ]; then
echo "[coordinator] CANARY SUITE FAILED — at least one role's canary did not pass." >&2
exit 3
fi
log "canary suite PASS: all ${#ROLES[@]} role(s) green."
exit 0
fi
# ==============================================================================
# 2) FAN-OUT under the SHARED cap. Order: DEFERRED roles first, then by rotation
# (oldest last_run first). A role whose est cost would exceed the ceiling is DEFERRED
# (recorded), never dropped. A degraded (canary-failed) role is skipped.
# ==============================================================================
# Build the run order: deferred-first, then never-run, then oldest-last_run.
order_roles() {
local entry role lr key
for entry in "${ROLES[@]}"; do
role="$(role_field "$entry" 1)"
# is it currently deferred?
if printf '%s\n' ${DEFERRED[@]+"${DEFERRED[@]}"} | grep -qxF "$role"; then
echo "0000000000|$role"; continue
fi
lr="${LAST_RUN[$role]:-}"
if [ -z "$lr" ]; then key="0000000001"; else key="$(to_epoch "$lr")"; fi
echo "$key|$role"
done | sort -n | cut -d'|' -f2
}
declare -a NEW_DEFERRED=(); declare -a RAN_ROLES=()
declare -a RUN_REPORT_JSONS=()
while IFS= read -r role; do
[ -n "$role" ] || continue
# find the registry entry
entry=""; for e in "${ROLES[@]}"; do [ "$(role_field "$e" 1)" = "$role" ] && entry="$e"; done
[ -n "$entry" ] || continue
script="$(role_field "$entry" 2)"; cost="$(role_field "$entry" 3)"
# Skip degraded roles (canary failed) — never run silently degraded.
if [ "${CANARY_OK[$role]:-0}" -ne 1 ]; then
log "skip $role: canary not green (already alarmed)"
continue
fi
# Budget headroom check: would this role's est cost push us over the SHARED ceiling?
projected="$(jq -n --argjson s "$TOTAL_SPEND" --argjson c "$cost" '$s + $c')"
if jq -n --argjson p "$projected" --argjson cap "$TOTAL_BUDGET_USD" -e '$cap > 0 and $p > $cap' >/dev/null 2>&1; then
NEW_DEFERRED+=( "$role" )
log "DEFER $role: est \$$cost would exceed shared cap \$$TOTAL_BUDGET_USD (spend \$$TOTAL_SPEND) — DEFERRED, not dropped"
ALARM_LINES+=( "*$role* DEFERRED: est \$$cost over shared cap \$$TOTAL_BUDGET_USD (day spend \$$TOTAL_SPEND). Will run next eligible night." )
continue
fi
# Run the checker in --dry-run (the coordinator owns routing; checkers must not post).
# In SQUEEZE mode (synthetic acceptance test, may run on a box without $MIRROR_DIR) point the
# checker at its own fixture via --targets so the "ran" role succeeds deterministically; this
# keeps the deferral/COVERAGE proof self-contained. Normal runs use the real mirror set.
log "--- run role: $role (est \$$cost) ---"
set +e
if [ "$SQUEEZE" -eq 1 ]; then
bash "$script" --dry-run --no-api --targets "$CHECKERS_DIR/fixtures/$role/clean-repo" \
>"$REPORT_DIR/$role.run.log" 2>&1
else
bash "$script" --dry-run >"$REPORT_DIR/$role.run.log" 2>&1
fi
rc=$?
set -e
if [ "$rc" -ne 0 ]; then
ALARM_LINES+=( "*$role*: checker run error (rc=$rc). See \`$REPORT_DIR/$role.run.log\`." )
log "$role run error rc=$rc (logged) — NOT collecting its report (avoid stale/partial findings)"
else
# Collect the checker's own report JSON (REPORT_ROOT/<role>/<date>/<role>.json) only on a
# clean run — a failed run could leave a stale report from an earlier (e.g. canary) pass,
# and folding that in would misattribute findings.
src="$REPORT_ROOT/$role/$UTC_DATE/$role.json"
if [ -f "$src" ]; then rj="$REPORT_DIR/$role.json"; cp -f "$src" "$rj"; RUN_REPORT_JSONS+=( "$rj" ); fi
fi
# Account spend, record last_run, drop from deferred.
add_spend "$cost"
LAST_RUN["$role"]="$UTC_DATE"
RAN_ROLES+=( "$role" )
done < <(order_roles)
# New deferral set = roles deferred this run, plus any previously-deferred role we did NOT run.
for role in ${DEFERRED[@]+"${DEFERRED[@]}"}; do
printf '%s\n' ${RAN_ROLES[@]+"${RAN_ROLES[@]}"} | grep -qxF "$role" && continue
printf '%s\n' ${NEW_DEFERRED[@]+"${NEW_DEFERRED[@]}"} | grep -qxF "$role" && continue
NEW_DEFERRED+=( "$role" )
done
DEFERRED=( ${NEW_DEFERRED[@]+"${NEW_DEFERRED[@]}"} )
log "ran: ${RAN_ROLES[*]:-none} | deferred: ${DEFERRED[*]:-none} | day spend \$$TOTAL_SPEND/\$$TOTAL_BUDGET_USD"
# ==============================================================================
# 3) COVERAGE ALARM — any role whose last_run is older than MAX_CYCLE_NIGHTS days
# (or never run and deferred that long) is behind (design §5).
# ==============================================================================
NOW_EPOCH="$(to_epoch "$UTC_DATE")"
for entry in "${ROLES[@]}"; do
role="$(role_field "$entry" 1)"
lr="${LAST_RUN[$role]:-}"
if [ -z "$lr" ]; then ref="$CYCLE_START"; else ref="$lr"; fi
age=$(( ( NOW_EPOCH - $(to_epoch "$ref") ) / 86400 ))
if [ "$age" -ge "$MAX_CYCLE_NIGHTS" ]; then
ALARM_LINES+=( "*COVERAGE ALARM*: role '$role' not run in ${age}d (last=${lr:-never, cycle since $CYCLE_START}, max $MAX_CYCLE_NIGHTS). Deferred=$(printf '%s\n' ${DEFERRED[@]+"${DEFERRED[@]}"} | grep -qxF "$role" && echo yes || echo no). Raise budget or check failures." )
log "COVERAGE ALARM: $role age ${age}d >= $MAX_CYCLE_NIGHTS"
fi
done
# Persist state (atomic + hashed). Even in dry-run we persist so rotation advances; the
# squeeze test runs dry, so guard: in SQUEEZE we do NOT persist (it is a synthetic scenario).
if [ "$SQUEEZE" -eq 0 ]; then
save_budget_ledger
save_coord_state
else
log "SQUEEZE: synthetic scenario — NOT persisting state."
fi
# Fold any state-integrity alarms in.
for x in ${STATE_ALARMS[@]+"${STATE_ALARMS[@]}"}; do ALARM_LINES+=( "$x" ); done
# ==============================================================================
# 4) COLLECT + DEDUP + PRIORITIZE across the run checkers' reports.
# DEDUP rule: same (repo + check + title) OR identical finding id -> one. Sort by severity.
# ==============================================================================
ALL_FINDINGS="[]"
if [ "${#RUN_REPORT_JSONS[@]}" -gt 0 ]; then
ALL_FINDINGS="$(jq -s '
[ .[].findings[]? ]
| unique_by(.id) # identical id -> one
| unique_by([.repo, .check, .title]) # same repo+check+title -> one
| sort_by( {critical:0, high:1, medium:2, low:3, info:4, unverified:5}[.severity] // 6 )
' "${RUN_REPORT_JSONS[@]}" 2>/dev/null || echo '[]')"
fi
N_FIND="$(echo "$ALL_FINDINGS" | jq 'length')"
N_CRITHIGH="$(echo "$ALL_FINDINGS" | jq '[.[]|select(.severity=="critical" or .severity=="high")] | length')"
declare -a CRITHIGH_LINES=()
while IFS= read -r line; do [ -n "$line" ] && CRITHIGH_LINES+=( "$line" ); done < <(
echo "$ALL_FINDINGS" | jq -r '.[] | select(.severity=="critical" or .severity=="high")
| "*\(.repo)* [\(.severity)] \(.title)"')
# ==============================================================================
# 5) ASSEMBLE the combined coordinator report (JSON + text), mode 600.
# ==============================================================================
DEFERRED_JSON="[]"; [ "${#DEFERRED[@]}" -gt 0 ] && DEFERRED_JSON="$(printf '%s\n' "${DEFERRED[@]}" | jq -R . | jq -cs .)"
RAN_JSON="[]"; [ "${#RAN_ROLES[@]}" -gt 0 ] && RAN_JSON="$(printf '%s\n' "${RAN_ROLES[@]}" | jq -R . | jq -cs .)"
ALARMS_JSON="[]"; [ "${#ALARM_LINES[@]}" -gt 0 ] && ALARMS_JSON="$(printf '%s\n' "${ALARM_LINES[@]}" | jq -R . | jq -cs .)"
jq -n \
--arg ts "$UTC_STAMP" --arg org "$GH_ORG" \
--argjson cap "$TOTAL_BUDGET_USD" --argjson spend "$TOTAL_SPEND" \
--argjson ran "$RAN_JSON" --argjson deferred "$DEFERRED_JSON" \
--argjson findings "$ALL_FINDINGS" --argjson alarms "$ALARMS_JSON" \
'{coordinator:"plane1", generated:$ts, org:$org,
shared_budget_usd:$cap, day_spend_usd:$spend,
ran_roles:$ran, deferred_roles:$deferred,
finding_count:($findings|length),
crit_high:([$findings[]|select(.severity=="critical" or .severity=="high")]|length),
findings:$findings, alarms:$alarms}' > "$REPORT_JSON"
{
echo "plane-1 coordinator report — $UTC_STAMP"
echo "org=$GH_ORG shared_cap=\$$TOTAL_BUDGET_USD day_spend=\$$TOTAL_SPEND"
echo "ran: ${RAN_ROLES[*]:-none}"
echo "deferred (NOT dropped): ${DEFERRED[*]:-none}"
echo "findings: $N_FIND ($N_CRITHIGH crit/high)"
echo
echo "$ALL_FINDINGS" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)"'
if [ "${#ALARM_LINES[@]}" -gt 0 ]; then
echo; echo "alarms:"; printf ' - %s\n' "${ALARM_LINES[@]}"
fi
} > "$REPORT_TXT"
chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true
log "report: $REPORT_JSON ($N_FIND finding(s), ${#ALARM_LINES[@]} alarm line(s))"
# ==============================================================================
# 6) ROUTE (ALARM-only, D3): confirmed crit/high OR any alarm line -> Slack ALARM;
# everything else -> the mode-600 report only; a fully clean run posts NOTHING.
# ==============================================================================
ALARM=0
[ "$N_CRITHIGH" -gt 0 ] && ALARM=1
[ "${#ALARM_LINES[@]}" -gt 0 ] && ALARM=1
# Squeeze acceptance: print the proof lines explicitly to stdout.
if [ "$SQUEEZE" -eq 1 ]; then
echo "=== SQUEEZE ACCEPTANCE (Phase-2) ==="
echo "DEFERRED (not dropped): ${DEFERRED[*]:-none}"
printf '%s\n' ${ALARM_LINES[@]+"${ALARM_LINES[@]}"} | grep -E 'COVERAGE ALARM|DEFERRED' || true
echo "===================================="
fi
if [ "$ALARM" -ne 1 ]; then
log "clean run — no crit/high findings, no alarm conditions. Posting NOTHING (ALARM-only policy)."
exit 0
fi
ALARM_BODY=""
[ "${#CRITHIGH_LINES[@]}" -gt 0 ] && ALARM_BODY="$(printf '%s\n' "${CRITHIGH_LINES[@]}" | sed 's/^/• /')"
META_BODY="$(printf '%s\n' ${ALARM_LINES[@]+"${ALARM_LINES[@]}"} | sed 's/^/• /')"
SLACK_TEXT=":satellite_antenna: *Sea Haven Plane-1 coordinator — ALARM* ($UTC_STAMP)
ran: ${RAN_ROLES[*]:-none} · deferred: ${DEFERRED[*]:-none} · spend \$$TOTAL_SPEND/\$$TOTAL_BUDGET_USD
$N_CRITHIGH confirmed crit/high finding(s):
$ALARM_BODY
coordination alarms:
$META_BODY
Combined report (mode 600): \`$REPORT_JSON\` (on R720)"
SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)"
echo "$SLACK_TEXT" >&2
if [ "$DRY_RUN" -eq 1 ]; then
log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 2)."
exit 0
fi
post_slack_alarm "$SLACK_TEXT"
exit 0
# ==============================================================================
# PROVISIONING (NOT DONE HERE — gated, Phase 6):
# - No systemd unit / timer is installed by this script. Wiring it into the live
# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated.
# - This coordinator runs ONLY the Plane-1 Tier-1 checkers (compliance-drift,
# dependency-cve). doc-drift / aws-posture / planner / fixer are later phases.
# - It does NOT re-clone (checkers reuse $MIRROR_DIR); a checker's own --refresh is the
# only network path and is not invoked here.
# - It does NOT touch agent_team/ or agent-team/, and installs no systemd units.
# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations.
# ==============================================================================

474
checkers/aws-posture.sh Executable file
View file

@ -0,0 +1,474 @@
#!/usr/bin/env bash
# aws-posture.sh — Plane-1 / Tier-2 checker for the R720 agent-team.
#
# Design refs: docs/r720-agent-team-design.md D5 / §4 (Tier 2 roster: aws-posture —
# "Idle/anomalous spend (≈$330/mo flagged) + reasoning layer over baseline findings. Auths via
# Roles Anywhere (short-lived leaf certs, auto-rotated by step-ca). Complements existing
# GuardDuty/Security Hub/Config, does not replace them") and §6.3 / §7 Phase 3 ("doc-drift +
# step-ca/Roles Anywhere + aws-posture"). This is a Tier-2 checker built on the Phase-0 shared
# substrate (lib/sweep_substrate.sh); it mirrors compliance-drift.sh / dependency-cve.sh /
# doc-drift.sh conventions VERBATIM so the coordinator (§5) can drive all of them identically.
#
# WHAT IT DOES (read-only):
# Watches the Sea Haven AWS account (328440206208, us-east-1) for IDLE / ANOMALOUS SPEND and
# idle-resource posture:
# - anomalous Cost Explorer deltas (ce get-anomalies above a $ impact threshold)
# - stopped EC2 instances still paying for attached EBS
# - unattached ("available") EBS volumes
# - unassociated Elastic IPs
# - idle NAT gateways (≈0 bytes out over the window)
# - idle load balancers (0 healthy targets)
# - idle RDS instances (0 connections over the window)
# It COMPLEMENTS GuardDuty / Security Hub / Config (design §4) — it is a spend/idle-posture
# watch, NOT a threat detector, and does not replace them.
#
# AUTH / PROVISIONING GATE (design D5 / §6.3 / §7 B3):
# The LIVE read-only AWS calls require credentials vended via IAM Roles Anywhere using a
# short-lived step-ca leaf cert — this is **PROVISIONING-GATED and NOT available yet** (the IAM
# cross-review PASSED 2026-06-18, which unblocked BUILDING this checker, but step-ca + the trust
# anchor + the role are not stood up). See security-review/iam/ for the reviewed artifacts.
# Therefore the checker:
# (a) attempts read-only `aws` CLI calls ONLY when credentials are actually available
# (an STS identity probe succeeds) AND --no-api/--canary were not passed;
# (b) when there are NO credentials, OR --no-api, OR --canary: it SKIPS the live calls and
# NOTES them — it NEVER alarms on missing data (memory feedback_cloudwatch_alarms: no
# false alarms on no-data). This mirrors compliance-drift's API-skip pattern EXACTLY.
#
# REPORTING (matches secrev sweep conventions):
# - Writes a per-run JSON + text report under $REPORT_ROOT/<UTC-date>/, mode 600 (umask 077).
# - Slack ALARM-ONLY: a clean run (no confirmed waste) posts NOTHING (memory
# feedback_cloudwatch_alarms). Secret-shaped values are redacted from the Slack string.
# - Reuses the substrate's redact() + post_slack_alarm() verbatim.
#
# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping):
# redact, post_slack_alarm -> Slack delivery (reads SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG)
# (aws-posture does NOT use discover_repos/mirror_repo — it scans an AWS account, not repos.)
#
# CANARY / DRY-RUN (offline, no network, no aws, no credentials):
# --canary runs the SAME detectors against a fixture of mocked AWS JSON responses
# (checkers/fixtures/aws-posture/) and asserts the known finding count against
# EXPECTED_FINDING_COUNT (exit 3 on mismatch). It makes ZERO `aws` calls and ZERO network
# calls. This is the anti-complacency floor (design §6.4) AND the routing dry-run (§7 Phase 3):
# --canary implies --dry-run + --no-api; with --dry-run the Slack alarm is composed + printed
# but NOT POSTed.
#
# SCOPE / SAFETY:
# Read-only. The reasoning ("Sonnet collectors + judge", design §4) is a LATER enhancement: a
# clearly-marked inert stub hook (maybe_judge) marks the future seam; it does NOTHING offline
# and NOTHING in this phase (the deterministic detectors are the whole checker here). Does NOT
# touch agent_team/ or agent-team/, is NOT wired into systemd, and stands NOTHING up in AWS —
# that is Phase-3/6 provisioning (gated). See the "PROVISIONING (NOT DONE HERE)" note at bottom.
#
# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED.
set -euo pipefail
export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
log() { echo "[aws-posture] $*" >&2; }
die() { echo "[aws-posture] FATAL: $*" >&2; exit 2; }
# --- Shared substrate ---------------------------------------------------------
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SUBSTRATE="$HERE/../lib/sweep_substrate.sh"
[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE"
# shellcheck source=../lib/sweep_substrate.sh
. "$SUBSTRATE"
# --- Config + defaults (env, all optional) ------------------------------------
AWS_ACCOUNT="${AWS_ACCOUNT:-328440206208}"
AWS_REGION="${AWS_REGION:-us-east-1}"
REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/aws-posture}"
# Cost-anomaly $ impact threshold: only anomalies whose TotalImpact >= this are flagged
# (a tiny anomaly is noise, not waste — no false alarm on a sub-threshold blip).
COST_ANOMALY_MIN_IMPACT="${COST_ANOMALY_MIN_IMPACT:-25}"
# A NAT gateway with bytes-out below this over the window is treated as idle.
NAT_IDLE_BYTES_MAX="${NAT_IDLE_BYTES_MAX:-1024}"
# An RDS instance with max connections at/below this over the window is treated as idle.
RDS_IDLE_CONN_MAX="${RDS_IDLE_CONN_MAX:-0}"
DO_API=1 # --no-api: skip ALL live AWS calls (offline). Without creds this is forced.
DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run).
CANARY=0 # --canary: run the detectors against the mocked-AWS fixture + assert count.
TARGETS_OVERRIDE="" # --targets DIR: read mocked-AWS JSON from DIR instead of the live account
# (offline + deterministic; same file shape as the canary fixture).
usage() {
cat >&2 <<EOF
aws-posture.sh — Plane-1 Tier-2 idle/anomalous-spend + idle-resource posture checker (read-only)
--canary run the detectors against the mocked-AWS fixture and assert the known
finding count (implies --dry-run + --no-api; fully offline — no aws, no network)
--dry-run compose the Slack alarm but DO NOT post it (routing dry-run)
--no-api skip ALL live AWS calls (offline). Forced when no credentials are available.
--targets DIR read mocked-AWS JSON responses from DIR instead of the live account
(offline + deterministic; same file shape as the canary fixture)
-h|--help this help
Env: AWS_ACCOUNT AWS_REGION REPORT_ROOT SLACK_WEBHOOK_URL
COST_ANOMALY_MIN_IMPACT NAT_IDLE_BYTES_MAX RDS_IDLE_CONN_MAX
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--canary) CANARY=1; DRY_RUN=1; DO_API=0 ;;
--dry-run) DRY_RUN=1 ;;
--no-api) DO_API=0 ;;
--targets) shift; TARGETS_OVERRIDE="${1:-}" ;;
-h|--help) usage; exit 0 ;;
*) die "unknown arg: $1 (see --help)" ;;
esac
shift
done
command -v jq >/dev/null || die "jq is required"
# --- Report dir (mode 600 reports; matches sweep conventions) -----------------
umask 077
UTC_DATE="$(date -u +%Y-%m-%d)"
UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
REPORT_DIR="$REPORT_ROOT/$UTC_DATE"
mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true
# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope
SWEEP_LOG="$REPORT_DIR/aws-posture.log" # name the substrate's post_slack_alarm() references
REPORT_JSON="$REPORT_DIR/aws-posture.json"
REPORT_TXT="$REPORT_DIR/aws-posture.txt"
log "=== aws-posture $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API account=$AWS_ACCOUNT region=$AWS_REGION) ==="
# ------------------------------------------------------------------------------
# FINDINGS (spirit of finding.schema.json so the coordinator can route like an agentic finding).
# category="other" (idle-spend is not one of the schema's security categories);
# status="confirmed" only for a deterministic idle/anomaly fact derived from a real response.
# A live call that could not be made (no creds / --no-api / transport failure) is a SKIP, never a
# finding (memory feedback_cloudwatch_alarms: no false alarms on missing data).
# ------------------------------------------------------------------------------
declare -a FINDINGS=()
add_finding() { # id title severity check resource proof
local id="$1" title="$2" sev="$3" check="$4" resource="$5" proof="$6"
FINDINGS+=( "$(jq -n \
--arg id "$id" --arg title "$title" --arg sev "$sev" \
--arg check "$check" --arg resource "$resource" --arg proof "$proof" \
'{account:env.AWS_ACCOUNT_FOR_FINDING, id:$id, title:$title, severity:$sev, category:"other",
check:$check, status:"confirmed", proof:{resource:$resource, outcome:$proof}}')" )
}
export AWS_ACCOUNT_FOR_FINDING="$AWS_ACCOUNT"
declare -a SKIPPED_CHECKS=() # (check:reason) live calls skipped on missing data — reported, never alarmed
note_skip() { SKIPPED_CHECKS+=( "$1" ); }
# Inert future seam (design §4 "Sonnet collectors + judge"): in LIVE mode an ambiguous idle
# candidate ("is this RDS truly idle or just low-traffic?") could be escalated to a reasoning
# judge. This phase keeps the deterministic detectors ONLY — the stub does nothing and is never
# reached offline / in canary / dry-run.
maybe_judge() { # candidate_json (no-op stub; Phase-3 intentionally inert)
return 0
}
# ==============================================================================
# DETECTORS — each consumes one AWS JSON response (live or fixture) and emits findings.
# Pure jq parsing; identical logic for the live `aws ... --output json` output and the canary
# fixture, so the canary genuinely exercises the production detectors.
# ==============================================================================
# cost anomalies: ce get-anomalies. Flag anomalies whose Impact.TotalImpact >= threshold.
detect_cost_anomalies() { # json
local json="$1"
while IFS=$'\t' read -r aid svc impact; do
[ -n "$aid" ] || continue
add_finding "cost-anomaly-$aid" \
"Cost anomaly: ${svc} (≈\$${impact} impact)" "high" "cost-anomaly" "$aid" \
"ce get-anomalies TotalImpact \$${impact} >= threshold \$${COST_ANOMALY_MIN_IMPACT} (service: ${svc})"
done < <(echo "$json" | jq -r --argjson thr "$COST_ANOMALY_MIN_IMPACT" '
(.Anomalies // [])[]
| select((.Impact.TotalImpact // 0) >= $thr)
| [.AnomalyId, (.DimensionValue // "unknown"), ((.Impact.TotalImpact // 0)|tostring)]
| @tsv')
}
# stopped EC2 still paying for attached EBS: describe-instances, State.Name=="stopped" with EBS.
detect_stopped_instances() { # json
local json="$1"
while IFS=$'\t' read -r iid itype; do
[ -n "$iid" ] || continue
add_finding "stopped-ec2-$iid" \
"Stopped EC2 instance still incurring EBS cost: $iid ($itype)" "medium" "stopped-instance" "$iid" \
"ec2 describe-instances: State=stopped with attached EBS (storage bills while stopped)"
done < <(echo "$json" | jq -r '
(.Reservations // [])[].Instances[]
| select((.State.Name // "") == "stopped")
| select(((.BlockDeviceMappings // []) | length) > 0)
| [.InstanceId, (.InstanceType // "?")] | @tsv')
}
# unattached EBS: describe-volumes, State=="available".
detect_unattached_volumes() { # json
local json="$1"
while IFS=$'\t' read -r vid size vtype; do
[ -n "$vid" ] || continue
add_finding "unattached-ebs-$vid" \
"Unattached EBS volume billing idle: $vid (${size}GiB $vtype)" "medium" "unattached-volume" "$vid" \
"ec2 describe-volumes: State=available (no attachment) — billed but unused"
done < <(echo "$json" | jq -r '
(.Volumes // [])[]
| select((.State // "") == "available")
| [.VolumeId, ((.Size // 0)|tostring), (.VolumeType // "?")] | @tsv')
}
# unassociated EIP: describe-addresses, no AssociationId/InstanceId.
detect_unassociated_eips() { # json
local json="$1"
while IFS=$'\t' read -r alloc ip; do
[ -n "$alloc" ] || continue
add_finding "unassociated-eip-$alloc" \
"Unassociated Elastic IP (hourly charge): $ip" "low" "unassociated-eip" "$alloc" \
"ec2 describe-addresses: no AssociationId/InstanceId — idle EIPs are billed hourly"
done < <(echo "$json" | jq -r '
(.Addresses // [])[]
| select((.AssociationId // "") == "" and (.InstanceId // "") == "")
| [(.AllocationId // .PublicIp), (.PublicIp // "?")] | @tsv')
}
# idle NAT gateway: describe-nat-gateways, available + bytes-out below threshold.
# Live path injects the CloudWatch-derived bytes-out as _FixtureBytesOutLast14d (same key the
# canary fixture uses) before calling this — keeping detector logic identical online/offline.
detect_idle_nat() { # json
local json="$1"
while IFS=$'\t' read -r nid bytes; do
[ -n "$nid" ] || continue
add_finding "idle-nat-$nid" \
"Idle NAT gateway (≈0 traffic, ~\$32/mo each): $nid" "medium" "idle-nat" "$nid" \
"ec2 describe-nat-gateways: available with ${bytes} bytes out over window (<= ${NAT_IDLE_BYTES_MAX})"
done < <(echo "$json" | jq -r --argjson mx "$NAT_IDLE_BYTES_MAX" '
(.NatGateways // [])[]
| select((.State // "") == "available")
| select((._FixtureBytesOutLast14d // 0) <= $mx)
| [.NatGatewayId, ((._FixtureBytesOutLast14d // 0)|tostring)] | @tsv')
}
# idle ELB: describe-load-balancers, 0 healthy targets.
# Live path injects the per-LB healthy-target count as _FixtureHealthyTargetCount (derived from
# elbv2 describe-target-health) before calling this — same key the canary fixture uses.
detect_idle_elb() { # json
local json="$1"
while IFS=$'\t' read -r name; do
[ -n "$name" ] || continue
add_finding "idle-elb-$name" \
"Idle load balancer (0 healthy targets, ~\$16/mo each): $name" "medium" "idle-elb" "$name" \
"elbv2 describe-load-balancers + describe-target-health: 0 healthy targets"
done < <(echo "$json" | jq -r '
(.LoadBalancers // [])[]
| select((._FixtureHealthyTargetCount // 0) == 0)
| [.LoadBalancerName // .LoadBalancerArn] | @tsv')
}
# idle RDS: describe-db-instances, available + max connections at/below threshold.
# Live path injects DatabaseConnections max as _FixtureMaxConnectionsLast14d (from CloudWatch).
detect_idle_rds() { # json
local json="$1"
while IFS=$'\t' read -r dbid class; do
[ -n "$dbid" ] || continue
add_finding "idle-rds-$dbid" \
"Idle RDS instance (0 connections over window): $dbid ($class)" "high" "idle-rds" "$dbid" \
"rds describe-db-instances: available with 0 connections over window (<= ${RDS_IDLE_CONN_MAX})"
done < <(echo "$json" | jq -r --argjson mx "$RDS_IDLE_CONN_MAX" '
(.DBInstances // [])[]
| select((.DBInstanceStatus // "") == "available")
| select((._FixtureMaxConnectionsLast14d // 1) <= $mx)
| [.DBInstanceIdentifier, (.DBInstanceClass // "?")] | @tsv')
}
# Run every detector over a directory of JSON responses (fixture dir or a collected-live dir).
# Missing files are tolerated (a detector with no input simply contributes nothing — never a skip
# that alarms; a genuinely uncollected live call is recorded as a SKIP by the live collector).
run_detectors_over_dir() { # dir
local dir="$1" f
f="$dir/cost-anomalies.json"; [ -f "$f" ] && detect_cost_anomalies "$(cat "$f")"
f="$dir/describe-instances.json"; [ -f "$f" ] && detect_stopped_instances "$(cat "$f")"
f="$dir/describe-volumes.json"; [ -f "$f" ] && detect_unattached_volumes "$(cat "$f")"
f="$dir/describe-addresses.json"; [ -f "$f" ] && detect_unassociated_eips "$(cat "$f")"
f="$dir/describe-nat-gateways.json";[ -f "$f" ] && detect_idle_nat "$(cat "$f")"
f="$dir/describe-load-balancers.json";[ -f "$f" ] && detect_idle_elb "$(cat "$f")"
f="$dir/describe-db-instances.json";[ -f "$f" ] && detect_idle_rds "$(cat "$f")"
}
# ==============================================================================
# LIVE COLLECTION (read-only AWS, ONLY when credentials are available + not --no-api/--canary).
# Each call is fail-safe: on a transport/permission failure the response is NOT written and the
# call is recorded as a SKIP — never a finding (memory feedback_cloudwatch_alarms).
# The CloudWatch-derived idle metrics (NAT bytes-out, ELB healthy targets, RDS connections) are
# injected into the describe-* JSON under the SAME _Fixture* keys the detectors read, so the live
# and canary code paths are identical.
# ==============================================================================
aws_creds_available() {
command -v aws >/dev/null || return 1
aws sts get-caller-identity --region "$AWS_REGION" >/dev/null 2>>"$REPORT_DIR/aws.log"
}
collect_live() { # out_dir
local out="$1"; mkdir -p "$out"
# NOTE: this live collector is PROVISIONING-GATED and only reached when real Roles Anywhere
# creds exist (aws_creds_available passed). Until step-ca/Roles Anywhere are stood up this path
# is never taken; it is written so the checker is complete + ready, not so it runs today.
_try() { # outfile aws-args...
local of="$1"; shift
if aws "$@" --region "$AWS_REGION" --output json >"$of" 2>>"$REPORT_DIR/aws.log"; then
return 0
else
rm -f "$of"; note_skip "live:$(basename "$of" .json)(aws-call-failed)"; return 1
fi
}
_try "$out/cost-anomalies.json" ce get-anomalies || true
_try "$out/describe-instances.json" ec2 describe-instances || true
_try "$out/describe-volumes.json" ec2 describe-volumes || true
_try "$out/describe-addresses.json" ec2 describe-addresses || true
_try "$out/describe-nat-gateways.json" ec2 describe-nat-gateways || true
_try "$out/describe-load-balancers.json" elbv2 describe-load-balancers || true
_try "$out/describe-db-instances.json" rds describe-db-instances || true
# Idle-metric enrichment (NAT bytes-out / ELB healthy targets / RDS connections from CloudWatch)
# is injected here in the live path under the _Fixture* keys before the detectors run. It is a
# provisioning-time follow-up — until creds exist this collector is unreachable, so the
# enrichment is intentionally a documented seam, not dead code that runs offline.
}
# ==============================================================================
# RESOLVE THE INPUT (fixture / explicit dir / live collection) + DECIDE API MODE
# ==============================================================================
SCAN_DIR=""
SCAN_MODE="none"
if [ "$CANARY" -eq 1 ]; then
FIXTURE_DIR="$HERE/fixtures/aws-posture"
[ -d "$FIXTURE_DIR" ] || die "canary fixture missing: $FIXTURE_DIR"
SCAN_DIR="$FIXTURE_DIR"; SCAN_MODE="canary-fixture"
log "canary: running detectors against mocked-AWS fixtures in $FIXTURE_DIR (no aws, no network)"
elif [ -n "$TARGETS_OVERRIDE" ]; then
d="${TARGETS_OVERRIDE/#\~/$HOME}"
[ -d "$d" ] || die "--targets dir not found: $d"
SCAN_DIR="$d"; SCAN_MODE="explicit-dir"
log "explicit targets dir (offline mocked-AWS JSON): $SCAN_DIR"
elif [ "$DO_API" -eq 1 ] && aws_creds_available; then
COLLECT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/aws-posture-live.XXXXXX")"
trap 'rm -rf "$COLLECT_DIR"' EXIT
log "live: AWS credentials present — collecting read-only responses into $COLLECT_DIR"
collect_live "$COLLECT_DIR"
SCAN_DIR="$COLLECT_DIR"; SCAN_MODE="live-aws"
else
# No creds, or --no-api: SKIP all live calls and note them. NEVER alarm on missing data.
if [ "$DO_API" -eq 1 ]; then
log "live AWS requested but no usable credentials (Roles Anywhere is PROVISIONING-GATED) — skipping all live calls (no false alarms on missing data)"
note_skip "live:all(no-credentials — Roles Anywhere gated; see security-review/iam/)"
else
log "--no-api: skipping all live AWS calls"
note_skip "live:all(--no-api)"
fi
SCAN_MODE="skipped-no-creds"
fi
# ==============================================================================
# RUN DETECTORS
# ==============================================================================
if [ -n "$SCAN_DIR" ]; then
run_detectors_over_dir "$SCAN_DIR"
maybe_judge "" # inert in this phase (future Sonnet-collector/judge seam)
fi
# ==============================================================================
# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to the other checkers)
# ==============================================================================
if [ "${#FINDINGS[@]}" -gt 0 ]; then
FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)"
else
FINDINGS_JSON="[]"
fi
if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then
SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)"
else
SKIPPED_JSON="[]"
fi
N_FIND="$(echo "$FINDINGS_JSON" | jq 'length')"
N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high" or .severity=="critical")] | length')"
jq -n \
--arg checker "aws-posture" --arg ts "$UTC_STAMP" --arg account "$AWS_ACCOUNT" \
--arg region "$AWS_REGION" --arg mode "$SCAN_MODE" \
--argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \
'{checker:$checker, generated:$ts, account:$account, region:$region, scan_mode:$mode,
finding_count:($findings|length),
findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON"
{
echo "aws-posture report — $UTC_STAMP"
echo "account=$AWS_ACCOUNT region=$AWS_REGION scan_mode=$SCAN_MODE"
echo "idle/anomalous-spend findings: $N_FIND ($N_HIGH high/critical)"
echo
echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.check): \(.title)\n proof: \(.proof.outcome)"'
if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then
echo; echo "skipped (missing data — NOT counted as a finding):"
echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"'
fi
} > "$REPORT_TXT"
chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true
log "report: $REPORT_JSON ($N_FIND finding(s), mode=$SCAN_MODE)"
# ==============================================================================
# CANARY ASSERTION (anti-complacency floor, design §6.4)
# ==============================================================================
if [ "$CANARY" -eq 1 ]; then
EXPECT_FILE="$HERE/fixtures/aws-posture/EXPECTED_FINDING_COUNT"
[ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE"
EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")"
log "canary assertion: expected findings=$EXPECTED, got=$N_FIND"
if [ "$N_FIND" -ne "$EXPECTED" ]; then
echo "[aws-posture] CANARY FAIL: planted-finding count mismatch (expected $EXPECTED, got $N_FIND)" >&2
echo " -> a detector regressed (stopped firing) or the fixture changed. See $REPORT_TXT." >&2
exit 3
fi
log "canary PASS: all $EXPECTED planted idle/anomaly findings detected."
fi
# ==============================================================================
# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms)
# ==============================================================================
if [ "$N_FIND" -eq 0 ]; then
log "no idle/anomalous spend detected — posting NOTHING to Slack (ALARM-only policy)."
exit 0
fi
ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r '
group_by(.check)[] | "*\(.[0].check)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')"
SLACK_TEXT=":money_with_wings: *Sea Haven aws-posture — ALARM* ($UTC_STAMP)
$N_FIND idle/anomalous-spend finding(s) in account $AWS_ACCOUNT/$AWS_REGION ($N_HIGH high/critical):
$ALARM_BODY
Scope: idle/anomalous SPEND + idle-resource posture (complements GuardDuty/SecurityHub/Config, mode=$SCAN_MODE)
Report (mode 600): \`$REPORT_JSON\` (on R720)"
SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)"
echo "$SLACK_TEXT" >&2
if [ "$DRY_RUN" -eq 1 ]; then
log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 3)."
exit 0
fi
post_slack_alarm "$SLACK_TEXT"
exit 0
# ==============================================================================
# PROVISIONING (NOT DONE HERE — gated, Phase 3 / Phase 6):
# - The LIVE AWS calls need credentials vended via IAM Roles Anywhere using a short-lived
# step-ca leaf cert. step-ca + the Roles Anywhere trust anchor + the read-only role are NOT
# stood up by this script. The reviewed IAM artifacts live in security-review/iam/ (GPT-4.1
# cross-review PASSED 2026-06-18: APPROVE, no BLOCKs). Provisioning happens only after that
# review is recorded (design §7, B3) and a VM snapshot is taken (feedback_ec2_replacement_snapshot).
# Until then aws_creds_available() returns false and the checker SKIPS all live calls (no
# false alarms on missing data) — only --canary / --targets exercise it offline.
# - No systemd unit / timer is installed by this script. Wiring it into the live secrev schedule
# (weekly cadence, design §4) is provisioning and is gated.
# - This script is NOT registered in checker_coordinator.sh; the coordinator registry is
# integrated centrally (separate change).
# - The LIVE "Sonnet collectors + judge" reasoning layer (design §4) is the only LLM seam; it is
# an inert stub here (maybe_judge) and stays off in canary / dry-run / offline.
# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations for the
# build session, tracked outside this script.
# ==============================================================================

492
checkers/compliance-drift.sh Executable file
View file

@ -0,0 +1,492 @@
#!/usr/bin/env bash
# compliance-drift.sh — Plane-1 / Tier-1 checker for the R720 agent-team.
#
# Design refs: docs/r720-agent-team-design.md §4 (Tier 1 roster: compliance-drift) and
# §7 Phase 1 ("one checker end to end"). This is the FIRST Plane-1 checker built on the
# Phase-0 shared substrate (lib/sweep_substrate.sh) — it proves the substrate generalizes
# beyond the secrev nightly sweep.
#
# WHAT IT DOES (read-only):
# Flags drift from Sea Haven engineering conventions across the org mirrors. It scans the
# SAME shallow clean clones that nightly_sweep.sh already produced in $MIRROR_DIR — it does
# NOT re-clone (mirrors-first; an optional --refresh re-runs discovery+mirror via the shared
# substrate). The checklist is GROUNDED in the engineering-handbook + this repo's README; it
# does not invent rules. See "CHECKLIST" below.
#
# REPORTING (matches secrev sweep conventions):
# - Writes a per-run JSON + text report under $REPORT_ROOT/<UTC-date>/, mode 600 (umask 077).
# - Slack ALARM-ONLY: a clean run (no confirmed drift) posts NOTHING (memory
# feedback_cloudwatch_alarms). Secret-shaped values are redacted from the Slack string.
# - Reuses the substrate's redact() + post_slack_alarm() verbatim.
#
# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping):
# redact, post_slack_alarm -> Slack delivery (reads SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG)
# discover_repos, mirror_repo-> ONLY on --refresh (reads GH_TOKEN, GH_ORG, MIRROR_DIR, REPORT_DIR)
# Default path enumerates EXISTING $MIRROR_DIR/*/.git dirs — zero clones, zero network.
#
# CANARY / DRY-RUN (offline, no network, no token):
# --canary runs the checklist against a planted-drift fixture (checkers/fixtures/compliance-drift/)
# and asserts the known drift count. This is the anti-complacency floor (design §6.4) AND the
# routing dry-run (§7 Phase 1, F4): with --dry-run, the Slack alarm is composed + printed but
# NOT POSTed. Fully offline-smoke-testable.
#
# SCOPE / SAFETY:
# Read-only. Filesystem checks need no network. The branch-protection / Dependabot-alerts /
# repo-settings checks call the GitHub REST API read-only with the same $GH_TOKEN the sweep
# uses (Contents+Metadata read). When GH_TOKEN is unset OR --no-api is passed (the offline
# default for --canary), API-only checks are SKIPPED and noted in the report — they are never
# reported as drift on missing data (memory feedback_cloudwatch_alarms: no false alarms on no-data).
#
# This script does NOT touch agent_team/ or agent-team/, and is NOT wired into systemd — that is
# Phase-6 provisioning (gated). See the "PROVISIONING (NOT DONE HERE)" note at the bottom.
#
# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED.
set -euo pipefail
export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
log() { echo "[compliance-drift] $*" >&2; }
die() { echo "[compliance-drift] FATAL: $*" >&2; exit 2; }
# --- Shared substrate ---------------------------------------------------------
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SUBSTRATE="$HERE/../lib/sweep_substrate.sh"
[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE"
# shellcheck source=../lib/sweep_substrate.sh
. "$SUBSTRATE"
# --- Config + defaults (env, all optional) ------------------------------------
GH_ORG="${GH_ORG:-Sea-Haven-Industries}"
MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}"
REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/compliance-drift}"
# Repos exempt from CodeQL/compliance tooling per github-standards.md ("Exceptions").
# Comma-separated; handbook lists shoc-backend, shoc-frontend-new (SHOC-owned) + docs repos.
COMPLIANCE_EXEMPT="${COMPLIANCE_EXEMPT:-shoc-backend,shoc-frontend-new}"
# Docs-only repos skip CodeQL/CI-deploy expectations (handbook exception); they still need README.
DOCS_ONLY_REPOS="${DOCS_ONLY_REPOS:-engineering-handbook}"
REFRESH=0 # --refresh: re-run discovery+mirror via substrate (network). Default: reuse mirrors.
DO_API=1 # --no-api: skip GitHub-API checks (branch protection / dependabot / settings).
DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run, F4).
CANARY=0 # --canary: run against the planted-drift fixture + assert the known count.
TARGETS_OVERRIDE="" # --targets "p1 p2": scan explicit dirs instead of the mirror set.
usage() {
cat >&2 <<EOF
compliance-drift.sh — Plane-1 Tier-1 conventions-drift checker (read-only)
--canary run against the planted-drift fixture and assert the known drift count
(implies --dry-run + --no-api; fully offline smoke test)
--dry-run compose the Slack alarm but DO NOT post it (routing dry-run)
--no-api skip GitHub-API checks (branch protection, dependabot alerts, repo settings)
--refresh re-discover + re-mirror via the shared substrate before scanning (network)
--targets "a b" scan these explicit repo dirs instead of \$MIRROR_DIR/* (no clone)
-h|--help this help
Env: GH_ORG MIRROR_DIR REPORT_ROOT GH_TOKEN SLACK_WEBHOOK_URL COMPLIANCE_EXEMPT DOCS_ONLY_REPOS
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--canary) CANARY=1; DRY_RUN=1; DO_API=0 ;;
--dry-run) DRY_RUN=1 ;;
--no-api) DO_API=0 ;;
--refresh) REFRESH=1 ;;
--targets) shift; TARGETS_OVERRIDE="${1:-}" ;;
-h|--help) usage; exit 0 ;;
*) die "unknown arg: $1 (see --help)" ;;
esac
shift
done
command -v jq >/dev/null || die "jq is required"
command -v git >/dev/null || die "git is required"
# --- Report dir (mode 600 reports; matches sweep conventions) -----------------
umask 077
UTC_DATE="$(date -u +%Y-%m-%d)"
UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
REPORT_DIR="$REPORT_ROOT/$UTC_DATE"
mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true
# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope
SWEEP_LOG="$REPORT_DIR/compliance-drift.log" # name the substrate's post_slack_alarm() references
REPORT_JSON="$REPORT_DIR/compliance-drift.json"
REPORT_TXT="$REPORT_DIR/compliance-drift.txt"
log "=== compliance-drift $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API refresh=$REFRESH) ==="
# ------------------------------------------------------------------------------
# CHECKLIST (grounded — every item cites a handbook/README rule; nothing invented):
#
# naming-repo repo dir name is kebab-case naming-conventions.md ("kebab-case for everything")
# readme-present README.md exists at repo root github-standards.md / global CLAUDE.md ("Every repo must have a README")
# cicd-present .github/workflows/ci.yaml|ci.yml cicd.md ("Every deployable repo must have a CI/CD pipeline"; ci.yaml)
# dependabot-config .github/dependabot.yml present when github-standards.md ("Every repo with dependencies gets a .github/dependabot.yml")
# dependency manifests exist
# secrets-committed no committed .env with real-looking secrets-and-config.md ("Never commit .env files containing real values")
# values (tracked-in-git, not gitignored)
# --- API-only (need GH_TOKEN; skipped offline / --no-api / --canary) ---
# branch-protection main requires PR, no force-push, github-standards.md ("Branch Protection")
# no deletion
# dependabot-alerts Dependabot alerts + security updates github-standards.md ("Dependabot alerts and security updates enabled")
# enabled
# merge-settings allow_auto_merge + delete_branch_on_ github-standards.md ("enable auto-merge and auto-delete head branch")
# merge enabled
#
# Each emitted finding follows the spirit of finding.schema.json (id/title/severity/category/
# proof/status) so a later phase can route it like an agentic finding. category="other" — this is
# convention drift, not the schema's security categories. status="confirmed" only for deterministic
# filesystem facts and explicit API "false" answers; API checks on missing data are NOT findings.
# ------------------------------------------------------------------------------
# Drift accumulator: one JSON object per finding, appended to a bash array.
declare -a FINDINGS=()
add_finding() { # repo id title severity check proof
local repo="$1" id="$2" title="$3" sev="$4" check="$5" proof="$6"
FINDINGS+=( "$(jq -n \
--arg repo "$repo" --arg id "$id" --arg title "$title" --arg sev "$sev" \
--arg check "$check" --arg proof "$proof" \
'{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other",
check:$check, status:"confirmed", proof:{outcome:$proof}}')" )
}
declare -a SKIPPED_CHECKS=() # (repo:check) checks skipped on missing data — reported, never alarmed
note_skip() { SKIPPED_CHECKS+=( "$1" ); }
in_csv() { # needle csv -> 0 if present
local n="$1" csv="$2"; case ",$csv," in *",$n,"*) return 0 ;; *) return 1 ;; esac
}
# --- kebab-case test (lowercase, digits, single hyphens; no leading/trailing hyphen) ---
is_kebab() { [[ "$1" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; }
# --- Does the repo carry dependency manifests that warrant a dependabot.yml? ----
has_dep_manifests() { # dir
local d="$1"
# Match handbook's ecosystem table: package.json / requirements.txt / *.csproj.
[ -f "$d/package.json" ] && return 0
find "$d" -maxdepth 3 -name requirements.txt -not -path '*/.git/*' -print -quit 2>/dev/null | grep -q . && return 0
find "$d" -maxdepth 3 -name '*.csproj' -not -path '*/.git/*' -print -quit 2>/dev/null | grep -q . && return 0
return 1
}
# ==============================================================================
# FILESYSTEM CHECKS (offline; run on every repo dir)
# ==============================================================================
check_repo_fs() { # repo_name repo_dir
local repo="$1" dir="$2"
local docs_only=0; in_csv "$repo" "$DOCS_ONLY_REPOS" && docs_only=1
local exempt=0; in_csv "$repo" "$COMPLIANCE_EXEMPT" && exempt=1
# naming-repo — repo dir name kebab-case
is_kebab "$repo" || add_finding "$repo" "naming-repo" \
"Repo name '$repo' is not kebab-case" "medium" "naming-repo" \
"naming-conventions.md: kebab-case for everything (repository names)"
# readme-present — every repo, no exceptions
[ -f "$dir/README.md" ] || add_finding "$repo" "readme-missing" \
"No README.md at repo root" "high" "readme-present" \
"global CLAUDE.md / github-standards.md: every repo must have a README"
# cicd-present — ci workflow expected unless docs-only or compliance-exempt
if [ "$docs_only" -eq 0 ] && [ "$exempt" -eq 0 ]; then
if [ ! -f "$dir/.github/workflows/ci.yaml" ] && [ ! -f "$dir/.github/workflows/ci.yml" ]; then
add_finding "$repo" "cicd-missing" \
"No .github/workflows/ci.yaml" "high" "cicd-present" \
"cicd.md: every deployable repo must have a CI/CD pipeline (ci.yaml)"
fi
else
note_skip "$repo:cicd-present(docs-only/exempt)"
fi
# dependabot-config — required only when dependency manifests exist, and not exempt
if [ "$exempt" -eq 0 ] && has_dep_manifests "$dir"; then
[ -f "$dir/.github/dependabot.yml" ] || [ -f "$dir/.github/dependabot.yaml" ] || \
add_finding "$repo" "dependabot-config-missing" \
"Has dependency manifests but no .github/dependabot.yml" "medium" "dependabot-config" \
"github-standards.md: every repo with dependencies gets a .github/dependabot.yml"
fi
# secrets-committed — a .env TRACKED in git (gitignored .env is fine; tracked is the drift)
if [ -d "$dir/.git" ]; then
while IFS= read -r envf; do
[ -n "$envf" ] || continue
# Only flag .env / .env.* that look like they hold real values, not .env.example/.sample/.template.
case "$envf" in *.example|*.sample|*.template|*.dist) continue ;; esac
# Fire only on secret-SHAPED entries: a secret-ish key name, or a long
# (>=20 char) high-entropy value. Benign config (PORT=3000, DEBUG=true)
# is NOT drift, so a tracked config-only .env raises no ALARM
# (feedback_cloudwatch_alarms: no false alarms on non-secret config).
if grep -qiE '(secret|token|key|password|passwd|api[_-]?key|credential|private)[^=]*=[^[:space:]#]+' "$dir/$envf" 2>/dev/null \
|| grep -qE '=[^[:space:]#]{20,}' "$dir/$envf" 2>/dev/null; then
add_finding "$repo" "secrets-committed-$(echo "$envf" | tr '/.' '--')" \
"Tracked env file with values committed: $envf" "high" "secrets-committed" \
"secrets-and-config.md: never commit .env files containing real values"
fi
done < <(git -C "$dir" ls-files -- '*.env' '.env' '.env.*' 2>/dev/null || true)
else
note_skip "$repo:secrets-committed(not-a-git-checkout)"
fi
}
# ==============================================================================
# API CHECKS (read-only GitHub REST; need GH_TOKEN; skipped offline/--no-api/--canary)
# ==============================================================================
gh_api() { # path -> body on stdout, non-zero on transport/HTTP error
curl -fsS \
-H "Authorization: Bearer $GH_TOKEN" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"https://api.github.com/$1" 2>>"$REPORT_DIR/api.log"
}
check_repo_api() { # repo_name
local repo="$1"
local exempt=0; in_csv "$repo" "$COMPLIANCE_EXEMPT" && exempt=1
# repo settings: merge baseline + vulnerability-alerts capability come off the repo object.
local body
if ! body="$(gh_api "repos/$GH_ORG/$repo")" || ! echo "$body" | jq -e 'type=="object" and has("name")' >/dev/null 2>&1; then
note_skip "$repo:api(repo-fetch-failed)"; return
fi
local default_branch; default_branch="$(echo "$body" | jq -r '.default_branch // "main"')"
# merge-settings — auto-merge + delete-branch-on-merge (per-repo, no org default)
if [ "$exempt" -eq 0 ]; then
local am dbm; am="$(echo "$body" | jq -r '.allow_auto_merge')"; dbm="$(echo "$body" | jq -r '.delete_branch_on_merge')"
[ "$am" = "true" ] || add_finding "$repo" "merge-automerge-off" \
"allow_auto_merge disabled" "low" "merge-settings" \
"github-standards.md: enable auto-merge (allow_auto_merge)"
[ "$dbm" = "true" ] || add_finding "$repo" "merge-deletebranch-off" \
"delete_branch_on_merge disabled" "low" "merge-settings" \
"github-standards.md: enable auto-delete head branch on merge (delete_branch_on_merge)"
fi
# dependabot-alerts — vulnerability alerts enabled (204 = enabled, 404 = disabled)
if [ "$exempt" -eq 0 ]; then
local code
# No -f: a 404 (alerts off) is a real HTTP response we must classify, so curl
# must exit 0 and -w must yield a clean "404" (with -f the body-fail path
# corrupts the captured code and a real 404 would be misread as a skip).
code="$(curl -sS -o /dev/null -w '%{http_code}' \
-H "Authorization: Bearer $GH_TOKEN" -H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"https://api.github.com/repos/$GH_ORG/$repo/vulnerability-alerts" 2>>"$REPORT_DIR/api.log" || echo 000)"
case "$code" in
204) : ;; # enabled
404) add_finding "$repo" "dependabot-alerts-off" \
"Dependabot vulnerability alerts disabled" "high" "dependabot-alerts" \
"github-standards.md: Dependabot alerts and security updates enabled on all active repos" ;;
*) note_skip "$repo:dependabot-alerts(http-$code)" ;; # missing data -> no alarm
esac
fi
# branch-protection — main: require PR, no force-push, no deletion.
# Status-code-aware (mirrors dependabot-alerts): 200 -> parse the rules,
# 404 -> no protection rule = real drift, anything else (403/5xx/000 transient
# or transport failure) -> skip with NO alarm (feedback_cloudwatch_alarms: a
# flaky API call must never raise a high-severity false alarm).
local prot_tmp prot_code prot
prot_tmp="$(mktemp)"
prot_code="$(curl -sS -o "$prot_tmp" -w '%{http_code}' \
-H "Authorization: Bearer $GH_TOKEN" -H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"https://api.github.com/repos/$GH_ORG/$repo/branches/$default_branch/protection" \
2>>"$REPORT_DIR/api.log" || echo 000)"
prot="$(cat "$prot_tmp" 2>/dev/null)"; rm -f "$prot_tmp"
case "$prot_code" in
200)
echo "$prot" | jq -e '.required_pull_request_reviews != null' >/dev/null 2>&1 || \
add_finding "$repo" "branchprot-no-pr" \
"main does not require a PR for merge" "high" "branch-protection" \
"github-standards.md: require a PR for merges to main (no direct push)"
echo "$prot" | jq -e '.allow_force_pushes.enabled == false' >/dev/null 2>&1 || \
add_finding "$repo" "branchprot-force-push" \
"main allows force-push" "high" "branch-protection" \
"github-standards.md: no force push to main"
echo "$prot" | jq -e '.allow_deletions.enabled == false' >/dev/null 2>&1 || \
add_finding "$repo" "branchprot-deletion" \
"main allows branch deletion" "high" "branch-protection" \
"github-standards.md: no branch deletion for main"
;;
404)
# 404 from this endpoint = no protection rule at all on the default branch -> that IS drift.
add_finding "$repo" "branchprot-absent" \
"No branch protection on '$default_branch'" "high" "branch-protection" \
"github-standards.md: require a PR for merges to main, no force push, no deletion"
;;
*) note_skip "$repo:branch-protection(http-$prot_code)" ;; # transient/forbidden -> no alarm
esac
}
# ==============================================================================
# TARGET RESOLUTION
# ==============================================================================
declare -a REPO_NAMES=(); declare -A REPO_DIR=()
if [ "$CANARY" -eq 1 ]; then
FIXTURE_ROOT="$HERE/fixtures/compliance-drift"
[ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT"
# Pin the exception lists the fixtures were authored against, so the canary is
# self-contained and deterministic regardless of the operator's env.
DOCS_ONLY_REPOS="docs-repo"
COMPLIANCE_EXEMPT=""
# Fixtures ship their git metadata as `dotgit/` (not `.git/`) so they are committable
# into THIS repo without becoming nested submodules. Materialize them into a temp work
# area — copy each fixture and rename dotgit -> .git — so the tracked-`.env`/ls-files
# checks run against a real git checkout. The temp area is mode 700 and removed on exit.
FIXTURE_WORK="$(mktemp -d "${TMPDIR:-/tmp}/compliance-drift-canary.XXXXXX")"
trap 'rm -rf "$FIXTURE_WORK"' EXIT
log "canary: materializing planted-drift fixtures from $FIXTURE_ROOT into $FIXTURE_WORK"
for d in "$FIXTURE_ROOT"/*/; do
[ -d "$d/dotgit" ] || continue # only fixture repos (skip README.md etc.)
nm="$(basename "$d")"
cp -R "$d" "$FIXTURE_WORK/$nm"
mv "$FIXTURE_WORK/$nm/dotgit" "$FIXTURE_WORK/$nm/.git"
# The planted-secret env file is shipped as `dotenv.fixture` (NOT `.env`): the repo's
# root .gitignore lists `.env`, so a literal `.env` fixture would never be committed and
# the secrets-committed drift would vanish on a fresh clone. Restore it to `.env` in the
# materialized work area (the dotgit/ index already TRACKS `.env`, so ls-files still
# reports it). Same committable-without-side-effects rationale as the `.fixture` suffix the
# dependency-cve fixtures use for their manifests.
[ -f "$FIXTURE_WORK/$nm/dotenv.fixture" ] && mv "$FIXTURE_WORK/$nm/dotenv.fixture" "$FIXTURE_WORK/$nm/.env"
REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$FIXTURE_WORK/$nm"
done
elif [ -n "$TARGETS_OVERRIDE" ]; then
# shellcheck disable=SC2206
arr=( $TARGETS_OVERRIDE )
for p in "${arr[@]}"; do p="${p/#\~/$HOME}"; nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p"; done
log "explicit targets: ${REPO_NAMES[*]}"
else
if [ "$REFRESH" -eq 1 ]; then
[ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN"
command -v curl >/dev/null || die "--refresh needs curl"
mkdir -p "$MIRROR_DIR"
log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)"
DISCOVERED="$REPORT_DIR/discovered.tsv"
if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then
while IFS=$'\t' read -r name url branch; do
[ -n "$name" ] || continue
mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)"
done < "$DISCOVERED"
else
log "discovery failed — falling back to existing mirrors (coverage may be stale)"
fi
fi
# Default + post-refresh: enumerate EXISTING mirrors. No clone here — reuse the sweep's clones.
[ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)"
for d in "$MIRROR_DIR"/*/; do
[ -d "$d/.git" ] || continue
nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}"
done
log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)"
fi
[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to scan"
# Decide whether API checks run: need a token, the API enabled, and not the offline canary.
RUN_API=0
if [ "$DO_API" -eq 1 ] && [ -n "${GH_TOKEN:-}" ] && command -v curl >/dev/null; then RUN_API=1
elif [ "$DO_API" -eq 1 ]; then log "API checks requested but GH_TOKEN/curl unavailable — skipping (no false alarms on missing data)"; fi
# ==============================================================================
# RUN CHECKS
# ==============================================================================
for nm in "${REPO_NAMES[@]}"; do
check_repo_fs "$nm" "${REPO_DIR[$nm]}"
[ "$RUN_API" -eq 1 ] && check_repo_api "$nm"
done
# ==============================================================================
# ASSEMBLE REPORT (JSON + text), mode 600
# ==============================================================================
if [ "${#FINDINGS[@]}" -gt 0 ]; then
FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)"
else
FINDINGS_JSON="[]"
fi
if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then
SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)"
else
SKIPPED_JSON="[]"
fi
N_DRIFT="$(echo "$FINDINGS_JSON" | jq 'length')"
N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high")] | length')"
N_REPOS_DRIFTED="$(echo "$FINDINGS_JSON" | jq '[.[].repo] | unique | length')"
jq -n \
--arg checker "compliance-drift" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \
--argjson api "$RUN_API" --argjson scanned "${#REPO_NAMES[@]}" \
--argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \
'{checker:$checker, generated:$ts, org:$org, api_checks_ran:($api==1),
repos_scanned:$scanned, drift_count:($findings|length),
repos_with_drift:([$findings[].repo]|unique|length),
findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON"
{
echo "compliance-drift report — $UTC_STAMP"
echo "org=$GH_ORG repos_scanned=${#REPO_NAMES[@]} api_checks=$([ "$RUN_API" -eq 1 ] && echo on || echo off)"
echo "drift findings: $N_DRIFT ($N_HIGH high) across $N_REPOS_DRIFTED repo(s)"
echo
echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n rule: \(.proof.outcome)"'
if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then
echo; echo "skipped checks (missing data — NOT counted as drift):"
echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"'
fi
} > "$REPORT_TXT"
chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true
log "report: $REPORT_JSON ($N_DRIFT drift finding(s), $N_REPOS_DRIFTED repo(s))"
# ==============================================================================
# CANARY ASSERTION (anti-complacency floor, design §6.4)
# ==============================================================================
if [ "$CANARY" -eq 1 ]; then
EXPECT_FILE="$HERE/fixtures/compliance-drift/EXPECTED_DRIFT_COUNT"
[ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE"
EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")"
log "canary assertion: expected drift=$EXPECTED, got=$N_DRIFT"
if [ "$N_DRIFT" -ne "$EXPECTED" ]; then
echo "[compliance-drift] CANARY FAIL: planted-drift count mismatch (expected $EXPECTED, got $N_DRIFT)" >&2
echo " -> the checklist regressed (a check stopped firing) or the fixture changed. See $REPORT_TXT." >&2
exit 3
fi
log "canary PASS: all $EXPECTED planted drifts detected."
fi
# ==============================================================================
# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms)
# ==============================================================================
if [ "$N_DRIFT" -eq 0 ]; then
log "no confirmed drift — posting NOTHING to Slack (ALARM-only policy)."
exit 0
fi
ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r '
group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')"
SLACK_TEXT=":triangular_flag_on_post: *Sea Haven compliance-drift — ALARM* ($UTC_STAMP)
$N_DRIFT drift finding(s) across $N_REPOS_DRIFTED repo(s) ($N_HIGH high):
$ALARM_BODY
Checks: naming · README · CI/CD · Dependabot · secrets-placement · branch-protection (api=$([ "$RUN_API" -eq 1 ] && echo on || echo off))
Report (mode 600): \`$REPORT_JSON\` (on R720)"
SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)"
echo "$SLACK_TEXT" >&2
if [ "$DRY_RUN" -eq 1 ]; then
log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 1 / F4)."
exit 0
fi
post_slack_alarm "$SLACK_TEXT"
exit 0
# ==============================================================================
# PROVISIONING (NOT DONE HERE — gated, Phase 6):
# - No systemd unit / timer is installed by this script. Wiring it into the live
# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated.
# - The coordinator (design §5) that runs this alongside other Tier-1 checkers under
# one shared budget + versioned rotation state is Phase 2, not built here.
# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations
# for the build session, tracked outside this script.
# ==============================================================================

542
checkers/confluence-doc.sh Executable file
View file

@ -0,0 +1,542 @@
#!/usr/bin/env bash
# confluence-doc.sh — Plane-1 SCHEDULED documentation gap-detector (RECOMMEND-ONLY).
#
# Design refs: docs/r720-agent-team-design.md §4 (confluence-doc row) and §7 Phase 4.
# Decisions D3 + D6 + D7:
# D3 Report/recommend-only to start; no auto-Notion/Jira writes.
# D6 Confluence writes (the LATER on-demand path) use a dedicated IT-space-scoped
# `confluence-bot` Atlassian service account — PROVISIONING, gated (see footer).
# D7 SCHEDULED mode = read + RECOMMEND only: doc gaps / stale pages / missing runbooks go
# INTO the mode-600 report, NEVER auto-written. The on-demand SSH-invoked WRITE path
# (including Mermaid edits via ~/.claude/scripts/confluence_mermaid.py) is a separate,
# LATER provisioning path and is NOT implemented here.
# This mirrors compliance-drift.sh / dependency-cve.sh conventions VERBATIM so the coordinator
# (§5) drives it identically.
#
# WHAT IT DOES (read-only, RECOMMEND-ONLY):
# Diffs three documentation INPUTS against what Confluence's IT space actually documents, and
# REPORTS the gaps as recommendations (never writes):
# 1. REPO SET — every non-archived org repo (from the same $MIRROR_DIR mirrors the
# sweep already produced; or --targets / a fixture repo list) SHOULD
# have a Confluence page in the IT page-ID map. A repo with no mapped
# page is a "doc gap" recommendation.
# 2. AWS INVENTORY — (optional) a read-only AWS resource inventory JSON (stacks/Lambdas)
# SHOULD each be represented in the AWS Architecture Map / a page.
# A resource absent from the map is a "missing-from-architecture-map"
# recommendation. Absent inventory file => that whole check is SKIPPED
# (noted, never a gap on missing data).
# 3. PAGE-ID MAP — required runbook/standing pages (Incident Response Runbooks, Backup &
# DR, IAM & Access) SHOULD exist in the map. A required page missing
# from the map is a "missing-runbook" recommendation. Optionally, the
# LIVE Confluence API confirms each mapped page still exists and is not
# stale (lastUpdated older than $STALE_DAYS).
#
# The page-ID map is the canonical one from memory project_confluence_migration (IT space
# 720900). It is supplied as a JSON file (--page-map / $PAGE_MAP_FILE); the canary ships a
# mock map. We do NOT hardcode the live IDs into this script — they live in the map file so
# the map can evolve without a code change.
#
# CONFLUENCE API (LIVE reads need the confluence-bot token — PROVISIONING):
# The staleness / page-existence checks call the Confluence Cloud REST API read-only using
# CONFLUENCE_BASE_URL + CONFLUENCE_EMAIL + CONFLUENCE_API_TOKEN (the confluence-bot creds,
# D6). When those are ABSENT, OR --no-api / --canary is passed, the API checks are SKIPPED
# and NOTED — they are NEVER reported as a gap on missing data (memory
# feedback_cloudwatch_alarms: no false alarms on no-data). This mirrors compliance-drift's
# GitHub-API-skip pattern EXACTLY (status-code-aware: 200 -> parse, 404 -> a real "page gone"
# gap, anything else -> skip with NO alarm). The token / service account is gated provisioning.
#
# ON-DEMAND WRITE PATH (NOT HERE — provisioning): an actual Confluence update, including Mermaid
# architecture-map edits, goes through ~/.claude/scripts/confluence_mermaid.py (ADF-only,
# dry-run-default, macro-count + revert-diff guarded — it has destroyed page 1540098 before via
# a full-body markdown round-trip, so ADF-only is load-bearing). That --apply / live-dry-run is
# the LATER on-demand path and is gated. See the PROVISIONING footer.
#
# CANARY / DRY-RUN (offline, no network, no token):
# --canary runs against a fixture (checkers/fixtures/confluence-doc/): a repo list, a MOCK
# page-ID map, and a MOCK "confluence inventory" JSON (what the API would have returned). It
# asserts the known gap count against EXPECTED_GAP_COUNT (exit 3 on mismatch). --canary implies
# --dry-run + --no-api, so it is fully offline + deterministic. This is the anti-complacency
# floor (design §6.4) AND the routing dry-run.
#
# SCOPE / SAFETY:
# Read-only + RECOMMEND-only. Never writes Confluence, never creates a service account, never
# calls the Mermaid --apply path. Not wired into systemd. See PROVISIONING footer.
#
# Exit: 0 = ran (whether or not it found gaps); 2 = setup/usage error; 3 = canary assertion FAILED.
set -euo pipefail
export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
log() { echo "[confluence-doc] $*" >&2; }
die() { echo "[confluence-doc] FATAL: $*" >&2; exit 2; }
# --- Shared substrate ---------------------------------------------------------
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SUBSTRATE="$HERE/../lib/sweep_substrate.sh"
[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE"
# shellcheck source=../lib/sweep_substrate.sh
. "$SUBSTRATE"
# --- Config + defaults (env, all optional) ------------------------------------
GH_ORG="${GH_ORG:-Sea-Haven-Industries}"
MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}"
REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/confluence-doc}"
# Canonical IT page-ID map (memory project_confluence_migration). JSON, NOT hardcoded here.
PAGE_MAP_FILE="${PAGE_MAP_FILE:-}"
# Optional read-only AWS inventory JSON (stacks/Lambdas) — absent => that check is SKIPPED.
AWS_INVENTORY_FILE="${AWS_INVENTORY_FILE:-}"
# Confluence Cloud REST (the confluence-bot creds, D6) — absent => API checks SKIPPED.
# TWO auth modes are supported; OAuth takes precedence when its creds are present:
# (A) OAuth 2.0 client-credentials (2LO) for an org SERVICE ACCOUNT (preferred for a
# headless bot — Atlassian org service accounts have no classic API token):
# POST https://auth.atlassian.com/oauth/token (client_id+client_secret+
# grant_type=client_credentials) -> 60-min Bearer token, then call
# https://api.atlassian.com/ex/confluence/<cloudId>/wiki/api/v2/...
# (B) Basic auth (account email + API token) against the site /wiki/api/v2/...
CONFLUENCE_BASE_URL="${CONFLUENCE_BASE_URL:-}"
CONFLUENCE_EMAIL="${CONFLUENCE_EMAIL:-}"
CONFLUENCE_API_TOKEN="${CONFLUENCE_API_TOKEN:-}"
CONFLUENCE_OAUTH_CLIENT_ID="${CONFLUENCE_OAUTH_CLIENT_ID:-}"
CONFLUENCE_OAUTH_CLIENT_SECRET="${CONFLUENCE_OAUTH_CLIENT_SECRET:-}"
# Optional: the site cloudId. If empty under OAuth, it is auto-resolved from the
# site's public /_edge/tenant_info (no auth needed).
CONFLUENCE_CLOUD_ID="${CONFLUENCE_CLOUD_ID:-}"
# Atlassian OAuth token endpoint (overridable only for testing).
CONFLUENCE_OAUTH_TOKEN_URL="${CONFLUENCE_OAUTH_TOKEN_URL:-https://auth.atlassian.com/oauth/token}"
# A mapped page is "stale" if its lastUpdated is older than this many days (API check only).
STALE_DAYS="${STALE_DAYS:-180}"
# Repos exempt from needing their own IT page (mirrors compliance-drift's exemption style).
DOC_EXEMPT_REPOS="${DOC_EXEMPT_REPOS:-engineering-handbook}"
# Required standing/runbook pages every IT space must document (page-map keys).
REQUIRED_PAGES="${REQUIRED_PAGES:-Incident Response Runbooks,Backup & Disaster Recovery,IAM & Access Management}"
REFRESH=0 # --refresh: re-discover + re-mirror via substrate (network). Default: reuse mirrors.
DO_API=1 # --no-api: skip the LIVE Confluence API checks (offline).
DRY_RUN=0 # --dry-run: compose any digest but DO NOT post/write (recommend-only).
CANARY=0 # --canary: run against the fixture + assert the known gap count.
TARGETS_OVERRIDE="" # --targets "p1 p2": use these repo names instead of the mirror set.
usage() {
cat >&2 <<EOF
confluence-doc.sh — Plane-1 scheduled doc-gap detector (read-only, RECOMMEND-only per D7)
--canary run against the fixture (mock page-map + mock inventory) and assert
the known gap count (implies --dry-run + --no-api; fully offline)
--dry-run compose recommendations but DO NOT post/write (recommend-only)
--no-api skip the LIVE Confluence API checks (page-existence + staleness)
--page-map PATH the IT page-ID map JSON (project_confluence_migration)
--aws-inventory PATH a read-only AWS inventory JSON (stacks/Lambdas); absent => check SKIPPED
--refresh re-discover + re-mirror via the shared substrate before scanning (network)
--targets "a b" use these repo names instead of \$MIRROR_DIR/* (no clone)
-h|--help this help
Env: GH_ORG MIRROR_DIR REPORT_ROOT PAGE_MAP_FILE AWS_INVENTORY_FILE STALE_DAYS
CONFLUENCE_BASE_URL CONFLUENCE_EMAIL CONFLUENCE_API_TOKEN (confluence-bot, D6)
DOC_EXEMPT_REPOS REQUIRED_PAGES SLACK_WEBHOOK_URL
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--canary) CANARY=1; DRY_RUN=1; DO_API=0 ;;
--dry-run) DRY_RUN=1 ;;
--no-api) DO_API=0 ;;
--page-map) shift; PAGE_MAP_FILE="${1:-}" ;;
--aws-inventory) shift; AWS_INVENTORY_FILE="${1:-}" ;;
--refresh) REFRESH=1 ;;
--targets) shift; TARGETS_OVERRIDE="${1:-}" ;;
-h|--help) usage; exit 0 ;;
*) die "unknown arg: $1 (see --help)" ;;
esac
shift
done
command -v jq >/dev/null || die "jq is required"
command -v git >/dev/null || die "git is required"
# --- Report dir (mode 600 reports; matches sweep conventions) -----------------
umask 077
UTC_DATE="$(date -u +%Y-%m-%d)"
UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
REPORT_DIR="$REPORT_ROOT/$UTC_DATE"
mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true
# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope
SWEEP_LOG="$REPORT_DIR/confluence-doc.log"
REPORT_JSON="$REPORT_DIR/confluence-doc.json"
REPORT_TXT="$REPORT_DIR/confluence-doc.txt"
log "=== confluence-doc $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API refresh=$REFRESH) ==="
# ------------------------------------------------------------------------------
# GAPS (spirit of finding.schema.json so the coordinator + plan-groomer can consume them like
# any finding). category="other" (a doc gap is not a security category). status="confirmed"
# only for deterministic facts: a repo absent from the supplied map, an AWS resource absent
# from the supplied inventory-vs-map diff, a required page missing from the map, or an explicit
# API 404 (mapped page gone). A SKIPPED API check is NEVER a gap (feedback_cloudwatch_alarms).
# ------------------------------------------------------------------------------
declare -a GAPS=()
add_gap() { # subject id title severity check proof
local subject="$1" id="$2" title="$3" sev="$4" check="$5" proof="$6"
GAPS+=( "$(jq -n \
--arg repo "$subject" --arg id "$id" --arg title "$title" --arg sev "$sev" \
--arg check "$check" --arg proof "$proof" \
'{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other",
check:$check, status:"confirmed", recommendation:$proof}')" )
}
declare -a SKIPPED_CHECKS=() # (subject:reason) checks skipped on missing data — never a gap
note_skip() { SKIPPED_CHECKS+=( "$1" ); }
in_csv() { # needle csv -> 0 if present
local n="$1" csv="$2"; case ",$csv," in *",$n,"*) return 0 ;; *) return 1 ;; esac
}
# --- Page-map lookup: is there a page whose key (page title) matches NAME? -----
# The map is a JSON object {"<page title>": <page-id>, ...} (the canary mock + the real
# project_confluence_migration export share this shape). A repo "documented" if a page title
# contains the repo name (case-insensitive), since IT pages are titled e.g. "Payments Dashboard"
# for repo "payments-dashboard".
map_has_page_for_repo() { # repo
local repo="$1"
# normalize repo (kebab) -> a loose token to match against page titles
local needle; needle="$(echo "$repo" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]')"
jq -e --arg n "$needle" '
(keys // [])[] | (ascii_downcase | gsub("[^a-z0-9]";"")) | select(contains($n))
' "$PAGE_MAP_FILE" >/dev/null 2>&1
}
map_has_exact_key() { # exact page title
local key="$1"
jq -e --arg k "$key" 'has($k)' "$PAGE_MAP_FILE" >/dev/null 2>&1
}
# ==============================================================================
# CONFLUENCE API (LIVE reads; need the confluence-bot creds; skipped offline/--no-api/--canary)
# ==============================================================================
# Confluence auth seam: OAuth 2.0 client-credentials (org service account, 2LO) OR
# Basic auth (email + API token). conf_api_init() resolves ONE mode (fetching a
# 60-min Bearer + the cloudId for OAuth); conf_get() does the authenticated GET
# with the right base + header. OAuth wins when its creds are present. Any
# failure (no cloudId, token request fails) returns non-zero so the caller SKIPS
# the live checks — never a false alarm on missing data.
# ==============================================================================
_CONF_MODE=""; _CONF_BASE=""; _CONF_BEARER=""
conf_api_init() {
if [ -n "$CONFLUENCE_OAUTH_CLIENT_ID" ] && [ -n "$CONFLUENCE_OAUTH_CLIENT_SECRET" ]; then
# 2LO client-credentials token FIRST (the secret goes in the request BODY via
# --data-urlencode and is never echoed/logged — matches the existing -u risk class).
local tok
tok="$(curl -sS -X POST "$CONFLUENCE_OAUTH_TOKEN_URL" \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode "client_id=$CONFLUENCE_OAUTH_CLIENT_ID" \
--data-urlencode "client_secret=$CONFLUENCE_OAUTH_CLIENT_SECRET" \
--data-urlencode 'grant_type=client_credentials' \
2>>"$REPORT_DIR/confluence-api.log" | jq -r '.access_token // empty' 2>/dev/null)"
[ -n "$tok" ] || { log "OAuth: token request failed — skipping API (no false alarm)"; return 1; }
# Resolve the cloudId: use CONFLUENCE_CLOUD_ID if given, else the OAuth-native
# accessible-resources endpoint (the public /_edge/tenant_info is not reliable).
# Prefer the resource whose url matches the configured site; else the first.
local cid="$CONFLUENCE_CLOUD_ID"
if [ -z "$cid" ]; then
cid="$(curl -sS -H "Authorization: Bearer $tok" -H 'Accept: application/json' \
'https://api.atlassian.com/oauth/token/accessible-resources' \
2>>"$REPORT_DIR/confluence-api.log" \
| jq -r --arg url "$CONFLUENCE_BASE_URL" \
'(map(select(.url==$url)) | .[0].id) // .[0].id // empty' 2>/dev/null)"
fi
[ -n "$cid" ] || { log "OAuth: could not resolve cloudId (set CONFLUENCE_CLOUD_ID) — skipping API"; return 1; }
_CONF_MODE="oauth"; _CONF_BEARER="$tok"
_CONF_BASE="https://api.atlassian.com/ex/confluence/$cid"
return 0
fi
if [ -n "$CONFLUENCE_BASE_URL" ] && [ -n "$CONFLUENCE_EMAIL" ] \
&& [ -n "$CONFLUENCE_API_TOKEN" ]; then
_CONF_MODE="basic"; _CONF_BASE="$CONFLUENCE_BASE_URL"
return 0
fi
return 1
}
conf_get() { # path_suffix outfile -> echoes http_code (both modes share /wiki/api/v2/...)
local path="$1" out="$2"
if [ "$_CONF_MODE" = "oauth" ]; then
curl -sS -o "$out" -w '%{http_code}' \
-H "Authorization: Bearer $_CONF_BEARER" -H 'Accept: application/json' \
"$_CONF_BASE$path" 2>>"$REPORT_DIR/confluence-api.log" || echo 000
else
curl -sS -o "$out" -w '%{http_code}' \
-u "$CONFLUENCE_EMAIL:$CONFLUENCE_API_TOKEN" -H 'Accept: application/json' \
"$_CONF_BASE$path" 2>>"$REPORT_DIR/confluence-api.log" || echo 000
fi
}
# ==============================================================================
# Confirm a mapped page still exists and is not stale. Status-code-aware, mirroring
# compliance-drift's branch-protection pattern exactly:
# 200 -> parse lastUpdated, flag if older than STALE_DAYS
# 404 -> a mapped page that is GONE -> that IS a confirmed gap
# anything else (401/403/5xx/000 transient) -> SKIP with NO gap (no false alarm on no-data)
conf_check_page() { # page_title page_id
local title="$1" pid="$2"
local tmp code body
tmp="$(mktemp)"
code="$(conf_get "/wiki/api/v2/pages/$pid?body-format=storage" "$tmp")"
body="$(cat "$tmp" 2>/dev/null)"; rm -f "$tmp"
case "$code" in
200)
local updated upd_epoch now_epoch age_days
updated="$(echo "$body" | jq -r '.version.createdAt // .createdAt // empty' 2>/dev/null)"
[ -n "$updated" ] || { note_skip "$title:staleness(no-timestamp)"; return; }
upd_epoch="$(to_epoch "${updated%%T*}")"; now_epoch="$(date -u +%s)"
[ "$upd_epoch" -gt 0 ] || { note_skip "$title:staleness(unparseable-date)"; return; }
age_days=$(( (now_epoch - upd_epoch) / 86400 ))
if [ "$age_days" -gt "$STALE_DAYS" ]; then
add_gap "$title" "stale-page" \
"Page '$title' is stale (last updated ${age_days}d ago, > ${STALE_DAYS}d)" "low" "stale-page" \
"review + refresh the IT page; docs must track the system (global CLAUDE.md docs obligation)"
fi
;;
404)
add_gap "$title" "page-gone" \
"Mapped page '$title' (id $pid) returns 404 — page deleted/moved" "high" "page-existence" \
"the page-ID map points at a non-existent page; fix the map or restore the page"
;;
*) note_skip "$title:api(http-$code)" ;; # transient/forbidden -> NO gap on missing data
esac
}
# ==============================================================================
# TARGET RESOLUTION (repo set + map + inventory)
# ==============================================================================
declare -a REPO_NAMES=()
if [ "$CANARY" -eq 1 ]; then
FIXTURE_ROOT="$HERE/fixtures/confluence-doc"
[ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT"
PAGE_MAP_FILE="$FIXTURE_ROOT/mock-page-map.json"
AWS_INVENTORY_FILE="$FIXTURE_ROOT/mock-aws-inventory.json"
[ -f "$PAGE_MAP_FILE" ] || die "canary mock page-map missing: $PAGE_MAP_FILE"
[ -f "$AWS_INVENTORY_FILE" ] || die "canary mock aws inventory missing: $AWS_INVENTORY_FILE"
# Pin the exception + required-page lists the fixture was authored against (deterministic).
DOC_EXEMPT_REPOS="engineering-handbook"
REQUIRED_PAGES="Incident Response Runbooks,Backup & Disaster Recovery,IAM & Access Management"
STALE_DAYS="180"
# The fixture repo set is a newline-delimited list (no git checkout needed — confluence-doc
# diffs NAMES against the map, it does not scan repo contents).
while IFS= read -r r; do
r="$(echo "$r" | tr -d '[:space:]')"; [ -n "$r" ] && REPO_NAMES+=( "$r" )
done < "$FIXTURE_ROOT/repos.txt"
log "canary: ${#REPO_NAMES[@]} fixture repo(s); mock map + mock inventory"
elif [ -n "$TARGETS_OVERRIDE" ]; then
# shellcheck disable=SC2206 # intentional word-split of the space-separated --targets list
arr=( $TARGETS_OVERRIDE )
for p in "${arr[@]}"; do nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); done
log "explicit targets: ${REPO_NAMES[*]}"
else
if [ "$REFRESH" -eq 1 ]; then
[ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN"
command -v curl >/dev/null || die "--refresh needs curl"
mkdir -p "$MIRROR_DIR"
log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)"
DISCOVERED="$REPORT_DIR/discovered.tsv"
if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then
while IFS=$'\t' read -r name url branch; do
[ -n "$name" ] || continue
mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)"
done < "$DISCOVERED"
else
log "discovery failed — falling back to existing mirrors (coverage may be stale)"
fi
fi
[ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)"
for d in "$MIRROR_DIR"/*/; do
[ -d "$d/.git" ] || continue
nm="$(basename "$d")"; REPO_NAMES+=( "$nm" )
done
log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)"
fi
[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to diff"
[ -n "$PAGE_MAP_FILE" ] || die "no page-ID map (--page-map PATH or \$PAGE_MAP_FILE); cannot diff repos vs Confluence"
[ -f "$PAGE_MAP_FILE" ] || die "page-ID map not found: $PAGE_MAP_FILE"
jq -e 'type=="object"' "$PAGE_MAP_FILE" >/dev/null 2>&1 || die "page-ID map is not a JSON object: $PAGE_MAP_FILE"
# Decide whether the LIVE Confluence API runs: need curl, API enabled, not offline
# canary, AND an auth mode that initializes (OAuth service account or Basic). A
# token/cloudId failure leaves RUN_API=0 → checks skipped, NO false alarm.
RUN_API=0
if [ "$DO_API" -eq 1 ] && command -v curl >/dev/null && conf_api_init; then
RUN_API=1
log "Confluence API: ${_CONF_MODE} auth ready"
elif [ "$DO_API" -eq 1 ]; then
log "Confluence API requested but confluence-bot creds/curl unavailable — skipping live checks (no false alarms on missing data; the service account is gated provisioning)."
fi
# ==============================================================================
# CHECK 1 — REPO SET vs page-ID map (every non-exempt repo SHOULD have an IT page)
# ==============================================================================
for nm in "${REPO_NAMES[@]}"; do
in_csv "$nm" "$DOC_EXEMPT_REPOS" && { note_skip "$nm:repo-page(doc-exempt)"; continue; }
if ! map_has_page_for_repo "$nm"; then
add_gap "$nm" "no-it-page" \
"Repo '$nm' has no Confluence IT page in the page-ID map" "medium" "repo-documented" \
"create an IT page for '$nm' (sh-confluence) and add it to project_confluence_migration"
fi
done
# ==============================================================================
# CHECK 2 — AWS INVENTORY vs page-ID map (optional; absent file => SKIP, never a gap)
# ==============================================================================
if [ -n "$AWS_INVENTORY_FILE" ] && [ -f "$AWS_INVENTORY_FILE" ]; then
if jq -e '.resources | type=="array"' "$AWS_INVENTORY_FILE" >/dev/null 2>&1; then
# Each resource SHOULD be represented on a page in the map (by name token match).
while IFS= read -r res; do
[ -n "$res" ] || continue
rname="$(echo "$res" | jq -r '.name // empty')"
rtype="$(echo "$res" | jq -r '.type // "resource"')"
[ -n "$rname" ] || continue
needle="$(echo "$rname" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]')"
if ! jq -e --arg n "$needle" '
(keys // [])[] | (ascii_downcase | gsub("[^a-z0-9]";"")) | select(contains($n))
' "$PAGE_MAP_FILE" >/dev/null 2>&1; then
add_gap "$rname" "aws-not-in-map" \
"AWS $rtype '$rname' is not represented in the IT page-ID map / architecture map" "medium" "aws-documented" \
"add '$rname' to the AWS Architecture Map (page 1540098) + an IT page; Mermaid edits via confluence_mermaid.py (on-demand path, provisioning)"
fi
done < <(jq -c '.resources[]' "$AWS_INVENTORY_FILE")
else
note_skip "aws-inventory:malformed(no-resources-array)"
fi
else
note_skip "aws-inventory:absent(check-skipped)" # missing inventory -> SKIP, never a gap
fi
# ==============================================================================
# CHECK 3 — REQUIRED standing/runbook pages present in the map
# ==============================================================================
IFS=',' read -r -a req_arr <<< "$REQUIRED_PAGES"
for page in "${req_arr[@]}"; do
page="$(echo "$page" | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//')"
[ -n "$page" ] || continue
if ! map_has_exact_key "$page"; then
add_gap "$page" "missing-runbook" \
"Required page '$page' is missing from the IT page-ID map" "high" "required-page" \
"create the '$page' page in the IT space and add it to project_confluence_migration"
fi
done
# ==============================================================================
# CHECK 4 — LIVE API: mapped pages still exist + are not stale (skipped offline/--no-api/--canary)
# ==============================================================================
if [ "$RUN_API" -eq 1 ]; then
while IFS=$'\t' read -r ptitle pid; do
[ -n "$pid" ] || continue
case "$pid" in ''|*[!0-9]*) note_skip "$ptitle:api(non-numeric-id)"; continue ;; esac
conf_check_page "$ptitle" "$pid"
done < <(jq -r 'to_entries[] | [.key, (.value|tostring)] | @tsv' "$PAGE_MAP_FILE")
else
note_skip "confluence-api:not-run(creds-absent-or-offline)"
fi
# ==============================================================================
# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to the other checkers)
# ==============================================================================
if [ "${#GAPS[@]}" -gt 0 ]; then
GAPS_JSON="$(printf '%s\n' "${GAPS[@]}" | jq -cs .)"
else
GAPS_JSON="[]"
fi
if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then
SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)"
else
SKIPPED_JSON="[]"
fi
N_GAPS="$(echo "$GAPS_JSON" | jq 'length')"
N_HIGH="$(echo "$GAPS_JSON" | jq '[.[]|select(.severity=="high" or .severity=="critical")] | length')"
N_SUBJECTS="$(echo "$GAPS_JSON" | jq '[.[].repo] | unique | length')"
jq -n \
--arg checker "confluence-doc" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \
--argjson api "$RUN_API" --argjson reposn "${#REPO_NAMES[@]}" \
--argjson gaps "$GAPS_JSON" --argjson skipped "$SKIPPED_JSON" \
'{checker:$checker, generated:$ts, org:$org, mode:"recommend-only",
api_checks_ran:($api==1), repos_diffed:$reposn,
gap_count:($gaps|length),
subjects_with_gaps:([$gaps[].repo]|unique|length),
findings:$gaps, skipped_checks:$skipped}' > "$REPORT_JSON"
{
echo "confluence-doc — documentation gap report — $UTC_STAMP"
echo "org=$GH_ORG repos_diffed=${#REPO_NAMES[@]} api_checks=$([ "$RUN_API" -eq 1 ] && echo on || echo off) mode=recommend-only (D7)"
echo "doc gaps: $N_GAPS ($N_HIGH high) across $N_SUBJECTS subject(s)"
echo
if [ "$N_GAPS" -gt 0 ]; then
echo "RECOMMENDATIONS (recommend-only — NEVER auto-written, D7):"
echo "$GAPS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n recommend: \(.recommendation)"'
else
echo "No documentation gaps detected this run."
fi
if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then
echo; echo "skipped checks (missing data — NOT counted as a gap):"
echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"'
fi
} > "$REPORT_TXT"
chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true
log "report: $REPORT_JSON ($N_GAPS gap(s), $N_SUBJECTS subject(s))"
# ==============================================================================
# CANARY ASSERTION (anti-complacency floor, design §6.4)
# ==============================================================================
if [ "$CANARY" -eq 1 ]; then
EXPECT_FILE="$HERE/fixtures/confluence-doc/EXPECTED_GAP_COUNT"
[ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE"
EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")"
log "canary assertion: expected gaps=$EXPECTED, got=$N_GAPS"
if [ "$N_GAPS" -ne "$EXPECTED" ]; then
echo "[confluence-doc] CANARY FAIL: doc-gap count mismatch (expected $EXPECTED, got $N_GAPS)" >&2
echo " -> a gap check regressed (stopped firing) or the fixture changed. See $REPORT_TXT." >&2
exit 3
fi
log "canary PASS: all $EXPECTED planted doc gaps detected."
fi
# ==============================================================================
# RECOMMEND-ONLY ROUTING (D3/D7): gaps live in the mode-600 report. Post NOTHING by default.
# Scheduled mode NEVER auto-writes Confluence; alarming is reserved for confirmed criticals via
# the coordinator's shared routing (kept ALARM-only there). Here, recommend-only = report-only.
# ==============================================================================
if [ "$N_GAPS" -eq 0 ]; then
log "no doc gaps — recommend-only report written; posting NOTHING (D7)."
exit 0
fi
# Compose a redacted digest for the report/log (defense-in-depth); do NOT post by default.
DIGEST="$(echo "$GAPS_JSON" | jq -r '
group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' \
| sed 's/^/• /' | redact)"
echo "$DIGEST" >&2
log "DRY-RUN/RECOMMEND-ONLY: $N_GAPS gap(s) written to the mode-600 report; nothing posted, nothing written to Confluence (D7)."
exit 0
# ==============================================================================
# PROVISIONING (NOT DONE HERE — gated):
# - confluence-bot SERVICE ACCOUNT (D6): create a dedicated Atlassian service account scoped
# to EDIT the IT space ONLY (Confluence API tokens inherit the whole user's permissions, so a
# scoped service account bounds blast radius; costs one Confluence seat). Mint its API token,
# store it in ~/secrev.env (mode 600) as CONFLUENCE_API_TOKEN (+ CONFLUENCE_BASE_URL/EMAIL).
# Rotate the token on a 90-DAY cadence. Until this exists, the LIVE API checks SKIP (above),
# never alarm. This whole step is gated (Adam-provisioned), not done by this script.
# - LIVE Confluence READ checks (page-existence + staleness) only run once those creds exist.
# - ON-DEMAND WRITE path (D7) — the actual Confluence update, including Mermaid architecture-map
# edits via ~/.claude/scripts/confluence_mermaid.py — is a SEPARATE, LATER, SSH-invoked path.
# Before any --apply, that script must pass a LIVE DRY-RUN against page 1540098: verify it
# lists all 16 weweave Mermaid macros and that a no-op set produces a clean (empty) revert-diff.
# ADF-only + macro-count + revert-diff guards are load-bearing (a full-body markdown round-trip
# has SILENTLY DELETED every diagram on 1540098 before). This script NEVER calls --apply.
# - No systemd unit / timer is installed here. Wiring the scheduled run (weekly) under the
# coordinator is provisioning and is gated.
# - The coordinator (design §5, checker_coordinator.sh) registers + drives this checker; that
# registry edit is done centrally, NOT in this script.
# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations for the
# build session, tracked outside this script.
# ==============================================================================

587
checkers/dependency-cve.sh Executable file
View file

@ -0,0 +1,587 @@
#!/usr/bin/env bash
# dependency-cve.sh — Plane-1 / Tier-1 checker for the R720 agent-team.
#
# Design refs: docs/r720-agent-team-design.md §4 (Tier 1 roster: dependency-cve —
# "Cross-ref lockfiles vs advisories org-wide; report + feed fixer. Complements Dependabot")
# and §7 Phase 2 ("coordinator + second checker"). This is the SECOND Plane-1 checker built
# on the Phase-0 shared substrate (lib/sweep_substrate.sh); it mirrors compliance-drift.sh's
# conventions verbatim so the coordinator (§5) can drive both identically.
#
# WHAT IT DOES (read-only):
# Scans the SAME shallow clean clones nightly_sweep.sh already produced in $MIRROR_DIR — it
# does NOT re-clone (mirrors-first; an optional --refresh re-runs discovery+mirror via the
# shared substrate). In each mirror it parses dependency lockfiles/manifests with PINNED,
# exact versions, extracts (ecosystem, package, version) tuples, and cross-references them
# against the OSV advisory database to flag known-vulnerable pinned deps. This complements
# Dependabot (design §4): it is org-wide, runs on the server-side mirrors, and feeds the
# fixer queue in a later phase.
#
# Manifests parsed (and the OSV ecosystem each maps to):
# requirements.txt -> PyPI (only EXACT '==' pins; ranges/unpinned are skipped)
# poetry.lock -> PyPI ([[package]] name/version blocks)
# Pipfile.lock -> PyPI (default+develop, "==x.y.z" version strings)
# package-lock.json -> npm (packages[].version / dependencies[].version)
# yarn.lock -> npm ("pkg@range:\n version \"x\"" stanzas)
# packages.lock.json -> NuGet (.dependencies[tfm][pkg].resolved)
# *.csproj -> NuGet (<PackageReference Include=.. Version=..>)
# Only EXACTLY-pinned versions are cross-referenced (an unpinned/range spec has no single
# version to query and is not a confirmed vulnerable artifact — no false alarms on no-data,
# memory feedback_cloudwatch_alarms).
#
# ADVISORY SOURCE (live): OSV batch API POST https://api.osv.dev/v1/querybatch (NO auth token).
# Guarded behind a --no-api / offline check exactly like compliance-drift's GitHub-API checks:
# on missing curl OR a failed/empty network response, the API lookup is SKIPPED and noted in
# the report — a vuln is NEVER reported on missing advisory data. Network calls are minimal
# (one batched POST) and fail-safe.
#
# AGENTIC TIEBREAK (design §4, "Claude + GPT tiebreak"): OPTIONAL and only relevant in LIVE mode
# for ambiguous severity. For THIS phase the deterministic OSV core is the whole checker — NO
# LLM is invoked in --canary/--dry-run. A clearly-marked inert stub hook (maybe_tiebreak) marks
# the future seam; it does nothing offline and nothing in this phase.
#
# CANARY / DRY-RUN (offline, no network, no token):
# --canary runs against a planted fixture (checkers/fixtures/dependency-cve/) and asserts the
# known vuln count against EXPECTED_VULN_COUNT (exit 3 on mismatch). Because OSV needs network,
# the canary consults a LOCAL offline advisory fixture (fixtures/dependency-cve/osv-advisories.json)
# INSTEAD of the network — so it is fully offline + deterministic. --canary implies --dry-run +
# --no-api. This is the anti-complacency floor (design §6.4) AND the routing dry-run (§7 Phase 2):
# with --dry-run the Slack alarm is composed + printed but NOT POSTed.
#
# SCOPE / SAFETY:
# Read-only. Fixtures ship git metadata as dotgit/ (renamed to .git/ at run time) so they
# commit into THIS repo without becoming submodules — the SAME trick compliance-drift uses.
# Does NOT touch agent_team/ or agent-team/, and is NOT wired into systemd — that is Phase-6
# provisioning (gated). See the "PROVISIONING (NOT DONE HERE)" note at the bottom.
#
# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED.
set -euo pipefail
export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
log() { echo "[dependency-cve] $*" >&2; }
die() { echo "[dependency-cve] FATAL: $*" >&2; exit 2; }
# --- Shared substrate ---------------------------------------------------------
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SUBSTRATE="$HERE/../lib/sweep_substrate.sh"
[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE"
# shellcheck source=../lib/sweep_substrate.sh
. "$SUBSTRATE"
# --- Config + defaults (env, all optional) ------------------------------------
GH_ORG="${GH_ORG:-Sea-Haven-Industries}"
MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}"
REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/dependency-cve}"
OSV_BATCH_URL="${OSV_BATCH_URL:-https://api.osv.dev/v1/querybatch}"
REFRESH=0 # --refresh: re-run discovery+mirror via substrate (network). Default: reuse mirrors.
DO_API=1 # --no-api: skip the OSV advisory lookup (offline). Without it, nothing matches.
DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run).
CANARY=0 # --canary: run against the planted fixture + assert the known vuln count.
TARGETS_OVERRIDE="" # --targets "p1 p2": scan explicit dirs instead of the mirror set.
ADVISORIES_FILE="" # --advisories-file PATH: consult a local advisory JSON instead of the OSV API.
usage() {
cat >&2 <<EOF
dependency-cve.sh — Plane-1 Tier-1 vulnerable-dependency checker (read-only)
--canary run against the planted fixture and assert the known vuln count
(implies --dry-run + --no-api; uses the OFFLINE advisory fixture)
--dry-run compose the Slack alarm but DO NOT post it (routing dry-run)
--no-api skip the OSV advisory lookup (offline; nothing can match)
--advisories-file P consult a LOCAL advisory JSON at P instead of the OSV network API
(offline + deterministic; same file shape as the canary fixture)
--refresh re-discover + re-mirror via the shared substrate before scanning (network)
--targets "a b" scan these explicit repo dirs instead of \$MIRROR_DIR/* (no clone)
-h|--help this help
Env: GH_ORG MIRROR_DIR REPORT_ROOT GH_TOKEN SLACK_WEBHOOK_URL OSV_BATCH_URL
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--canary) CANARY=1; DRY_RUN=1; DO_API=0 ;;
--dry-run) DRY_RUN=1 ;;
--no-api) DO_API=0 ;;
--advisories-file) shift; ADVISORIES_FILE="${1:-}" ;;
--refresh) REFRESH=1 ;;
--targets) shift; TARGETS_OVERRIDE="${1:-}" ;;
-h|--help) usage; exit 0 ;;
*) die "unknown arg: $1 (see --help)" ;;
esac
shift
done
command -v jq >/dev/null || die "jq is required"
command -v git >/dev/null || die "git is required"
# --- Report dir (mode 600 reports; matches sweep conventions) -----------------
umask 077
UTC_DATE="$(date -u +%Y-%m-%d)"
UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
REPORT_DIR="$REPORT_ROOT/$UTC_DATE"
mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true
# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope
SWEEP_LOG="$REPORT_DIR/dependency-cve.log" # name the substrate's post_slack_alarm() references
REPORT_JSON="$REPORT_DIR/dependency-cve.json"
REPORT_TXT="$REPORT_DIR/dependency-cve.txt"
log "=== dependency-cve $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API refresh=$REFRESH) ==="
# ------------------------------------------------------------------------------
# FINDINGS (spirit of finding.schema.json so the coordinator can route like an agentic
# finding). category="other" (a vulnerable-dependency is not one of the schema's security
# categories); status="confirmed" only for an exact pinned version that MATCHES an advisory.
# A pinned dep with NO advisory match is NOT a finding; an unqueryable/skipped advisory lookup
# is NOT a finding (memory feedback_cloudwatch_alarms: no false alarms on missing data).
# ------------------------------------------------------------------------------
declare -a FINDINGS=()
add_finding() { # repo id title severity pkg version advisory_id summary fixed_version
local repo="$1" id="$2" title="$3" sev="$4" pkg="$5" ver="$6" adv="$7" summ="$8" fixed="$9"
FINDINGS+=( "$(jq -n \
--arg repo "$repo" --arg id "$id" --arg title "$title" --arg sev "$sev" \
--arg pkg "$pkg" --arg ver "$ver" --arg adv "$adv" --arg summ "$summ" --arg fixed "$fixed" \
'{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other",
check:"vulnerable-dependency", status:"confirmed",
proof:{package:$pkg, version:$ver, advisory_id:$adv, summary:$summ, fixed_version:$fixed}}')" )
}
declare -a SKIPPED_CHECKS=() # (repo:reason) lookups skipped on missing data — reported, never alarmed
note_skip() { SKIPPED_CHECKS+=( "$1" ); }
# Severity normalizer: map OSV/GHSA strings + CVSS scores into the schema's enum.
norm_sev() { # raw_severity cvss_score -> critical|high|medium|low
local raw; raw="$(echo "${1:-}" | tr '[:upper:]' '[:lower:]')"
local cvss="${2:-}"
case "$raw" in
critical) echo critical; return ;;
high) echo high; return ;;
moderate|medium) echo medium; return ;;
low) echo low; return ;;
esac
# Fall back to CVSS base score banding (NVD/CVSSv3 thresholds).
if [ -n "$cvss" ] && [ "$cvss" != "null" ]; then
awk -v c="$cvss" 'BEGIN{
if (c+0>=9.0) print "critical";
else if (c+0>=7.0) print "high";
else if (c+0>=4.0) print "medium";
else print "low"; }'
return
fi
echo medium # unknown severity: medium (a real match we cannot rank), never dropped
}
# ==============================================================================
# MANIFEST PARSERS — each emits "ECOSYSTEM<TAB>package<TAB>version" lines (exact pins only).
# Pure text/jq parsing; no project tooling invoked. Unknown/odd lines are skipped silently.
# ==============================================================================
# requirements.txt: only EXACT '==' pins (skip ranges, markers, comments, -e/-r includes, extras).
parse_requirements() { # file
local f="$1"
sed -E 's/[[:space:]]*#.*$//' "$f" 2>/dev/null \
| grep -E '==' \
| while IFS= read -r line; do
line="$(echo "$line" | tr -d '[:space:]')"
[ -n "$line" ] || continue
case "$line" in -*|.*|git+*|http*) continue ;; esac
# strip extras: pkg[extra]==1.2.3 -> pkg
local name ver
name="$(echo "$line" | sed -E 's/\[[^]]*\].*//; s/[<>=!~;].*$//')"
ver="$(echo "$line" | sed -E 's/^[^=]*==//; s/[ ;].*$//')"
# only a clean exact version (digits/dots/alnum), no range operators left
case "$ver" in *','*|*'<'*|*'>'*|*'*'*|'') continue ;; esac
[ -n "$name" ] && [ -n "$ver" ] && printf 'PyPI\t%s\t%s\n' "$name" "$ver"
done
}
# poetry.lock: [[package]] blocks with name = "x" / version = "y".
parse_poetry_lock() { # file
local f="$1"
awk '
/^\[\[package\]\]/ { name=""; ver=""; next }
/^name = / { gsub(/^name = "|"$/,""); name=$0; next }
/^version = / { gsub(/^version = "|"$/,""); ver=$0;
if (name!="" && ver!="") printf "PyPI\t%s\t%s\n", name, ver; next }
' "$f" 2>/dev/null
}
# Pipfile.lock: JSON; default + develop maps; versions look like "==1.2.3".
parse_pipfile_lock() { # file
local f="$1"
jq -r '
(.default // {}) * (.develop // {}) | to_entries[]
| select(.value.version != null)
| .key as $n | (.value.version | sub("^=="; "")) as $v
| select($v | test("^[0-9][0-9A-Za-z.+-]*$"))
| "PyPI\t\($n)\t\($v)"
' "$f" 2>/dev/null || true
}
# package-lock.json: prefer v2/v3 .packages (node_modules/<name> keys), else v1 .dependencies.
parse_package_lock() { # file
local f="$1"
jq -r '
if (.packages != null) then
(.packages | to_entries[]
| select(.key | startswith("node_modules/"))
| select(.value.version != null)
| (.key | sub("^.*node_modules/"; "")) as $n
| "npm\t\($n)\t\(.value.version)")
elif (.dependencies != null) then
[paths(objects | has("version")) as $p | {n: $p[-1], v: (getpath($p).version)}]
| .[] | select(.v != null) | "npm\t\(.n)\t\(.v)"
else empty end
' "$f" 2>/dev/null || true
}
# yarn.lock: stanzas "spec@range, spec@range:\n version \"x.y.z\"".
parse_yarn_lock() { # file
local f="$1"
awk '
/^[^[:space:]#].*:[[:space:]]*$/ {
# header line: take first spec, strip trailing colon + quotes, derive package name
hdr=$0; sub(/:[[:space:]]*$/,"",hdr);
split(hdr, specs, ", "); first=specs[1]; gsub(/"/,"",first);
# package name = everything before the LAST @ (handles @scope/pkg@range)
at=0; for (i=2;i<=length(first);i++){ if (substr(first,i,1)=="@") at=i }
pkg=(at>1)? substr(first,1,at-1) : first;
next
}
/^[[:space:]]+version / {
v=$0; gsub(/^[[:space:]]+version[[:space:]]+"?|"?[[:space:]]*$/,"",v);
if (pkg!="" && v!="") printf "npm\t%s\t%s\n", pkg, v;
pkg=""; next
}
' "$f" 2>/dev/null
}
# packages.lock.json (NuGet): .dependencies[tfm][pkg].resolved.
parse_packages_lock() { # file
local f="$1"
jq -r '
(.dependencies // {}) | to_entries[] | .value | to_entries[]
| select(.value.resolved != null)
| "NuGet\t\(.key)\t\(.value.resolved)"
' "$f" 2>/dev/null || true
}
# *.csproj (NuGet): <PackageReference Include="X" Version="Y" />.
parse_csproj() { # file
local f="$1"
grep -oE '<PackageReference[^>]*>' "$f" 2>/dev/null \
| while IFS= read -r tag; do
local inc ver
inc="$(echo "$tag" | sed -nE 's/.*Include="([^"]+)".*/\1/p')"
ver="$(echo "$tag" | sed -nE 's/.*Version="([^"]+)".*/\1/p')"
# only exact versions (no range brackets/commas/wildcards)
case "$ver" in ''|*'['*|*']'*|*'('*|*')'*|*','*|*'*'*) continue ;; esac
[ -n "$inc" ] && [ -n "$ver" ] && printf 'NuGet\t%s\t%s\n' "$inc" "$ver"
done
}
# Extract ALL (ecosystem, package, version) tuples from one repo dir. Dedup at the end.
extract_deps() { # repo_dir -> TSV "ECOSYSTEM\tpackage\tversion" on stdout
local dir="$1" f
# requirements.txt (any depth, excluding .git)
while IFS= read -r f; do [ -n "$f" ] && parse_requirements "$f"; done \
< <(find "$dir" -maxdepth 4 -name requirements.txt -not -path '*/.git/*' 2>/dev/null)
while IFS= read -r f; do [ -n "$f" ] && parse_poetry_lock "$f"; done \
< <(find "$dir" -maxdepth 4 -name poetry.lock -not -path '*/.git/*' 2>/dev/null)
while IFS= read -r f; do [ -n "$f" ] && parse_pipfile_lock "$f"; done \
< <(find "$dir" -maxdepth 4 -name Pipfile.lock -not -path '*/.git/*' 2>/dev/null)
while IFS= read -r f; do [ -n "$f" ] && parse_package_lock "$f"; done \
< <(find "$dir" -maxdepth 4 -name package-lock.json -not -path '*/.git/*' 2>/dev/null)
while IFS= read -r f; do [ -n "$f" ] && parse_yarn_lock "$f"; done \
< <(find "$dir" -maxdepth 4 -name yarn.lock -not -path '*/.git/*' 2>/dev/null)
while IFS= read -r f; do [ -n "$f" ] && parse_packages_lock "$f"; done \
< <(find "$dir" -maxdepth 4 -name packages.lock.json -not -path '*/.git/*' 2>/dev/null)
while IFS= read -r f; do [ -n "$f" ] && parse_csproj "$f"; done \
< <(find "$dir" -maxdepth 4 -name '*.csproj' -not -path '*/.git/*' 2>/dev/null)
}
# ==============================================================================
# ADVISORY LOOKUP
# ==============================================================================
# OFFLINE: consult a local advisory file (the canary fixture, or --advisories-file). Keyed by
# "ECOSYSTEM|package|version" -> array of {id,summary,severity,cvss,fixed_version}. Deterministic.
lookup_offline() { # advisories_file ecosystem package version -> advisory JSON array (or [])
local af="$1" eco="$2" pkg="$3" ver="$4"
jq -c --arg k "$eco|$pkg|$ver" '(.advisories[$k] // [])' "$af" 2>/dev/null || echo '[]'
}
# LIVE: one batched POST to the OSV querybatch API (no token). Returns one results[] per query
# in input order. Fail-safe: on missing curl, transport failure, or a non-array body, returns ""
# (the caller then SKIPS — never alarms on missing advisory data).
osv_querybatch() { # queries_json (array of {package:{ecosystem,name},version}) -> results JSON or ""
local queries="$1"
command -v curl >/dev/null || { return 1; }
local body
body="$(curl -fsS -X POST -H 'Content-Type: application/json' \
--max-time 30 \
--data "$(jq -n --argjson q "$queries" '{queries:$q}')" \
"$OSV_BATCH_URL" 2>>"$REPORT_DIR/osv.log")" || return 1
echo "$body" | jq -e '.results | type=="array"' >/dev/null 2>&1 || return 1
echo "$body"
}
# Inert future seam (design §4 "Claude + GPT tiebreak"): in LIVE mode, an ambiguous-severity
# advisory could be escalated to a cross-family judge. This phase keeps the deterministic core
# ONLY — the stub does nothing and is never reached offline / in canary / dry-run.
maybe_tiebreak() { # advisory_json (no-op stub; phase-2 intentionally inert)
return 0
}
# ==============================================================================
# TARGET RESOLUTION
# ==============================================================================
declare -a REPO_NAMES=(); declare -A REPO_DIR=()
if [ "$CANARY" -eq 1 ]; then
FIXTURE_ROOT="$HERE/fixtures/dependency-cve"
[ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT"
# The canary is OFFLINE: it consults the planted advisory fixture instead of the OSV network,
# unless an explicit --advisories-file override was given.
[ -n "$ADVISORIES_FILE" ] || ADVISORIES_FILE="$FIXTURE_ROOT/osv-advisories.json"
[ -f "$ADVISORIES_FILE" ] || die "canary advisory fixture missing: $ADVISORIES_FILE"
# Fixtures ship git metadata as dotgit/ (not .git/) so they are committable into THIS repo
# without becoming nested submodules. Materialize: copy + rename dotgit -> .git into a mode-700
# temp area removed on exit (same trick as compliance-drift.sh).
FIXTURE_WORK="$(mktemp -d "${TMPDIR:-/tmp}/dependency-cve-canary.XXXXXX")"
trap 'rm -rf "$FIXTURE_WORK"' EXIT
log "canary: materializing planted fixtures from $FIXTURE_ROOT into $FIXTURE_WORK"
for d in "$FIXTURE_ROOT"/*/; do
[ -d "$d/dotgit" ] || continue # only fixture repos (skip README.md, *.json etc.)
nm="$(basename "$d")"
cp -R "$d" "$FIXTURE_WORK/$nm"
mv "$FIXTURE_WORK/$nm/dotgit" "$FIXTURE_WORK/$nm/.git"
# Manifests are stored as <name>.fixture so GitHub's dependency graph / the
# dependency-review CI action does NOT parse the deliberately-vulnerable canary
# pins as real project dependencies. Restore their real names in the materialized
# work area so the checker's per-ecosystem parsers dispatch correctly (same
# committable-without-side-effects rationale as the dotgit/ rename above).
while IFS= read -r ff; do
[ -n "$ff" ] && mv "$ff" "${ff%.fixture}"
done < <(find "$FIXTURE_WORK/$nm" -name '*.fixture' -not -path '*/.git/*' 2>/dev/null)
REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$FIXTURE_WORK/$nm"
done
elif [ -n "$TARGETS_OVERRIDE" ]; then
# shellcheck disable=SC2206 # intentional word-split of the space-separated --targets list
arr=( $TARGETS_OVERRIDE )
for p in "${arr[@]}"; do p="${p/#\~/$HOME}"; nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p"; done
log "explicit targets: ${REPO_NAMES[*]}"
else
if [ "$REFRESH" -eq 1 ]; then
[ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN"
command -v curl >/dev/null || die "--refresh needs curl"
mkdir -p "$MIRROR_DIR"
log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)"
DISCOVERED="$REPORT_DIR/discovered.tsv"
if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then
while IFS=$'\t' read -r name url branch; do
[ -n "$name" ] || continue
mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)"
done < "$DISCOVERED"
else
log "discovery failed — falling back to existing mirrors (coverage may be stale)"
fi
fi
# Default + post-refresh: enumerate EXISTING mirrors. No clone here — reuse the sweep's clones.
[ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)"
for d in "$MIRROR_DIR"/*/; do
[ -d "$d/.git" ] || continue
nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}"
done
log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)"
fi
[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to scan"
# Decide HOW advisories are looked up: offline file, or the live OSV API, or skip entirely.
# An explicit --advisories-file always wins (offline + deterministic, even without --canary).
ADV_MODE="none"
if [ -n "$ADVISORIES_FILE" ]; then
[ -f "$ADVISORIES_FILE" ] || die "advisories file not found: $ADVISORIES_FILE"
ADV_MODE="offline"
elif [ "$DO_API" -eq 1 ] && command -v curl >/dev/null; then
ADV_MODE="api"
elif [ "$DO_API" -eq 1 ]; then
log "OSV lookup requested but curl unavailable — skipping advisory match (no false alarms on missing data)"
fi
log "advisory mode: $ADV_MODE"
# ==============================================================================
# RUN: extract deps per repo, then cross-reference against advisories
# ==============================================================================
for nm in "${REPO_NAMES[@]}"; do
dir="${REPO_DIR[$nm]}"
# Unique (ecosystem, package, version) tuples for this repo.
deps_tsv="$(extract_deps "$dir" | sort -u || true)"
ndeps=0; [ -n "$deps_tsv" ] && ndeps="$(printf '%s\n' "$deps_tsv" | grep -c . || true)"
log " [$nm] extracted $ndeps pinned dependency tuple(s)"
[ "$ndeps" -gt 0 ] || { note_skip "$nm:no-pinned-deps"; continue; }
if [ "$ADV_MODE" = "none" ]; then
note_skip "$nm:advisory-lookup-skipped(offline/no-curl)"
continue
fi
if [ "$ADV_MODE" = "offline" ]; then
# Deterministic local lookup, one tuple at a time.
while IFS=$'\t' read -r eco pkg ver; do
[ -n "$pkg" ] || continue
advs="$(lookup_offline "$ADVISORIES_FILE" "$eco" "$pkg" "$ver")"
cnt="$(echo "$advs" | jq 'length' 2>/dev/null || echo 0)"
[ "${cnt:-0}" -gt 0 ] || continue
i=0
while [ "$i" -lt "$cnt" ]; do
adv="$(echo "$advs" | jq -c --argjson i "$i" '.[$i]')"
aid="$(echo "$adv" | jq -r '.id // "UNKNOWN"')"
summ="$(echo "$adv" | jq -r '.summary // ""')"
rawsev="$(echo "$adv"| jq -r '.severity // ""')"
cvss="$(echo "$adv" | jq -r '.cvss // empty')"
fixed="$(echo "$adv" | jq -r '.fixed_version // ""')"
sev="$(norm_sev "$rawsev" "$cvss")"
maybe_tiebreak "$adv" # inert in this phase
add_finding "$nm" "vuln-$(echo "${pkg}-${ver}-${aid}" | tr -c 'A-Za-z0-9-' '-')" \
"$pkg $ver is vulnerable ($aid)" "$sev" \
"$pkg" "$ver" "$aid" "$summ" "$fixed"
i=$((i+1))
done
done <<< "$deps_tsv"
continue
fi
# ADV_MODE = api: build ONE batched OSV query for all this repo's tuples (minimal network).
queries="$(printf '%s\n' "$deps_tsv" | jq -R -s '
[ split("\n")[] | select(length>0) | split("\t")
| {package:{ecosystem:.[0], name:.[1]}, version:.[2]} ]')"
# Keep a parallel TSV array so we can re-associate results[] (OSV preserves input order).
if ! results="$(osv_querybatch "$queries")"; then
note_skip "$nm:osv-querybatch-failed" # transport/HTTP failure -> skip, NEVER alarm
continue
fi
# Walk each tuple alongside its result entry.
idx=0
while IFS=$'\t' read -r eco pkg ver; do
[ -n "$pkg" ] || continue
vulns="$(echo "$results" | jq -c --argjson i "$idx" '(.results[$i].vulns // [])')"
idx=$((idx+1))
vcnt="$(echo "$vulns" | jq 'length' 2>/dev/null || echo 0)"
[ "${vcnt:-0}" -gt 0 ] || continue
j=0
while [ "$j" -lt "$vcnt" ]; do
v="$(echo "$vulns" | jq -c --argjson j "$j" '.[$j]')"
aid="$(echo "$v" | jq -r '.id // "UNKNOWN"')"
summ="$(echo "$v" | jq -r '.summary // (.details // "" | .[0:160])')"
# OSV severity: prefer database_specific.severity, else the CVSS vector score band.
rawsev="$(echo "$v" | jq -r '.database_specific.severity // ""')"
cvss="$(echo "$v" | jq -r '[.severity[]? | select(.type|test("CVSS")) | .score] | .[0] // empty' \
| grep -oE '[0-9]+\.[0-9]+' | head -1 || true)"
fixed="$(echo "$v" | jq -r '
[.affected[]?.ranges[]?.events[]? | select(.fixed != null) | .fixed] | .[0] // ""')"
sev="$(norm_sev "$rawsev" "$cvss")"
maybe_tiebreak "$v" # inert in this phase
add_finding "$nm" "vuln-$(echo "${pkg}-${ver}-${aid}" | tr -c 'A-Za-z0-9-' '-')" \
"$pkg $ver is vulnerable ($aid)" "$sev" \
"$pkg" "$ver" "$aid" "$summ" "$fixed"
j=$((j+1))
done
done <<< "$deps_tsv"
done
# ==============================================================================
# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to compliance-drift)
# ==============================================================================
if [ "${#FINDINGS[@]}" -gt 0 ]; then
FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)"
else
FINDINGS_JSON="[]"
fi
if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then
SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)"
else
SKIPPED_JSON="[]"
fi
N_VULN="$(echo "$FINDINGS_JSON" | jq 'length')"
N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high" or .severity=="critical")] | length')"
N_REPOS_VULN="$(echo "$FINDINGS_JSON" | jq '[.[].repo] | unique | length')"
jq -n \
--arg checker "dependency-cve" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \
--arg advmode "$ADV_MODE" --argjson scanned "${#REPO_NAMES[@]}" \
--argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \
'{checker:$checker, generated:$ts, org:$org, advisory_mode:$advmode,
repos_scanned:$scanned, vuln_count:($findings|length),
repos_with_vulns:([$findings[].repo]|unique|length),
findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON"
{
echo "dependency-cve report — $UTC_STAMP"
echo "org=$GH_ORG repos_scanned=${#REPO_NAMES[@]} advisory_mode=$ADV_MODE"
echo "vulnerable deps: $N_VULN ($N_HIGH high/critical) across $N_REPOS_VULN repo(s)"
echo
echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n fix: upgrade \(.proof.package) -> \(.proof.fixed_version) (\(.proof.summary))"'
if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then
echo; echo "skipped (missing data — NOT counted as a vuln):"
echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"'
fi
} > "$REPORT_TXT"
chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true
log "report: $REPORT_JSON ($N_VULN vuln finding(s), $N_REPOS_VULN repo(s))"
# ==============================================================================
# CANARY ASSERTION (anti-complacency floor, design §6.4)
# ==============================================================================
if [ "$CANARY" -eq 1 ]; then
EXPECT_FILE="$HERE/fixtures/dependency-cve/EXPECTED_VULN_COUNT"
[ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE"
EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")"
log "canary assertion: expected vuln=$EXPECTED, got=$N_VULN"
if [ "$N_VULN" -ne "$EXPECTED" ]; then
echo "[dependency-cve] CANARY FAIL: planted-vuln count mismatch (expected $EXPECTED, got $N_VULN)" >&2
echo " -> a parser or the advisory match regressed, or the fixture changed. See $REPORT_TXT." >&2
exit 3
fi
log "canary PASS: all $EXPECTED planted vulnerable deps detected."
fi
# ==============================================================================
# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms)
# ==============================================================================
if [ "$N_VULN" -eq 0 ]; then
log "no vulnerable dependencies — posting NOTHING to Slack (ALARM-only policy)."
exit 0
fi
ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r '
group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')"
SLACK_TEXT=":lock: *Sea Haven dependency-cve — ALARM* ($UTC_STAMP)
$N_VULN vulnerable pinned dependency(ies) across $N_REPOS_VULN repo(s) ($N_HIGH high/critical):
$ALARM_BODY
Source: OSV advisory DB ($ADV_MODE) · complements Dependabot
Report (mode 600): \`$REPORT_JSON\` (on R720)"
SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)"
echo "$SLACK_TEXT" >&2
if [ "$DRY_RUN" -eq 1 ]; then
log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 2)."
exit 0
fi
post_slack_alarm "$SLACK_TEXT"
exit 0
# ==============================================================================
# PROVISIONING (NOT DONE HERE — gated, Phase 6):
# - No systemd unit / timer is installed by this script. Wiring it into the live
# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated.
# - The coordinator (design §5, checker_coordinator.sh) runs this alongside other
# Tier-1 checkers under one shared budget + versioned rotation state.
# - The LIVE "Claude + GPT tiebreak" severity-judge (design §4) is the only LLM seam;
# it is an inert stub here (maybe_tiebreak) and stays off in canary/dry-run/offline.
# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations
# for the build session, tracked outside this script.
# ==============================================================================

482
checkers/doc-drift.sh Executable file
View file

@ -0,0 +1,482 @@
#!/usr/bin/env bash
# doc-drift.sh — Plane-1 / Tier-1 checker for the R720 agent-team.
#
# Design refs: docs/r720-agent-team-design.md §4 (Tier 1 roster: doc-drift —
# "Flags repos whose architecture moved but Confluence/README did not") and §7 Phase 3
# ("doc-drift + step-ca/Roles Anywhere + aws-posture"). This is the THIRD Plane-1 checker
# built on the Phase-0 shared substrate (lib/sweep_substrate.sh); it mirrors
# compliance-drift.sh / dependency-cve.sh conventions VERBATIM so the coordinator (§5) can
# drive all of them identically. doc-drift is UNGATED (only aws-posture in this phase is
# hard-gated behind the GPT-4.1 IAM cross-review; that checker is NOT built here).
#
# WHAT IT DOES (read-only):
# Scans the SAME shallow clean clones nightly_sweep.sh already produced in $MIRROR_DIR — it
# does NOT re-clone (mirrors-first; an optional --refresh re-runs discovery+mirror via the
# shared substrate). In each mirror it flags repos whose ARCHITECTURE MOVED but the README
# DID NOT — i.e. documentation drift. The checklist is DETERMINISTIC and GROUNDED in the
# global CLAUDE.md README obligation; it does NOT invent fuzzy judgments. See "CHECKLIST".
#
# This phase is the deterministic core ONLY. The design's "Gemini (large context)" judge
# layer (§4) is a LATER enhancement: a clearly-marked inert stub hook (maybe_judge) marks
# the future seam; it does NOTHING offline and NOTHING in this phase.
#
# REPORTING (matches secrev sweep conventions):
# - Writes a per-run JSON + text report under $REPORT_ROOT/<UTC-date>/, mode 600 (umask 077).
# - Slack ALARM-ONLY: a clean run (no confirmed drift) posts NOTHING (memory
# feedback_cloudwatch_alarms). Secret-shaped values are redacted from the Slack string.
# - Reuses the substrate's redact() + post_slack_alarm() verbatim.
#
# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping):
# redact, post_slack_alarm -> Slack delivery (reads SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG)
# discover_repos, mirror_repo-> ONLY on --refresh (reads GH_TOKEN, GH_ORG, MIRROR_DIR, REPORT_DIR)
# Default path enumerates EXISTING $MIRROR_DIR/*/.git dirs — zero clones, zero network.
#
# CANARY / DRY-RUN (offline, no network, no token):
# --canary runs the checklist against a planted-drift fixture (checkers/fixtures/doc-drift/)
# and asserts the known drift count. This is the anti-complacency floor (design §6.4) AND the
# routing dry-run (§7 Phase 3): with --dry-run, the Slack alarm is composed + printed but NOT
# POSTed. Fully offline-smoke-testable (the checks are filesystem + `git log`, no network).
#
# SCOPE / SAFETY:
# Read-only. All checks are filesystem + local `git log`; NO network, NO token, NO GitHub API
# (doc-drift has no API-only checks — it is purely tree+history). Fixtures ship git metadata as
# dotgit/ (renamed to .git/ at run time) so they commit into THIS repo without becoming
# submodules — the SAME trick compliance-drift / dependency-cve use. A repo with NO README is
# SKIPPED (compliance-drift owns readme-present); doc-drift never double-flags a missing README.
#
# This script does NOT touch agent_team/ or agent-team/, is NOT wired into systemd, and does NOT
# stand up step-ca / Roles Anywhere / aws-posture — that is Phase-3/6 provisioning (gated). See
# the "PROVISIONING (NOT DONE HERE)" note at the bottom.
#
# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED.
set -euo pipefail
export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
log() { echo "[doc-drift] $*" >&2; }
die() { echo "[doc-drift] FATAL: $*" >&2; exit 2; }
# --- Shared substrate ---------------------------------------------------------
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SUBSTRATE="$HERE/../lib/sweep_substrate.sh"
[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE"
# shellcheck source=../lib/sweep_substrate.sh
. "$SUBSTRATE"
# --- Config + defaults (env, all optional) ------------------------------------
GH_ORG="${GH_ORG:-Sea-Haven-Industries}"
MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}"
REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/doc-drift}"
# Docs-only repos describe themselves differently (a handbook is its own doc); skip the
# architecture-omission scan for them. They still get the staleness check.
DOCS_ONLY_REPOS="${DOCS_ONLY_REPOS:-engineering-handbook}"
# Staleness thresholds: README must lag the newest code by BOTH at least this many days AND
# this many substantial code commits before we call it drift (two-factor = no false alarm on a
# single quick fix landed after a doc commit; memory feedback_cloudwatch_alarms).
DOC_DRIFT_STALE_DAYS="${DOC_DRIFT_STALE_DAYS:-60}"
DOC_DRIFT_STALE_COMMITS="${DOC_DRIFT_STALE_COMMITS:-3}"
REFRESH=0 # --refresh: re-run discovery+mirror via substrate (network). Default: reuse mirrors.
DO_API=1 # --no-api: accepted for interface-parity with the other checkers; doc-drift makes
# NO API calls, so this flag is a documented no-op (kept so the coordinator
# can pass a uniform flag set to every Tier-1 checker).
DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run).
CANARY=0 # --canary: run against the planted-drift fixture + assert the known count.
TARGETS_OVERRIDE="" # --targets "p1 p2": scan explicit dirs instead of the mirror set.
usage() {
cat >&2 <<EOF
doc-drift.sh — Plane-1 Tier-1 documentation-drift checker (read-only)
--canary run against the planted-drift fixture and assert the known drift count
(implies --dry-run; fully offline smoke test — no network, no token)
--dry-run compose the Slack alarm but DO NOT post it (routing dry-run)
--no-api accepted for parity with the other Tier-1 checkers; doc-drift makes NO
API calls, so this is a documented no-op
--refresh re-discover + re-mirror via the shared substrate before scanning (network)
--targets "a b" scan these explicit repo dirs instead of \$MIRROR_DIR/* (no clone)
-h|--help this help
Env: GH_ORG MIRROR_DIR REPORT_ROOT GH_TOKEN SLACK_WEBHOOK_URL DOCS_ONLY_REPOS
DOC_DRIFT_STALE_DAYS DOC_DRIFT_STALE_COMMITS
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--canary) CANARY=1; DRY_RUN=1 ;;
--dry-run) DRY_RUN=1 ;;
--no-api) DO_API=0 ;;
--refresh) REFRESH=1 ;;
--targets) shift; TARGETS_OVERRIDE="${1:-}" ;;
-h|--help) usage; exit 0 ;;
*) die "unknown arg: $1 (see --help)" ;;
esac
shift
done
command -v jq >/dev/null || die "jq is required"
command -v git >/dev/null || die "git is required"
# --- Report dir (mode 600 reports; matches sweep conventions) -----------------
umask 077
UTC_DATE="$(date -u +%Y-%m-%d)"
UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
REPORT_DIR="$REPORT_ROOT/$UTC_DATE"
mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true
# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope
SWEEP_LOG="$REPORT_DIR/doc-drift.log" # name the substrate's post_slack_alarm() references
REPORT_JSON="$REPORT_DIR/doc-drift.json"
REPORT_TXT="$REPORT_DIR/doc-drift.txt"
# doc-drift makes NO API calls, so DO_API is a documented no-op kept only for coordinator
# flag-parity; surface it in the run banner so the chosen value is auditable (and used).
log "=== doc-drift $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN refresh=$REFRESH api=${DO_API}[no-op] stale_days=$DOC_DRIFT_STALE_DAYS stale_commits=$DOC_DRIFT_STALE_COMMITS) ==="
# ------------------------------------------------------------------------------
# CHECKLIST (grounded — every item cites the README obligation; nothing invented):
#
# readme-omits-component README exists but omits a major existing component
# present in the tree (top-level service dir, SAM/CDK stack,
# Lambda handler dir, openapi/docs API spec)
# -> global CLAUDE.md: "README must accurately describe
# architecture, services, data flow, and configuration"
# readme-stale-vs-code README last-touched commit far older than the newest code
# commit (>= DOC_DRIFT_STALE_DAYS) AND >= DOC_DRIFT_STALE_COMMITS
# substantial code commits landed after the README was touched
# -> global CLAUDE.md: "update the README in the same commit"
#
# A repo with NO README is SKIPPED (compliance-drift owns readme-present; double-flagging would
# be a false alarm). Each emitted finding follows the spirit of finding.schema.json
# (id/title/severity/category/proof/status) so the coordinator can route it like an agentic
# finding. category="other" (doc drift is not one of the schema's security categories).
# status="confirmed" only for deterministic filesystem/git-history facts. The future Gemini
# judge (design §4) is an inert stub (maybe_judge) — never invoked offline / in this phase.
# ------------------------------------------------------------------------------
# Drift accumulator: one JSON object per finding, appended to a bash array.
declare -a FINDINGS=()
add_finding() { # repo id title severity check proof
local repo="$1" id="$2" title="$3" sev="$4" check="$5" proof="$6"
FINDINGS+=( "$(jq -n \
--arg repo "$repo" --arg id "$id" --arg title "$title" --arg sev "$sev" \
--arg check "$check" --arg proof "$proof" \
'{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other",
check:$check, status:"confirmed", proof:{outcome:$proof}}')" )
}
declare -a SKIPPED_CHECKS=() # (repo:check) checks skipped on missing data — reported, never alarmed
note_skip() { SKIPPED_CHECKS+=( "$1" ); }
in_csv() { # needle csv -> 0 if present
local n="$1" csv="$2"; case ",$csv," in *",$n,"*) return 0 ;; *) return 1 ;; esac
}
# Inert future seam (design §4 "Gemini (large context)" judge): in LIVE mode an ambiguous
# omission ("is this component material enough to require a README mention?") could be escalated
# to a large-context judge. This phase keeps the deterministic core ONLY — the stub does nothing
# and is never reached offline / in canary / dry-run.
maybe_judge() { # candidate_json (no-op stub; Phase-3 intentionally inert)
return 0
}
# --- Does a README mention a component name? (case-insensitive, word-ish, deterministic) ----
# Matches the bare name OR the name with a trailing slash (how a dir is usually cited). Strips
# a leading "the " never matters; we test the literal token. Pure grep, no fuzzy matching.
readme_mentions() { # readme_file name
local rf="$1" name="$2"
# Escape regex metacharacters in the component name (defensive; dir names are usually plain).
local esc; esc="$(printf '%s' "$name" | sed -E 's/[][(){}.*+?^$|\\/]/\\&/g')"
grep -qiE "(^|[^A-Za-z0-9_-])${esc}([^A-Za-z0-9_-]|/|$)" "$rf" 2>/dev/null
}
# --- Enumerate the major components present in a repo tree (deterministic) ------
# Emits "TYPE<TAB>label<TAB>mention_token" lines. mention_token is what the README must contain.
# service-dir a top-level directory whose name ends in -service or -api, or named api/web/worker
# sam-cdk-stack a SAM/CDK stack root (template.yaml | app.py at a stack root | cdk.json)
# lambda-dir a Lambda handler dir (a dir named handlers/ or containing handler.* / app.py under handlers/)
# api-spec an openapi/ or docs/ directory or an openapi.* / swagger.* spec file
enumerate_components() { # repo_dir -> TSV lines
local dir="$1" d nm
# 1) top-level service-ish directories (the unit a README is expected to name)
for d in "$dir"/*/; do
[ -d "$d" ] || continue
nm="$(basename "$d")"
case "$nm" in
.git|.github|node_modules|dist|build|vendor|__pycache__|.venv) continue ;;
esac
case "$nm" in
*-service|*-api|api|web|worker|backend|frontend)
printf 'service-dir\t%s\t%s\n' "$nm" "$nm" ;;
esac
done
# 2) SAM / CDK stack roots
if [ -f "$dir/template.yaml" ] || [ -f "$dir/template.yml" ]; then
printf 'sam-cdk-stack\t%s\t%s\n' "template.yaml (SAM stack)" "template.yaml"
fi
if [ -f "$dir/cdk.json" ]; then
printf 'sam-cdk-stack\t%s\t%s\n' "cdk.json (CDK app)" "cdk.json"
fi
# 3) Lambda handler dirs: a top-level/handlers-rooted dir literally named "handlers"
while IFS= read -r d; do
[ -n "$d" ] || continue
printf 'lambda-dir\t%s\t%s\n' "handlers/ (Lambda handlers)" "handlers"
break # one mention requirement for the handlers tree is enough
done < <(find "$dir" -maxdepth 2 -type d -name handlers -not -path '*/.git/*' 2>/dev/null)
# 4) API spec: an openapi/ or docs/ dir, or an openapi.*/swagger.* file
if [ -d "$dir/openapi" ]; then
printf 'api-spec\t%s\t%s\n' "openapi/ (API spec)" "openapi"
elif find "$dir" -maxdepth 2 \( -iname 'openapi.*' -o -iname 'swagger.*' \) -not -path '*/.git/*' -print -quit 2>/dev/null | grep -q .; then
printf 'api-spec\t%s\t%s\n' "openapi/swagger spec" "openapi"
fi
}
# --- README last-touch epoch vs newest code commit (staleness, deterministic git log) -------
# Returns the staleness facts on stdout as TSV "readme_epoch<TAB>newest_code_epoch<TAB>commits_after".
# commits_after = count of commits that touched code (non-doc) files AFTER the README's last touch.
# Code = anything that is NOT a README/markdown/LICENSE/.gitignore/docs file. Prints nothing if
# the repo has no git history or no README in history (caller treats that as "cannot assess").
readme_staleness_facts() { # repo_dir
local dir="$1"
command -v git >/dev/null || return 0
git -C "$dir" rev-parse --git-dir >/dev/null 2>&1 || return 0
# README last-touch (committer epoch of the most recent commit touching README.md).
local rd_epoch
rd_epoch="$(git -C "$dir" log -1 --format='%ct' -- README.md 2>/dev/null || true)"
[ -n "$rd_epoch" ] || return 0 # README not in history -> cannot assess staleness
# Newest commit touching a CODE path (exclude docs/markdown/license/config-noise).
local code_epoch
code_epoch="$(git -C "$dir" log -1 --format='%ct' -- \
':(exclude)README.md' ':(exclude)*.md' ':(exclude)docs/**' \
':(exclude)LICENSE' ':(exclude).gitignore' ':(exclude).github/**' \
2>/dev/null || true)"
[ -n "$code_epoch" ] || return 0 # no code commits -> nothing to be stale against
# Count CODE commits strictly AFTER the README's last touch.
local commits_after
commits_after="$(git -C "$dir" rev-list --count "--since=@${rd_epoch}" HEAD -- \
':(exclude)README.md' ':(exclude)*.md' ':(exclude)docs/**' \
':(exclude)LICENSE' ':(exclude).gitignore' ':(exclude).github/**' \
2>/dev/null || echo 0)"
printf '%s\t%s\t%s\n' "$rd_epoch" "$code_epoch" "${commits_after:-0}"
}
# ==============================================================================
# PER-REPO CHECK (offline; filesystem + local git log only)
# ==============================================================================
check_repo() { # repo_name repo_dir
local repo="$1" dir="$2"
local docs_only=0; in_csv "$repo" "$DOCS_ONLY_REPOS" && docs_only=1
# No README -> doc-drift cannot assess drift; compliance-drift owns readme-present. SKIP.
if [ ! -f "$dir/README.md" ]; then
note_skip "$repo:doc-drift(no-readme — compliance-drift owns readme-present)"
return
fi
local readme="$dir/README.md"
# --- readme-omits-component (skip for docs-only repos: they document differently) ---
if [ "$docs_only" -eq 0 ]; then
local type label token
while IFS=$'\t' read -r type label token; do
[ -n "$token" ] || continue
if ! readme_mentions "$readme" "$token"; then
add_finding "$repo" "readme-omits-$(printf '%s' "$type-$token" | tr -c 'A-Za-z0-9-' '-')" \
"README omits existing component: $label" "medium" "readme-omits-component" \
"global CLAUDE.md: README must accurately describe architecture/services (present in tree, absent from README: $label)"
fi
done < <(enumerate_components "$dir")
else
note_skip "$repo:readme-omits-component(docs-only)"
fi
# --- readme-stale-vs-code (two-factor: age in days AND code-commits-after) ---
local facts; facts="$(readme_staleness_facts "$dir")"
if [ -z "$facts" ]; then
note_skip "$repo:readme-stale-vs-code(no-history-or-no-readme-in-history)"
else
local rd_epoch code_epoch commits_after age_days
IFS=$'\t' read -r rd_epoch code_epoch commits_after <<< "$facts"
age_days=$(( (code_epoch - rd_epoch) / 86400 ))
[ "$age_days" -lt 0 ] && age_days=0
if [ "$age_days" -ge "$DOC_DRIFT_STALE_DAYS" ] && [ "$commits_after" -ge "$DOC_DRIFT_STALE_COMMITS" ]; then
add_finding "$repo" "readme-stale" \
"README is stale: ${age_days}d behind newest code, ${commits_after} code commit(s) since last README touch" \
"medium" "readme-stale-vs-code" \
"global CLAUDE.md: update the README in the same commit as functionality changes (thresholds: >=${DOC_DRIFT_STALE_DAYS}d AND >=${DOC_DRIFT_STALE_COMMITS} code commits)"
fi
fi
maybe_judge "" # inert in this phase (future Gemini large-context seam)
}
# ==============================================================================
# TARGET RESOLUTION
# ==============================================================================
declare -a REPO_NAMES=(); declare -A REPO_DIR=()
if [ "$CANARY" -eq 1 ]; then
FIXTURE_ROOT="$HERE/fixtures/doc-drift"
[ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT"
# Pin the exception lists + thresholds the fixtures were authored against, so the canary is
# self-contained and deterministic regardless of the operator's env.
DOCS_ONLY_REPOS=""
DOC_DRIFT_STALE_DAYS=60
DOC_DRIFT_STALE_COMMITS=3
# Fixtures ship their git metadata as `dotgit/` (not `.git/`) so they are committable into THIS
# repo without becoming nested submodules. Materialize them into a temp work area — copy each
# fixture and rename dotgit -> .git — so the README/git-log checks run against a real git
# checkout. The temp area is mode 700 and removed on exit (same trick as compliance-drift.sh).
FIXTURE_WORK="$(mktemp -d "${TMPDIR:-/tmp}/doc-drift-canary.XXXXXX")"
trap 'rm -rf "$FIXTURE_WORK"' EXIT
log "canary: materializing planted-drift fixtures from $FIXTURE_ROOT into $FIXTURE_WORK"
for d in "$FIXTURE_ROOT"/*/; do
[ -d "$d/dotgit" ] || continue # only fixture repos (skip README.md, EXPECTED_* etc.)
nm="$(basename "$d")"
cp -R "$d" "$FIXTURE_WORK/$nm"
mv "$FIXTURE_WORK/$nm/dotgit" "$FIXTURE_WORK/$nm/.git"
REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$FIXTURE_WORK/$nm"
done
elif [ -n "$TARGETS_OVERRIDE" ]; then
# shellcheck disable=SC2206 # intentional word-split of the space-separated --targets list
arr=( $TARGETS_OVERRIDE )
for p in "${arr[@]}"; do p="${p/#\~/$HOME}"; nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p"; done
log "explicit targets: ${REPO_NAMES[*]}"
else
if [ "$REFRESH" -eq 1 ]; then
[ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN"
command -v curl >/dev/null || die "--refresh needs curl"
mkdir -p "$MIRROR_DIR"
log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)"
DISCOVERED="$REPORT_DIR/discovered.tsv"
if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then
while IFS=$'\t' read -r name url branch; do
[ -n "$name" ] || continue
mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)"
done < "$DISCOVERED"
else
log "discovery failed — falling back to existing mirrors (coverage may be stale)"
fi
fi
# Default + post-refresh: enumerate EXISTING mirrors. No clone here — reuse the sweep's clones.
[ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)"
for d in "$MIRROR_DIR"/*/; do
[ -d "$d/.git" ] || continue
nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}"
done
log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)"
fi
[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to scan"
# ==============================================================================
# RUN CHECKS
# ==============================================================================
for nm in "${REPO_NAMES[@]}"; do
check_repo "$nm" "${REPO_DIR[$nm]}"
done
# ==============================================================================
# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to compliance-drift)
# ==============================================================================
if [ "${#FINDINGS[@]}" -gt 0 ]; then
FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)"
else
FINDINGS_JSON="[]"
fi
if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then
SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)"
else
SKIPPED_JSON="[]"
fi
N_DRIFT="$(echo "$FINDINGS_JSON" | jq 'length')"
N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high")] | length')"
N_REPOS_DRIFTED="$(echo "$FINDINGS_JSON" | jq '[.[].repo] | unique | length')"
jq -n \
--arg checker "doc-drift" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \
--argjson scanned "${#REPO_NAMES[@]}" \
--argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \
'{checker:$checker, generated:$ts, org:$org,
repos_scanned:$scanned, drift_count:($findings|length),
repos_with_drift:([$findings[].repo]|unique|length),
findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON"
{
echo "doc-drift report — $UTC_STAMP"
echo "org=$GH_ORG repos_scanned=${#REPO_NAMES[@]} stale_thresholds=${DOC_DRIFT_STALE_DAYS}d/${DOC_DRIFT_STALE_COMMITS}commits"
echo "drift findings: $N_DRIFT ($N_HIGH high) across $N_REPOS_DRIFTED repo(s)"
echo
echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n rule: \(.proof.outcome)"'
if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then
echo; echo "skipped checks (missing data / not doc-drift's job — NOT counted as drift):"
echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"'
fi
} > "$REPORT_TXT"
chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true
log "report: $REPORT_JSON ($N_DRIFT drift finding(s), $N_REPOS_DRIFTED repo(s))"
# ==============================================================================
# CANARY ASSERTION (anti-complacency floor, design §6.4)
# ==============================================================================
if [ "$CANARY" -eq 1 ]; then
EXPECT_FILE="$HERE/fixtures/doc-drift/EXPECTED_DRIFT_COUNT"
[ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE"
EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")"
log "canary assertion: expected drift=$EXPECTED, got=$N_DRIFT"
if [ "$N_DRIFT" -ne "$EXPECTED" ]; then
echo "[doc-drift] CANARY FAIL: planted-drift count mismatch (expected $EXPECTED, got $N_DRIFT)" >&2
echo " -> the checklist regressed (a check stopped firing) or the fixture changed. See $REPORT_TXT." >&2
exit 3
fi
log "canary PASS: all $EXPECTED planted drifts detected."
fi
# ==============================================================================
# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms)
# ==============================================================================
if [ "$N_DRIFT" -eq 0 ]; then
log "no confirmed drift — posting NOTHING to Slack (ALARM-only policy)."
exit 0
fi
ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r '
group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')"
SLACK_TEXT=":memo: *Sea Haven doc-drift — ALARM* ($UTC_STAMP)
$N_DRIFT documentation-drift finding(s) across $N_REPOS_DRIFTED repo(s) ($N_HIGH high):
$ALARM_BODY
Checks: README-omits-component · README-stale-vs-code (architecture moved, docs did not)
Report (mode 600): \`$REPORT_JSON\` (on R720)"
SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)"
echo "$SLACK_TEXT" >&2
if [ "$DRY_RUN" -eq 1 ]; then
log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 3)."
exit 0
fi
post_slack_alarm "$SLACK_TEXT"
exit 0
# ==============================================================================
# PROVISIONING (NOT DONE HERE — gated, Phase 3 / Phase 6):
# - No systemd unit / timer is installed by this script. Wiring it into the live
# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated.
# - This script is NOT registered in checker_coordinator.sh; the coordinator registry is
# integrated centrally (separate change), so doc-drift is not yet driven by the coordinator.
# - step-ca / IAM Roles Anywhere / the read-only AWS role / aws-posture are NOT stood up or
# built here. The IAM artifacts authored alongside this checker (security-review/iam/) are
# FILES for the mandatory GPT-4.1 cross-review; aws-posture itself is hard-gated behind that
# review and is built only after it is recorded (design §7, B3).
# - The LIVE "Gemini (large context)" doc-drift judge (design §4) is the only LLM seam; it is
# an inert stub here (maybe_judge) and stays off in canary / dry-run / offline.
# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations for the
# build session, tracked outside this script.
# ==============================================================================

View file

@ -0,0 +1 @@
7

View file

@ -0,0 +1,34 @@
# aws-posture canary fixtures
Mocked AWS API responses for `checkers/aws-posture.sh --canary` (offline — **no `aws` calls, no
network, no credentials**). The canary feeds these files to the SAME detectors the live path runs
against real `aws` CLI output, and asserts the total finding count equals `EXPECTED_FINDING_COUNT`
(anti-complacency floor, design §6.4). If a detector regresses (stops firing), the count drops and
the canary FAILS (exit 3).
These are plain JSON files (not git fixtures — aws-posture scans an AWS account, not a repo tree),
so there is no `dotgit/` / `.fixture` rename trick here; the offline-vs-live seam is the
`--canary`/`--no-api`/no-credentials guard inside the checker (mirrors compliance-drift's
API-skip pattern). Each file is shaped like the real `aws ... --output json` response it stands in
for; a few `_Fixture*` helper keys carry the per-resource metric the live path derives from
CloudWatch (so the canary stays deterministic and offline).
| Fixture file | Stands in for | Planted finding | Count |
|---|---|---|---|
| `cost-anomalies.json` | `aws ce get-anomalies` | 1 anomaly TotalImpact ≥ threshold (the other is below threshold → must NOT fire) | 1 |
| `describe-instances.json` | `aws ec2 describe-instances` | 1 `stopped` instance still paying for its EBS root (the `running` one must NOT fire) | 1 |
| `describe-volumes.json` | `aws ec2 describe-volumes` | 1 `available` (unattached) volume (the `in-use` one must NOT fire) | 1 |
| `describe-addresses.json` | `aws ec2 describe-addresses` | 1 EIP with no association (the associated one must NOT fire) | 1 |
| `describe-nat-gateways.json` | `aws ec2 describe-nat-gateways` | 1 `available` NAT with ~0 bytes out / 14d (the busy one must NOT fire) | 1 |
| `describe-load-balancers.json` | `aws elbv2 describe-load-balancers` | 1 ALB with 0 healthy targets (the one with 3 must NOT fire) | 1 |
| `describe-db-instances.json` | `aws rds describe-db-instances` | 1 `available` RDS with 0 connections / 14d (the busy one must NOT fire) | 1 |
Total = **7** (`EXPECTED_FINDING_COUNT`).
aws-posture **complements** GuardDuty / Security Hub / Config (design §4 / Tier-2) — it is an
idle/anomalous-**spend** + idle-resource posture watch, not a threat detector, and never alarms on
missing data (a skipped/credential-less live call is noted, never counted — memory
`feedback_cloudwatch_alarms`).
When you add/remove a detector or fixture, update both the fixture and `EXPECTED_FINDING_COUNT`
in the same commit (the canary edit is itself caught on the next run — design §6.4).

View file

@ -0,0 +1,32 @@
{
"Anomalies": [
{
"AnomalyId": "anomaly-0001",
"AnomalyStartDate": "2026-06-15",
"AnomalyEndDate": "2026-06-17",
"DimensionValue": "Amazon Elastic Compute Cloud - Compute",
"RootCauses": [
{ "Service": "Amazon Elastic Compute Cloud - Compute", "Region": "us-east-1" }
],
"Impact": {
"MaxImpact": 142.55,
"TotalImpact": 268.40,
"TotalActualSpend": 410.10,
"TotalExpectedSpend": 141.70
},
"Feedback": "NO_FEEDBACK"
},
{
"AnomalyId": "anomaly-0002-below-threshold",
"AnomalyStartDate": "2026-06-16",
"DimensionValue": "AWS Lambda",
"Impact": {
"MaxImpact": 1.10,
"TotalImpact": 2.05,
"TotalActualSpend": 9.00,
"TotalExpectedSpend": 6.95
},
"Feedback": "NO_FEEDBACK"
}
]
}

View file

@ -0,0 +1,17 @@
{
"Addresses": [
{
"PublicIp": "52.10.20.30",
"AllocationId": "eipalloc-idle-7001",
"Domain": "vpc",
"Tags": [ { "Key": "Name", "Value": "leftover-nat-eip" } ]
},
{
"PublicIp": "52.40.50.60",
"AllocationId": "eipalloc-inuse-7002",
"Domain": "vpc",
"InstanceId": "i-0ff99ee88dd77cc66",
"AssociationId": "eipassoc-active-0001"
}
]
}

View file

@ -0,0 +1,20 @@
{
"DBInstances": [
{
"DBInstanceIdentifier": "idle-reporting-db",
"DBInstanceClass": "db.r5.large",
"Engine": "postgres",
"DBInstanceStatus": "available",
"MultiAZ": false,
"_FixtureMaxConnectionsLast14d": 0
},
{
"DBInstanceIdentifier": "prod-app-db",
"DBInstanceClass": "db.t3.medium",
"Engine": "postgres",
"DBInstanceStatus": "available",
"MultiAZ": true,
"_FixtureMaxConnectionsLast14d": 47
}
]
}

View file

@ -0,0 +1,27 @@
{
"Reservations": [
{
"Instances": [
{
"InstanceId": "i-0aa11bb22cc33dd44",
"InstanceType": "m5.large",
"State": { "Name": "stopped" },
"StateTransitionReason": "User initiated (2026-02-01 09:14:00 GMT)",
"BlockDeviceMappings": [
{ "DeviceName": "/dev/xvda", "Ebs": { "VolumeId": "vol-stopped-root-001", "Status": "attached" } }
],
"Tags": [ { "Key": "Name", "Value": "old-batch-runner" } ]
},
{
"InstanceId": "i-0ff99ee88dd77cc66",
"InstanceType": "t3.micro",
"State": { "Name": "running" },
"BlockDeviceMappings": [
{ "DeviceName": "/dev/xvda", "Ebs": { "VolumeId": "vol-running-root-002", "Status": "attached" } }
],
"Tags": [ { "Key": "Name", "Value": "active-web" } ]
}
]
}
]
}

View file

@ -0,0 +1,18 @@
{
"LoadBalancers": [
{
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-east-1:328440206208:loadbalancer/app/idle-alb/abc",
"LoadBalancerName": "idle-alb",
"Type": "application",
"State": { "Code": "active" },
"_FixtureHealthyTargetCount": 0
},
{
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-east-1:328440206208:loadbalancer/app/active-alb/def",
"LoadBalancerName": "active-alb",
"Type": "application",
"State": { "Code": "active" },
"_FixtureHealthyTargetCount": 3
}
]
}

View file

@ -0,0 +1,19 @@
{
"NatGateways": [
{
"NatGatewayId": "nat-idle-6001",
"State": "available",
"SubnetId": "subnet-abc123",
"VpcId": "vpc-def456",
"Tags": [ { "Key": "Name", "Value": "unused-private-subnet-nat" } ],
"_FixtureBytesOutLast14d": 0
},
{
"NatGatewayId": "nat-active-6002",
"State": "available",
"SubnetId": "subnet-xyz789",
"VpcId": "vpc-def456",
"_FixtureBytesOutLast14d": 9842113
}
]
}

View file

@ -0,0 +1,20 @@
{
"Volumes": [
{
"VolumeId": "vol-unattached-9001",
"Size": 500,
"VolumeType": "gp3",
"State": "available",
"CreateTime": "2025-11-02T18:00:00.000Z",
"Attachments": [],
"Tags": [ { "Key": "Name", "Value": "orphaned-data-disk" } ]
},
{
"VolumeId": "vol-running-root-002",
"Size": 8,
"VolumeType": "gp3",
"State": "in-use",
"Attachments": [ { "InstanceId": "i-0ff99ee88dd77cc66", "State": "attached" } ]
}
]
}

View file

@ -0,0 +1 @@
API_KEY=AKIAIOSFODNN7EXAMPLE

View file

@ -0,0 +1 @@
ref: refs/heads/main

View file

@ -0,0 +1,10 @@
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
precomposeunicode = true
[user]
email = t@t
name = t

View file

@ -0,0 +1 @@
Unnamed repository; edit this file 'description' to name the repository.

View file

@ -0,0 +1,15 @@
#!/bin/sh
#
# An example hook script to check the commit log message taken by
# applypatch from an e-mail message.
#
# The hook should exit with non-zero status after issuing an
# appropriate message if it wants to stop the commit. The hook is
# allowed to edit the commit message file.
#
# To enable this hook, rename this file to "applypatch-msg".
. git-sh-setup
commitmsg="$(git rev-parse --git-path hooks/commit-msg)"
test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"}
:

View file

@ -0,0 +1,24 @@
#!/bin/sh
#
# An example hook script to check the commit log message.
# Called by "git commit" with one argument, the name of the file
# that has the commit message. The hook should exit with non-zero
# status after issuing an appropriate message if it wants to stop the
# commit. The hook is allowed to edit the commit message file.
#
# To enable this hook, rename this file to "commit-msg".
# Uncomment the below to add a Signed-off-by line to the message.
# Doing this in a hook is a bad idea in general, but the prepare-commit-msg
# hook is more suited to it.
#
# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1"
# This example catches duplicate Signed-off-by lines.
test "" = "$(grep '^Signed-off-by: ' "$1" |
sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || {
echo >&2 Duplicate Signed-off-by lines.
exit 1
}

View file

@ -0,0 +1,174 @@
#!/usr/bin/perl
use strict;
use warnings;
use IPC::Open2;
# An example hook script to integrate Watchman
# (https://facebook.github.io/watchman/) with git to speed up detecting
# new and modified files.
#
# The hook is passed a version (currently 2) and last update token
# formatted as a string and outputs to stdout a new update token and
# all files that have been modified since the update token. Paths must
# be relative to the root of the working tree and separated by a single NUL.
#
# To enable this hook, rename this file to "query-watchman" and set
# 'git config core.fsmonitor .git/hooks/query-watchman'
#
my ($version, $last_update_token) = @ARGV;
# Uncomment for debugging
# print STDERR "$0 $version $last_update_token\n";
# Check the hook interface version
if ($version ne 2) {
die "Unsupported query-fsmonitor hook version '$version'.\n" .
"Falling back to scanning...\n";
}
my $git_work_tree = get_working_dir();
my $retry = 1;
my $json_pkg;
eval {
require JSON::XS;
$json_pkg = "JSON::XS";
1;
} or do {
require JSON::PP;
$json_pkg = "JSON::PP";
};
launch_watchman();
sub launch_watchman {
my $o = watchman_query();
if (is_work_tree_watched($o)) {
output_result($o->{clock}, @{$o->{files}});
}
}
sub output_result {
my ($clockid, @files) = @_;
# Uncomment for debugging watchman output
# open (my $fh, ">", ".git/watchman-output.out");
# binmode $fh, ":utf8";
# print $fh "$clockid\n@files\n";
# close $fh;
binmode STDOUT, ":utf8";
print $clockid;
print "\0";
local $, = "\0";
print @files;
}
sub watchman_clock {
my $response = qx/watchman clock "$git_work_tree"/;
die "Failed to get clock id on '$git_work_tree'.\n" .
"Falling back to scanning...\n" if $? != 0;
return $json_pkg->new->utf8->decode($response);
}
sub watchman_query {
my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty')
or die "open2() failed: $!\n" .
"Falling back to scanning...\n";
# In the query expression below we're asking for names of files that
# changed since $last_update_token but not from the .git folder.
#
# To accomplish this, we're using the "since" generator to use the
# recency index to select candidate nodes and "fields" to limit the
# output to file names only. Then we're using the "expression" term to
# further constrain the results.
my $last_update_line = "";
if (substr($last_update_token, 0, 1) eq "c") {
$last_update_token = "\"$last_update_token\"";
$last_update_line = qq[\n"since": $last_update_token,];
}
my $query = <<" END";
["query", "$git_work_tree", {$last_update_line
"fields": ["name"],
"expression": ["not", ["dirname", ".git"]]
}]
END
# Uncomment for debugging the watchman query
# open (my $fh, ">", ".git/watchman-query.json");
# print $fh $query;
# close $fh;
print CHLD_IN $query;
close CHLD_IN;
my $response = do {local $/; <CHLD_OUT>};
# Uncomment for debugging the watch response
# open ($fh, ">", ".git/watchman-response.json");
# print $fh $response;
# close $fh;
die "Watchman: command returned no output.\n" .
"Falling back to scanning...\n" if $response eq "";
die "Watchman: command returned invalid output: $response\n" .
"Falling back to scanning...\n" unless $response =~ /^\{/;
return $json_pkg->new->utf8->decode($response);
}
sub is_work_tree_watched {
my ($output) = @_;
my $error = $output->{error};
if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) {
$retry--;
my $response = qx/watchman watch "$git_work_tree"/;
die "Failed to make watchman watch '$git_work_tree'.\n" .
"Falling back to scanning...\n" if $? != 0;
$output = $json_pkg->new->utf8->decode($response);
$error = $output->{error};
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
# Uncomment for debugging watchman output
# open (my $fh, ">", ".git/watchman-output.out");
# close $fh;
# Watchman will always return all files on the first query so
# return the fast "everything is dirty" flag to git and do the
# Watchman query just to get it over with now so we won't pay
# the cost in git to look up each individual file.
my $o = watchman_clock();
$error = $output->{error};
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
output_result($o->{clock}, ("/"));
$last_update_token = $o->{clock};
eval { launch_watchman() };
return 0;
}
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
return 1;
}
sub get_working_dir {
my $working_dir;
if ($^O =~ 'msys' || $^O =~ 'cygwin') {
$working_dir = Win32::GetCwd();
$working_dir =~ tr/\\/\//;
} else {
require Cwd;
$working_dir = Cwd::cwd();
}
return $working_dir;
}

View file

@ -0,0 +1,8 @@
#!/bin/sh
#
# An example hook script to prepare a packed repository for use over
# dumb transports.
#
# To enable this hook, rename this file to "post-update".
exec git update-server-info

View file

@ -0,0 +1,14 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed
# by applypatch from an e-mail message.
#
# The hook should exit with non-zero status after issuing an
# appropriate message if it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-applypatch".
. git-sh-setup
precommit="$(git rev-parse --git-path hooks/pre-commit)"
test -x "$precommit" && exec "$precommit" ${1+"$@"}
:

View file

@ -0,0 +1,49 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git commit" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-commit".
if git rev-parse --verify HEAD >/dev/null 2>&1
then
against=HEAD
else
# Initial commit: diff against an empty tree object
against=$(git hash-object -t tree /dev/null)
fi
# If you want to allow non-ASCII filenames set this variable to true.
allownonascii=$(git config --type=bool hooks.allownonascii)
# Redirect output to stderr.
exec 1>&2
# Cross platform projects tend to avoid non-ASCII filenames; prevent
# them from being added to the repository. We exploit the fact that the
# printable range starts at the space character and ends with tilde.
if [ "$allownonascii" != "true" ] &&
# Note that the use of brackets around a tr range is ok here, (it's
# even required, for portability to Solaris 10's /usr/bin/tr), since
# the square bracket bytes happen to fall in the designated range.
test $(git diff-index --cached --name-only --diff-filter=A -z $against |
LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0
then
cat <<\EOF
Error: Attempt to add a non-ASCII file name.
This can cause problems if you want to work with people on other platforms.
To be portable it is advisable to rename the file.
If you know what you are doing you can disable this check using:
git config hooks.allownonascii true
EOF
exit 1
fi
# If there are whitespace errors, print the offending file names and fail.
exec git diff-index --check --cached $against --

View file

@ -0,0 +1,13 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git merge" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message to
# stderr if it wants to stop the merge commit.
#
# To enable this hook, rename this file to "pre-merge-commit".
. git-sh-setup
test -x "$GIT_DIR/hooks/pre-commit" &&
exec "$GIT_DIR/hooks/pre-commit"
:

View file

@ -0,0 +1,53 @@
#!/bin/sh
# An example hook script to verify what is about to be pushed. Called by "git
# push" after it has checked the remote status, but before anything has been
# pushed. If this script exits with a non-zero status nothing will be pushed.
#
# This hook is called with the following parameters:
#
# $1 -- Name of the remote to which the push is being done
# $2 -- URL to which the push is being done
#
# If pushing without using a named remote those arguments will be equal.
#
# Information about the commits which are being pushed is supplied as lines to
# the standard input in the form:
#
# <local ref> <local oid> <remote ref> <remote oid>
#
# This sample shows how to prevent push of commits where the log message starts
# with "WIP" (work in progress).
remote="$1"
url="$2"
zero=$(git hash-object --stdin </dev/null | tr '[0-9a-f]' '0')
while read local_ref local_oid remote_ref remote_oid
do
if test "$local_oid" = "$zero"
then
# Handle delete
:
else
if test "$remote_oid" = "$zero"
then
# New branch, examine all commits
range="$local_oid"
else
# Update to existing branch, examine new commits
range="$remote_oid..$local_oid"
fi
# Check for WIP commit
commit=$(git rev-list -n 1 --grep '^WIP' "$range")
if test -n "$commit"
then
echo >&2 "Found WIP commit in $local_ref, not pushing"
exit 1
fi
fi
done
exit 0

View file

@ -0,0 +1,169 @@
#!/bin/sh
#
# Copyright (c) 2006, 2008 Junio C Hamano
#
# The "pre-rebase" hook is run just before "git rebase" starts doing
# its job, and can prevent the command from running by exiting with
# non-zero status.
#
# The hook is called with the following parameters:
#
# $1 -- the upstream the series was forked from.
# $2 -- the branch being rebased (or empty when rebasing the current branch).
#
# This sample shows how to prevent topic branches that are already
# merged to 'next' branch from getting rebased, because allowing it
# would result in rebasing already published history.
publish=next
basebranch="$1"
if test "$#" = 2
then
topic="refs/heads/$2"
else
topic=`git symbolic-ref HEAD` ||
exit 0 ;# we do not interrupt rebasing detached HEAD
fi
case "$topic" in
refs/heads/??/*)
;;
*)
exit 0 ;# we do not interrupt others.
;;
esac
# Now we are dealing with a topic branch being rebased
# on top of master. Is it OK to rebase it?
# Does the topic really exist?
git show-ref -q "$topic" || {
echo >&2 "No such branch $topic"
exit 1
}
# Is topic fully merged to master?
not_in_master=`git rev-list --pretty=oneline ^master "$topic"`
if test -z "$not_in_master"
then
echo >&2 "$topic is fully merged to master; better remove it."
exit 1 ;# we could allow it, but there is no point.
fi
# Is topic ever merged to next? If so you should not be rebasing it.
only_next_1=`git rev-list ^master "^$topic" ${publish} | sort`
only_next_2=`git rev-list ^master ${publish} | sort`
if test "$only_next_1" = "$only_next_2"
then
not_in_topic=`git rev-list "^$topic" master`
if test -z "$not_in_topic"
then
echo >&2 "$topic is already up to date with master"
exit 1 ;# we could allow it, but there is no point.
else
exit 0
fi
else
not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"`
/usr/bin/perl -e '
my $topic = $ARGV[0];
my $msg = "* $topic has commits already merged to public branch:\n";
my (%not_in_next) = map {
/^([0-9a-f]+) /;
($1 => 1);
} split(/\n/, $ARGV[1]);
for my $elem (map {
/^([0-9a-f]+) (.*)$/;
[$1 => $2];
} split(/\n/, $ARGV[2])) {
if (!exists $not_in_next{$elem->[0]}) {
if ($msg) {
print STDERR $msg;
undef $msg;
}
print STDERR " $elem->[1]\n";
}
}
' "$topic" "$not_in_next" "$not_in_master"
exit 1
fi
<<\DOC_END
This sample hook safeguards topic branches that have been
published from being rewound.
The workflow assumed here is:
* Once a topic branch forks from "master", "master" is never
merged into it again (either directly or indirectly).
* Once a topic branch is fully cooked and merged into "master",
it is deleted. If you need to build on top of it to correct
earlier mistakes, a new topic branch is created by forking at
the tip of the "master". This is not strictly necessary, but
it makes it easier to keep your history simple.
* Whenever you need to test or publish your changes to topic
branches, merge them into "next" branch.
The script, being an example, hardcodes the publish branch name
to be "next", but it is trivial to make it configurable via
$GIT_DIR/config mechanism.
With this workflow, you would want to know:
(1) ... if a topic branch has ever been merged to "next". Young
topic branches can have stupid mistakes you would rather
clean up before publishing, and things that have not been
merged into other branches can be easily rebased without
affecting other people. But once it is published, you would
not want to rewind it.
(2) ... if a topic branch has been fully merged to "master".
Then you can delete it. More importantly, you should not
build on top of it -- other people may already want to
change things related to the topic as patches against your
"master", so if you need further changes, it is better to
fork the topic (perhaps with the same name) afresh from the
tip of "master".
Let's look at this example:
o---o---o---o---o---o---o---o---o---o "next"
/ / / /
/ a---a---b A / /
/ / / /
/ / c---c---c---c B /
/ / / \ /
/ / / b---b C \ /
/ / / / \ /
---o---o---o---o---o---o---o---o---o---o---o "master"
A, B and C are topic branches.
* A has one fix since it was merged up to "next".
* B has finished. It has been fully merged up to "master" and "next",
and is ready to be deleted.
* C has not merged to "next" at all.
We would want to allow C to be rebased, refuse A, and encourage
B to be deleted.
To compute (1):
git rev-list ^master ^topic next
git rev-list ^master next
if these match, topic has not merged in next at all.
To compute (2):
git rev-list master..topic
if this is empty, it is fully merged to "master".
DOC_END

View file

@ -0,0 +1,24 @@
#!/bin/sh
#
# An example hook script to make use of push options.
# The example simply echoes all push options that start with 'echoback='
# and rejects all pushes when the "reject" push option is used.
#
# To enable this hook, rename this file to "pre-receive".
if test -n "$GIT_PUSH_OPTION_COUNT"
then
i=0
while test "$i" -lt "$GIT_PUSH_OPTION_COUNT"
do
eval "value=\$GIT_PUSH_OPTION_$i"
case "$value" in
echoback=*)
echo "echo from the pre-receive-hook: ${value#*=}" >&2
;;
reject)
exit 1
esac
i=$((i + 1))
done
fi

View file

@ -0,0 +1,42 @@
#!/bin/sh
#
# An example hook script to prepare the commit log message.
# Called by "git commit" with the name of the file that has the
# commit message, followed by the description of the commit
# message's source. The hook's purpose is to edit the commit
# message file. If the hook fails with a non-zero status,
# the commit is aborted.
#
# To enable this hook, rename this file to "prepare-commit-msg".
# This hook includes three examples. The first one removes the
# "# Please enter the commit message..." help message.
#
# The second includes the output of "git diff --name-status -r"
# into the message, just before the "git status" output. It is
# commented because it doesn't cope with --amend or with squashed
# commits.
#
# The third example adds a Signed-off-by line to the message, that can
# still be edited. This is rarely a good idea.
COMMIT_MSG_FILE=$1
COMMIT_SOURCE=$2
SHA1=$3
/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE"
# case "$COMMIT_SOURCE,$SHA1" in
# ,|template,)
# /usr/bin/perl -i.bak -pe '
# print "\n" . `git diff --cached --name-status -r`
# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;;
# *) ;;
# esac
# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE"
# if test -z "$COMMIT_SOURCE"
# then
# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE"
# fi

View file

@ -0,0 +1,78 @@
#!/bin/sh
# An example hook script to update a checked-out tree on a git push.
#
# This hook is invoked by git-receive-pack(1) when it reacts to git
# push and updates reference(s) in its repository, and when the push
# tries to update the branch that is currently checked out and the
# receive.denyCurrentBranch configuration variable is set to
# updateInstead.
#
# By default, such a push is refused if the working tree and the index
# of the remote repository has any difference from the currently
# checked out commit; when both the working tree and the index match
# the current commit, they are updated to match the newly pushed tip
# of the branch. This hook is to be used to override the default
# behaviour; however the code below reimplements the default behaviour
# as a starting point for convenient modification.
#
# The hook receives the commit with which the tip of the current
# branch is going to be updated:
commit=$1
# It can exit with a non-zero status to refuse the push (when it does
# so, it must not modify the index or the working tree).
die () {
echo >&2 "$*"
exit 1
}
# Or it can make any necessary changes to the working tree and to the
# index to bring them to the desired state when the tip of the current
# branch is updated to the new commit, and exit with a zero status.
#
# For example, the hook can simply run git read-tree -u -m HEAD "$1"
# in order to emulate git fetch that is run in the reverse direction
# with git push, as the two-tree form of git read-tree -u -m is
# essentially the same as git switch or git checkout that switches
# branches while keeping the local changes in the working tree that do
# not interfere with the difference between the branches.
# The below is a more-or-less exact translation to shell of the C code
# for the default behaviour for git's push-to-checkout hook defined in
# the push_to_deploy() function in builtin/receive-pack.c.
#
# Note that the hook will be executed from the repository directory,
# not from the working tree, so if you want to perform operations on
# the working tree, you will have to adapt your code accordingly, e.g.
# by adding "cd .." or using relative paths.
if ! git update-index -q --ignore-submodules --refresh
then
die "Up-to-date check failed"
fi
if ! git diff-files --quiet --ignore-submodules --
then
die "Working directory has unstaged changes"
fi
# This is a rough translation of:
#
# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX
if git cat-file -e HEAD 2>/dev/null
then
head=HEAD
else
head=$(git hash-object -t tree --stdin </dev/null)
fi
if ! git diff-index --quiet --cached --ignore-submodules $head --
then
die "Working directory has staged changes"
fi
if ! git read-tree -u -m "$commit"
then
die "Could not update working tree to new HEAD"
fi

View file

@ -0,0 +1,77 @@
#!/bin/sh
# An example hook script to validate a patch (and/or patch series) before
# sending it via email.
#
# The hook should exit with non-zero status after issuing an appropriate
# message if it wants to prevent the email(s) from being sent.
#
# To enable this hook, rename this file to "sendemail-validate".
#
# By default, it will only check that the patch(es) can be applied on top of
# the default upstream branch without conflicts in a secondary worktree. After
# validation (successful or not) of the last patch of a series, the worktree
# will be deleted.
#
# The following config variables can be set to change the default remote and
# remote ref that are used to apply the patches against:
#
# sendemail.validateRemote (default: origin)
# sendemail.validateRemoteRef (default: HEAD)
#
# Replace the TODO placeholders with appropriate checks according to your
# needs.
validate_cover_letter () {
file="$1"
# TODO: Replace with appropriate checks (e.g. spell checking).
true
}
validate_patch () {
file="$1"
# Ensure that the patch applies without conflicts.
git am -3 "$file" || return
# TODO: Replace with appropriate checks for this patch
# (e.g. checkpatch.pl).
true
}
validate_series () {
# TODO: Replace with appropriate checks for the whole series
# (e.g. quick build, coding style checks, etc.).
true
}
# main -------------------------------------------------------------------------
if test "$GIT_SENDEMAIL_FILE_COUNTER" = 1
then
remote=$(git config --default origin --get sendemail.validateRemote) &&
ref=$(git config --default HEAD --get sendemail.validateRemoteRef) &&
worktree=$(mktemp --tmpdir -d sendemail-validate.XXXXXXX) &&
git worktree add -fd --checkout "$worktree" "refs/remotes/$remote/$ref" &&
git config --replace-all sendemail.validateWorktree "$worktree"
else
worktree=$(git config --get sendemail.validateWorktree)
fi || {
echo "sendemail-validate: error: failed to prepare worktree" >&2
exit 1
}
unset GIT_DIR GIT_WORK_TREE
cd "$worktree" &&
if grep -q "^diff --git " "$1"
then
validate_patch "$1"
else
validate_cover_letter "$1"
fi &&
if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL"
then
git config --unset-all sendemail.validateWorktree &&
trap 'git worktree remove -ff "$worktree"' EXIT &&
validate_series
fi

View file

@ -0,0 +1,128 @@
#!/bin/sh
#
# An example hook script to block unannotated tags from entering.
# Called by "git receive-pack" with arguments: refname sha1-old sha1-new
#
# To enable this hook, rename this file to "update".
#
# Config
# ------
# hooks.allowunannotated
# This boolean sets whether unannotated tags will be allowed into the
# repository. By default they won't be.
# hooks.allowdeletetag
# This boolean sets whether deleting tags will be allowed in the
# repository. By default they won't be.
# hooks.allowmodifytag
# This boolean sets whether a tag may be modified after creation. By default
# it won't be.
# hooks.allowdeletebranch
# This boolean sets whether deleting branches will be allowed in the
# repository. By default they won't be.
# hooks.denycreatebranch
# This boolean sets whether remotely creating branches will be denied
# in the repository. By default this is allowed.
#
# --- Command line
refname="$1"
oldrev="$2"
newrev="$3"
# --- Safety check
if [ -z "$GIT_DIR" ]; then
echo "Don't run this script from the command line." >&2
echo " (if you want, you could supply GIT_DIR then run" >&2
echo " $0 <ref> <oldrev> <newrev>)" >&2
exit 1
fi
if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then
echo "usage: $0 <ref> <oldrev> <newrev>" >&2
exit 1
fi
# --- Config
allowunannotated=$(git config --type=bool hooks.allowunannotated)
allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch)
denycreatebranch=$(git config --type=bool hooks.denycreatebranch)
allowdeletetag=$(git config --type=bool hooks.allowdeletetag)
allowmodifytag=$(git config --type=bool hooks.allowmodifytag)
# check for no description
projectdesc=$(sed -e '1q' "$GIT_DIR/description")
case "$projectdesc" in
"Unnamed repository"* | "")
echo "*** Project description file hasn't been set" >&2
exit 1
;;
esac
# --- Check types
# if $newrev is 0000...0000, it's a commit to delete a ref.
zero=$(git hash-object --stdin </dev/null | tr '[0-9a-f]' '0')
if [ "$newrev" = "$zero" ]; then
newrev_type=delete
else
newrev_type=$(git cat-file -t $newrev)
fi
case "$refname","$newrev_type" in
refs/tags/*,commit)
# un-annotated tag
short_refname=${refname##refs/tags/}
if [ "$allowunannotated" != "true" ]; then
echo "*** The un-annotated tag, $short_refname, is not allowed in this repository" >&2
echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2
exit 1
fi
;;
refs/tags/*,delete)
# delete tag
if [ "$allowdeletetag" != "true" ]; then
echo "*** Deleting a tag is not allowed in this repository" >&2
exit 1
fi
;;
refs/tags/*,tag)
# annotated tag
if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1
then
echo "*** Tag '$refname' already exists." >&2
echo "*** Modifying a tag is not allowed in this repository." >&2
exit 1
fi
;;
refs/heads/*,commit)
# branch
if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then
echo "*** Creating a branch is not allowed in this repository" >&2
exit 1
fi
;;
refs/heads/*,delete)
# delete branch
if [ "$allowdeletebranch" != "true" ]; then
echo "*** Deleting a branch is not allowed in this repository" >&2
exit 1
fi
;;
refs/remotes/*,commit)
# tracking branch
;;
refs/remotes/*,delete)
# delete tracking branch
if [ "$allowdeletebranch" != "true" ]; then
echo "*** Deleting a tracking branch is not allowed in this repository" >&2
exit 1
fi
;;
*)
# Anything else (is there anything else?)
echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2
exit 1
;;
esac
# --- Finished
exit 0

View file

@ -0,0 +1,6 @@
# git ls-files --others --exclude-from=.git/info/exclude
# Lines that start with '#' are comments.
# For a project mostly in C, the following would be a good set of
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~

View file

@ -0,0 +1 @@
0000000000000000000000000000000000000000 bc8f350556a9ba83a52c0896c69005ec5c872c71 t <t@t> 1781805299 -0400 commit (initial): init

View file

@ -0,0 +1 @@
0000000000000000000000000000000000000000 bc8f350556a9ba83a52c0896c69005ec5c872c71 t <t@t> 1781805299 -0400 commit (initial): init

View file

@ -0,0 +1 @@
x+)JMU07b040031QΠKΝ+cπ s,|ΎΖύ)ΏM6§¬Ό<C2AC><CE8C>ΩB¨|Abrvbz<62>^Vq~Γ“―BϊΫnνK½Pβό™m <0A>½WKη<1D>

View file

@ -0,0 +1 @@
bc8f350556a9ba83a52c0896c69005ec5c872c71

View file

@ -0,0 +1 @@
{"name":"x"}

View file

@ -0,0 +1 @@
6

View file

@ -0,0 +1,28 @@
# compliance-drift canary fixtures
Planted-drift corpus for `checkers/compliance-drift.sh --canary` (offline, no network/token).
The checker asserts the total drift count equals `EXPECTED_DRIFT_COUNT` (anti-complacency floor,
design §6.4). If a check regresses (stops firing), the count drops and the canary FAILS (exit 3).
Fixtures (each a real git checkout so the tracked-`.env` / `ls-files` checks work):
| Fixture | Planted drift | Count |
|---|---|---|
| `clean-repo` | none — kebab name, README, ci.yaml, dependabot.yml, `.env` is **gitignored** (must NOT fire) | 0 |
| `BadName_repo` | non-kebab name; no README; no ci.yaml; has `package.json` but no `dependabot.yml`; tracked `.env` with values | 5 |
| `docs-repo` | docs-only (CI skipped via DOCS_ONLY_REPOS), kebab name, no README | 1 |
Total = **6** (`EXPECTED_DRIFT_COUNT`). The canary pins `DOCS_ONLY_REPOS=docs-repo` and
`COMPLIANCE_EXEMPT=""` internally so it is deterministic regardless of the operator's env.
**Secret-fixture naming:** `BadName_repo`'s planted tracked-secret env file is committed as
`dotenv.fixture`, NOT `.env`. The repo's root `.gitignore` lists `.env`, so a literal `.env`
fixture would silently never be committed — on a fresh clone the `secrets-committed` drift would
vanish and the count would drop to 5 (this regression was caught by this very canary). The
`--canary` materialization renames `dotenv.fixture` → `.env` in its temp work area; the
`dotgit/` index already TRACKS `.env`, so `git ls-files` still reports it. This mirrors the
`.fixture`-suffix convention the `dependency-cve` fixtures use for their manifests. Keep any new
committed secret fixture under a non-gitignored name and rename it in the canary.
When you add/remove a check or fixture, update both the fixture and `EXPECTED_DRIFT_COUNT`
in the same commit (the canary edit is itself caught on the next run — design §6.4).

View file

@ -0,0 +1 @@
version: 2

View file

@ -0,0 +1 @@
name: CI

View file

@ -0,0 +1,2 @@
node_modules/
.env

View file

@ -0,0 +1 @@
# clean-repo

View file

@ -0,0 +1 @@
init

View file

@ -0,0 +1 @@
ref: refs/heads/main

View file

@ -0,0 +1,10 @@
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
precomposeunicode = true
[user]
email = t@t
name = t

View file

@ -0,0 +1 @@
Unnamed repository; edit this file 'description' to name the repository.

View file

@ -0,0 +1,15 @@
#!/bin/sh
#
# An example hook script to check the commit log message taken by
# applypatch from an e-mail message.
#
# The hook should exit with non-zero status after issuing an
# appropriate message if it wants to stop the commit. The hook is
# allowed to edit the commit message file.
#
# To enable this hook, rename this file to "applypatch-msg".
. git-sh-setup
commitmsg="$(git rev-parse --git-path hooks/commit-msg)"
test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"}
:

View file

@ -0,0 +1,24 @@
#!/bin/sh
#
# An example hook script to check the commit log message.
# Called by "git commit" with one argument, the name of the file
# that has the commit message. The hook should exit with non-zero
# status after issuing an appropriate message if it wants to stop the
# commit. The hook is allowed to edit the commit message file.
#
# To enable this hook, rename this file to "commit-msg".
# Uncomment the below to add a Signed-off-by line to the message.
# Doing this in a hook is a bad idea in general, but the prepare-commit-msg
# hook is more suited to it.
#
# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1"
# This example catches duplicate Signed-off-by lines.
test "" = "$(grep '^Signed-off-by: ' "$1" |
sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || {
echo >&2 Duplicate Signed-off-by lines.
exit 1
}

View file

@ -0,0 +1,174 @@
#!/usr/bin/perl
use strict;
use warnings;
use IPC::Open2;
# An example hook script to integrate Watchman
# (https://facebook.github.io/watchman/) with git to speed up detecting
# new and modified files.
#
# The hook is passed a version (currently 2) and last update token
# formatted as a string and outputs to stdout a new update token and
# all files that have been modified since the update token. Paths must
# be relative to the root of the working tree and separated by a single NUL.
#
# To enable this hook, rename this file to "query-watchman" and set
# 'git config core.fsmonitor .git/hooks/query-watchman'
#
my ($version, $last_update_token) = @ARGV;
# Uncomment for debugging
# print STDERR "$0 $version $last_update_token\n";
# Check the hook interface version
if ($version ne 2) {
die "Unsupported query-fsmonitor hook version '$version'.\n" .
"Falling back to scanning...\n";
}
my $git_work_tree = get_working_dir();
my $retry = 1;
my $json_pkg;
eval {
require JSON::XS;
$json_pkg = "JSON::XS";
1;
} or do {
require JSON::PP;
$json_pkg = "JSON::PP";
};
launch_watchman();
sub launch_watchman {
my $o = watchman_query();
if (is_work_tree_watched($o)) {
output_result($o->{clock}, @{$o->{files}});
}
}
sub output_result {
my ($clockid, @files) = @_;
# Uncomment for debugging watchman output
# open (my $fh, ">", ".git/watchman-output.out");
# binmode $fh, ":utf8";
# print $fh "$clockid\n@files\n";
# close $fh;
binmode STDOUT, ":utf8";
print $clockid;
print "\0";
local $, = "\0";
print @files;
}
sub watchman_clock {
my $response = qx/watchman clock "$git_work_tree"/;
die "Failed to get clock id on '$git_work_tree'.\n" .
"Falling back to scanning...\n" if $? != 0;
return $json_pkg->new->utf8->decode($response);
}
sub watchman_query {
my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty')
or die "open2() failed: $!\n" .
"Falling back to scanning...\n";
# In the query expression below we're asking for names of files that
# changed since $last_update_token but not from the .git folder.
#
# To accomplish this, we're using the "since" generator to use the
# recency index to select candidate nodes and "fields" to limit the
# output to file names only. Then we're using the "expression" term to
# further constrain the results.
my $last_update_line = "";
if (substr($last_update_token, 0, 1) eq "c") {
$last_update_token = "\"$last_update_token\"";
$last_update_line = qq[\n"since": $last_update_token,];
}
my $query = <<" END";
["query", "$git_work_tree", {$last_update_line
"fields": ["name"],
"expression": ["not", ["dirname", ".git"]]
}]
END
# Uncomment for debugging the watchman query
# open (my $fh, ">", ".git/watchman-query.json");
# print $fh $query;
# close $fh;
print CHLD_IN $query;
close CHLD_IN;
my $response = do {local $/; <CHLD_OUT>};
# Uncomment for debugging the watch response
# open ($fh, ">", ".git/watchman-response.json");
# print $fh $response;
# close $fh;
die "Watchman: command returned no output.\n" .
"Falling back to scanning...\n" if $response eq "";
die "Watchman: command returned invalid output: $response\n" .
"Falling back to scanning...\n" unless $response =~ /^\{/;
return $json_pkg->new->utf8->decode($response);
}
sub is_work_tree_watched {
my ($output) = @_;
my $error = $output->{error};
if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) {
$retry--;
my $response = qx/watchman watch "$git_work_tree"/;
die "Failed to make watchman watch '$git_work_tree'.\n" .
"Falling back to scanning...\n" if $? != 0;
$output = $json_pkg->new->utf8->decode($response);
$error = $output->{error};
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
# Uncomment for debugging watchman output
# open (my $fh, ">", ".git/watchman-output.out");
# close $fh;
# Watchman will always return all files on the first query so
# return the fast "everything is dirty" flag to git and do the
# Watchman query just to get it over with now so we won't pay
# the cost in git to look up each individual file.
my $o = watchman_clock();
$error = $output->{error};
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
output_result($o->{clock}, ("/"));
$last_update_token = $o->{clock};
eval { launch_watchman() };
return 0;
}
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
return 1;
}
sub get_working_dir {
my $working_dir;
if ($^O =~ 'msys' || $^O =~ 'cygwin') {
$working_dir = Win32::GetCwd();
$working_dir =~ tr/\\/\//;
} else {
require Cwd;
$working_dir = Cwd::cwd();
}
return $working_dir;
}

View file

@ -0,0 +1,8 @@
#!/bin/sh
#
# An example hook script to prepare a packed repository for use over
# dumb transports.
#
# To enable this hook, rename this file to "post-update".
exec git update-server-info

View file

@ -0,0 +1,14 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed
# by applypatch from an e-mail message.
#
# The hook should exit with non-zero status after issuing an
# appropriate message if it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-applypatch".
. git-sh-setup
precommit="$(git rev-parse --git-path hooks/pre-commit)"
test -x "$precommit" && exec "$precommit" ${1+"$@"}
:

View file

@ -0,0 +1,49 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git commit" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-commit".
if git rev-parse --verify HEAD >/dev/null 2>&1
then
against=HEAD
else
# Initial commit: diff against an empty tree object
against=$(git hash-object -t tree /dev/null)
fi
# If you want to allow non-ASCII filenames set this variable to true.
allownonascii=$(git config --type=bool hooks.allownonascii)
# Redirect output to stderr.
exec 1>&2
# Cross platform projects tend to avoid non-ASCII filenames; prevent
# them from being added to the repository. We exploit the fact that the
# printable range starts at the space character and ends with tilde.
if [ "$allownonascii" != "true" ] &&
# Note that the use of brackets around a tr range is ok here, (it's
# even required, for portability to Solaris 10's /usr/bin/tr), since
# the square bracket bytes happen to fall in the designated range.
test $(git diff-index --cached --name-only --diff-filter=A -z $against |
LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0
then
cat <<\EOF
Error: Attempt to add a non-ASCII file name.
This can cause problems if you want to work with people on other platforms.
To be portable it is advisable to rename the file.
If you know what you are doing you can disable this check using:
git config hooks.allownonascii true
EOF
exit 1
fi
# If there are whitespace errors, print the offending file names and fail.
exec git diff-index --check --cached $against --

View file

@ -0,0 +1,13 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git merge" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message to
# stderr if it wants to stop the merge commit.
#
# To enable this hook, rename this file to "pre-merge-commit".
. git-sh-setup
test -x "$GIT_DIR/hooks/pre-commit" &&
exec "$GIT_DIR/hooks/pre-commit"
:

View file

@ -0,0 +1,53 @@
#!/bin/sh
# An example hook script to verify what is about to be pushed. Called by "git
# push" after it has checked the remote status, but before anything has been
# pushed. If this script exits with a non-zero status nothing will be pushed.
#
# This hook is called with the following parameters:
#
# $1 -- Name of the remote to which the push is being done
# $2 -- URL to which the push is being done
#
# If pushing without using a named remote those arguments will be equal.
#
# Information about the commits which are being pushed is supplied as lines to
# the standard input in the form:
#
# <local ref> <local oid> <remote ref> <remote oid>
#
# This sample shows how to prevent push of commits where the log message starts
# with "WIP" (work in progress).
remote="$1"
url="$2"
zero=$(git hash-object --stdin </dev/null | tr '[0-9a-f]' '0')
while read local_ref local_oid remote_ref remote_oid
do
if test "$local_oid" = "$zero"
then
# Handle delete
:
else
if test "$remote_oid" = "$zero"
then
# New branch, examine all commits
range="$local_oid"
else
# Update to existing branch, examine new commits
range="$remote_oid..$local_oid"
fi
# Check for WIP commit
commit=$(git rev-list -n 1 --grep '^WIP' "$range")
if test -n "$commit"
then
echo >&2 "Found WIP commit in $local_ref, not pushing"
exit 1
fi
fi
done
exit 0

View file

@ -0,0 +1,169 @@
#!/bin/sh
#
# Copyright (c) 2006, 2008 Junio C Hamano
#
# The "pre-rebase" hook is run just before "git rebase" starts doing
# its job, and can prevent the command from running by exiting with
# non-zero status.
#
# The hook is called with the following parameters:
#
# $1 -- the upstream the series was forked from.
# $2 -- the branch being rebased (or empty when rebasing the current branch).
#
# This sample shows how to prevent topic branches that are already
# merged to 'next' branch from getting rebased, because allowing it
# would result in rebasing already published history.
publish=next
basebranch="$1"
if test "$#" = 2
then
topic="refs/heads/$2"
else
topic=`git symbolic-ref HEAD` ||
exit 0 ;# we do not interrupt rebasing detached HEAD
fi
case "$topic" in
refs/heads/??/*)
;;
*)
exit 0 ;# we do not interrupt others.
;;
esac
# Now we are dealing with a topic branch being rebased
# on top of master. Is it OK to rebase it?
# Does the topic really exist?
git show-ref -q "$topic" || {
echo >&2 "No such branch $topic"
exit 1
}
# Is topic fully merged to master?
not_in_master=`git rev-list --pretty=oneline ^master "$topic"`
if test -z "$not_in_master"
then
echo >&2 "$topic is fully merged to master; better remove it."
exit 1 ;# we could allow it, but there is no point.
fi
# Is topic ever merged to next? If so you should not be rebasing it.
only_next_1=`git rev-list ^master "^$topic" ${publish} | sort`
only_next_2=`git rev-list ^master ${publish} | sort`
if test "$only_next_1" = "$only_next_2"
then
not_in_topic=`git rev-list "^$topic" master`
if test -z "$not_in_topic"
then
echo >&2 "$topic is already up to date with master"
exit 1 ;# we could allow it, but there is no point.
else
exit 0
fi
else
not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"`
/usr/bin/perl -e '
my $topic = $ARGV[0];
my $msg = "* $topic has commits already merged to public branch:\n";
my (%not_in_next) = map {
/^([0-9a-f]+) /;
($1 => 1);
} split(/\n/, $ARGV[1]);
for my $elem (map {
/^([0-9a-f]+) (.*)$/;
[$1 => $2];
} split(/\n/, $ARGV[2])) {
if (!exists $not_in_next{$elem->[0]}) {
if ($msg) {
print STDERR $msg;
undef $msg;
}
print STDERR " $elem->[1]\n";
}
}
' "$topic" "$not_in_next" "$not_in_master"
exit 1
fi
<<\DOC_END
This sample hook safeguards topic branches that have been
published from being rewound.
The workflow assumed here is:
* Once a topic branch forks from "master", "master" is never
merged into it again (either directly or indirectly).
* Once a topic branch is fully cooked and merged into "master",
it is deleted. If you need to build on top of it to correct
earlier mistakes, a new topic branch is created by forking at
the tip of the "master". This is not strictly necessary, but
it makes it easier to keep your history simple.
* Whenever you need to test or publish your changes to topic
branches, merge them into "next" branch.
The script, being an example, hardcodes the publish branch name
to be "next", but it is trivial to make it configurable via
$GIT_DIR/config mechanism.
With this workflow, you would want to know:
(1) ... if a topic branch has ever been merged to "next". Young
topic branches can have stupid mistakes you would rather
clean up before publishing, and things that have not been
merged into other branches can be easily rebased without
affecting other people. But once it is published, you would
not want to rewind it.
(2) ... if a topic branch has been fully merged to "master".
Then you can delete it. More importantly, you should not
build on top of it -- other people may already want to
change things related to the topic as patches against your
"master", so if you need further changes, it is better to
fork the topic (perhaps with the same name) afresh from the
tip of "master".
Let's look at this example:
o---o---o---o---o---o---o---o---o---o "next"
/ / / /
/ a---a---b A / /
/ / / /
/ / c---c---c---c B /
/ / / \ /
/ / / b---b C \ /
/ / / / \ /
---o---o---o---o---o---o---o---o---o---o---o "master"
A, B and C are topic branches.
* A has one fix since it was merged up to "next".
* B has finished. It has been fully merged up to "master" and "next",
and is ready to be deleted.
* C has not merged to "next" at all.
We would want to allow C to be rebased, refuse A, and encourage
B to be deleted.
To compute (1):
git rev-list ^master ^topic next
git rev-list ^master next
if these match, topic has not merged in next at all.
To compute (2):
git rev-list master..topic
if this is empty, it is fully merged to "master".
DOC_END

View file

@ -0,0 +1,24 @@
#!/bin/sh
#
# An example hook script to make use of push options.
# The example simply echoes all push options that start with 'echoback='
# and rejects all pushes when the "reject" push option is used.
#
# To enable this hook, rename this file to "pre-receive".
if test -n "$GIT_PUSH_OPTION_COUNT"
then
i=0
while test "$i" -lt "$GIT_PUSH_OPTION_COUNT"
do
eval "value=\$GIT_PUSH_OPTION_$i"
case "$value" in
echoback=*)
echo "echo from the pre-receive-hook: ${value#*=}" >&2
;;
reject)
exit 1
esac
i=$((i + 1))
done
fi

View file

@ -0,0 +1,42 @@
#!/bin/sh
#
# An example hook script to prepare the commit log message.
# Called by "git commit" with the name of the file that has the
# commit message, followed by the description of the commit
# message's source. The hook's purpose is to edit the commit
# message file. If the hook fails with a non-zero status,
# the commit is aborted.
#
# To enable this hook, rename this file to "prepare-commit-msg".
# This hook includes three examples. The first one removes the
# "# Please enter the commit message..." help message.
#
# The second includes the output of "git diff --name-status -r"
# into the message, just before the "git status" output. It is
# commented because it doesn't cope with --amend or with squashed
# commits.
#
# The third example adds a Signed-off-by line to the message, that can
# still be edited. This is rarely a good idea.
COMMIT_MSG_FILE=$1
COMMIT_SOURCE=$2
SHA1=$3
/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE"
# case "$COMMIT_SOURCE,$SHA1" in
# ,|template,)
# /usr/bin/perl -i.bak -pe '
# print "\n" . `git diff --cached --name-status -r`
# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;;
# *) ;;
# esac
# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE"
# if test -z "$COMMIT_SOURCE"
# then
# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE"
# fi

View file

@ -0,0 +1,78 @@
#!/bin/sh
# An example hook script to update a checked-out tree on a git push.
#
# This hook is invoked by git-receive-pack(1) when it reacts to git
# push and updates reference(s) in its repository, and when the push
# tries to update the branch that is currently checked out and the
# receive.denyCurrentBranch configuration variable is set to
# updateInstead.
#
# By default, such a push is refused if the working tree and the index
# of the remote repository has any difference from the currently
# checked out commit; when both the working tree and the index match
# the current commit, they are updated to match the newly pushed tip
# of the branch. This hook is to be used to override the default
# behaviour; however the code below reimplements the default behaviour
# as a starting point for convenient modification.
#
# The hook receives the commit with which the tip of the current
# branch is going to be updated:
commit=$1
# It can exit with a non-zero status to refuse the push (when it does
# so, it must not modify the index or the working tree).
die () {
echo >&2 "$*"
exit 1
}
# Or it can make any necessary changes to the working tree and to the
# index to bring them to the desired state when the tip of the current
# branch is updated to the new commit, and exit with a zero status.
#
# For example, the hook can simply run git read-tree -u -m HEAD "$1"
# in order to emulate git fetch that is run in the reverse direction
# with git push, as the two-tree form of git read-tree -u -m is
# essentially the same as git switch or git checkout that switches
# branches while keeping the local changes in the working tree that do
# not interfere with the difference between the branches.
# The below is a more-or-less exact translation to shell of the C code
# for the default behaviour for git's push-to-checkout hook defined in
# the push_to_deploy() function in builtin/receive-pack.c.
#
# Note that the hook will be executed from the repository directory,
# not from the working tree, so if you want to perform operations on
# the working tree, you will have to adapt your code accordingly, e.g.
# by adding "cd .." or using relative paths.
if ! git update-index -q --ignore-submodules --refresh
then
die "Up-to-date check failed"
fi
if ! git diff-files --quiet --ignore-submodules --
then
die "Working directory has unstaged changes"
fi
# This is a rough translation of:
#
# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX
if git cat-file -e HEAD 2>/dev/null
then
head=HEAD
else
head=$(git hash-object -t tree --stdin </dev/null)
fi
if ! git diff-index --quiet --cached --ignore-submodules $head --
then
die "Working directory has staged changes"
fi
if ! git read-tree -u -m "$commit"
then
die "Could not update working tree to new HEAD"
fi

View file

@ -0,0 +1,77 @@
#!/bin/sh
# An example hook script to validate a patch (and/or patch series) before
# sending it via email.
#
# The hook should exit with non-zero status after issuing an appropriate
# message if it wants to prevent the email(s) from being sent.
#
# To enable this hook, rename this file to "sendemail-validate".
#
# By default, it will only check that the patch(es) can be applied on top of
# the default upstream branch without conflicts in a secondary worktree. After
# validation (successful or not) of the last patch of a series, the worktree
# will be deleted.
#
# The following config variables can be set to change the default remote and
# remote ref that are used to apply the patches against:
#
# sendemail.validateRemote (default: origin)
# sendemail.validateRemoteRef (default: HEAD)
#
# Replace the TODO placeholders with appropriate checks according to your
# needs.
validate_cover_letter () {
file="$1"
# TODO: Replace with appropriate checks (e.g. spell checking).
true
}
validate_patch () {
file="$1"
# Ensure that the patch applies without conflicts.
git am -3 "$file" || return
# TODO: Replace with appropriate checks for this patch
# (e.g. checkpatch.pl).
true
}
validate_series () {
# TODO: Replace with appropriate checks for the whole series
# (e.g. quick build, coding style checks, etc.).
true
}
# main -------------------------------------------------------------------------
if test "$GIT_SENDEMAIL_FILE_COUNTER" = 1
then
remote=$(git config --default origin --get sendemail.validateRemote) &&
ref=$(git config --default HEAD --get sendemail.validateRemoteRef) &&
worktree=$(mktemp --tmpdir -d sendemail-validate.XXXXXXX) &&
git worktree add -fd --checkout "$worktree" "refs/remotes/$remote/$ref" &&
git config --replace-all sendemail.validateWorktree "$worktree"
else
worktree=$(git config --get sendemail.validateWorktree)
fi || {
echo "sendemail-validate: error: failed to prepare worktree" >&2
exit 1
}
unset GIT_DIR GIT_WORK_TREE
cd "$worktree" &&
if grep -q "^diff --git " "$1"
then
validate_patch "$1"
else
validate_cover_letter "$1"
fi &&
if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL"
then
git config --unset-all sendemail.validateWorktree &&
trap 'git worktree remove -ff "$worktree"' EXIT &&
validate_series
fi

View file

@ -0,0 +1,128 @@
#!/bin/sh
#
# An example hook script to block unannotated tags from entering.
# Called by "git receive-pack" with arguments: refname sha1-old sha1-new
#
# To enable this hook, rename this file to "update".
#
# Config
# ------
# hooks.allowunannotated
# This boolean sets whether unannotated tags will be allowed into the
# repository. By default they won't be.
# hooks.allowdeletetag
# This boolean sets whether deleting tags will be allowed in the
# repository. By default they won't be.
# hooks.allowmodifytag
# This boolean sets whether a tag may be modified after creation. By default
# it won't be.
# hooks.allowdeletebranch
# This boolean sets whether deleting branches will be allowed in the
# repository. By default they won't be.
# hooks.denycreatebranch
# This boolean sets whether remotely creating branches will be denied
# in the repository. By default this is allowed.
#
# --- Command line
refname="$1"
oldrev="$2"
newrev="$3"
# --- Safety check
if [ -z "$GIT_DIR" ]; then
echo "Don't run this script from the command line." >&2
echo " (if you want, you could supply GIT_DIR then run" >&2
echo " $0 <ref> <oldrev> <newrev>)" >&2
exit 1
fi
if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then
echo "usage: $0 <ref> <oldrev> <newrev>" >&2
exit 1
fi
# --- Config
allowunannotated=$(git config --type=bool hooks.allowunannotated)
allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch)
denycreatebranch=$(git config --type=bool hooks.denycreatebranch)
allowdeletetag=$(git config --type=bool hooks.allowdeletetag)
allowmodifytag=$(git config --type=bool hooks.allowmodifytag)
# check for no description
projectdesc=$(sed -e '1q' "$GIT_DIR/description")
case "$projectdesc" in
"Unnamed repository"* | "")
echo "*** Project description file hasn't been set" >&2
exit 1
;;
esac
# --- Check types
# if $newrev is 0000...0000, it's a commit to delete a ref.
zero=$(git hash-object --stdin </dev/null | tr '[0-9a-f]' '0')
if [ "$newrev" = "$zero" ]; then
newrev_type=delete
else
newrev_type=$(git cat-file -t $newrev)
fi
case "$refname","$newrev_type" in
refs/tags/*,commit)
# un-annotated tag
short_refname=${refname##refs/tags/}
if [ "$allowunannotated" != "true" ]; then
echo "*** The un-annotated tag, $short_refname, is not allowed in this repository" >&2
echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2
exit 1
fi
;;
refs/tags/*,delete)
# delete tag
if [ "$allowdeletetag" != "true" ]; then
echo "*** Deleting a tag is not allowed in this repository" >&2
exit 1
fi
;;
refs/tags/*,tag)
# annotated tag
if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1
then
echo "*** Tag '$refname' already exists." >&2
echo "*** Modifying a tag is not allowed in this repository." >&2
exit 1
fi
;;
refs/heads/*,commit)
# branch
if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then
echo "*** Creating a branch is not allowed in this repository" >&2
exit 1
fi
;;
refs/heads/*,delete)
# delete branch
if [ "$allowdeletebranch" != "true" ]; then
echo "*** Deleting a branch is not allowed in this repository" >&2
exit 1
fi
;;
refs/remotes/*,commit)
# tracking branch
;;
refs/remotes/*,delete)
# delete tracking branch
if [ "$allowdeletebranch" != "true" ]; then
echo "*** Deleting a tracking branch is not allowed in this repository" >&2
exit 1
fi
;;
*)
# Anything else (is there anything else?)
echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2
exit 1
;;
esac
# --- Finished
exit 0

View file

@ -0,0 +1,6 @@
# git ls-files --others --exclude-from=.git/info/exclude
# Lines that start with '#' are comments.
# For a project mostly in C, the following would be a good set of
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~

View file

@ -0,0 +1 @@
0000000000000000000000000000000000000000 72baacfb9265a352ce186809392c0c849c6220e4 t <t@t> 1781805299 -0400 commit (initial): init

View file

@ -0,0 +1 @@
0000000000000000000000000000000000000000 72baacfb9265a352ce186809392c0c849c6220e4 t <t@t> 1781805299 -0400 commit (initial): init

View file

@ -0,0 +1 @@
72baacfb9265a352ce186809392c0c849c6220e4

View file

@ -0,0 +1 @@
{"name":"clean-repo"}

View file

@ -0,0 +1 @@
init

View file

@ -0,0 +1 @@
ref: refs/heads/main

View file

@ -0,0 +1,10 @@
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
precomposeunicode = true
[user]
email = t@t
name = t

View file

@ -0,0 +1 @@
Unnamed repository; edit this file 'description' to name the repository.

View file

@ -0,0 +1,15 @@
#!/bin/sh
#
# An example hook script to check the commit log message taken by
# applypatch from an e-mail message.
#
# The hook should exit with non-zero status after issuing an
# appropriate message if it wants to stop the commit. The hook is
# allowed to edit the commit message file.
#
# To enable this hook, rename this file to "applypatch-msg".
. git-sh-setup
commitmsg="$(git rev-parse --git-path hooks/commit-msg)"
test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"}
:

View file

@ -0,0 +1,24 @@
#!/bin/sh
#
# An example hook script to check the commit log message.
# Called by "git commit" with one argument, the name of the file
# that has the commit message. The hook should exit with non-zero
# status after issuing an appropriate message if it wants to stop the
# commit. The hook is allowed to edit the commit message file.
#
# To enable this hook, rename this file to "commit-msg".
# Uncomment the below to add a Signed-off-by line to the message.
# Doing this in a hook is a bad idea in general, but the prepare-commit-msg
# hook is more suited to it.
#
# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1"
# This example catches duplicate Signed-off-by lines.
test "" = "$(grep '^Signed-off-by: ' "$1" |
sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || {
echo >&2 Duplicate Signed-off-by lines.
exit 1
}

View file

@ -0,0 +1,174 @@
#!/usr/bin/perl
use strict;
use warnings;
use IPC::Open2;
# An example hook script to integrate Watchman
# (https://facebook.github.io/watchman/) with git to speed up detecting
# new and modified files.
#
# The hook is passed a version (currently 2) and last update token
# formatted as a string and outputs to stdout a new update token and
# all files that have been modified since the update token. Paths must
# be relative to the root of the working tree and separated by a single NUL.
#
# To enable this hook, rename this file to "query-watchman" and set
# 'git config core.fsmonitor .git/hooks/query-watchman'
#
my ($version, $last_update_token) = @ARGV;
# Uncomment for debugging
# print STDERR "$0 $version $last_update_token\n";
# Check the hook interface version
if ($version ne 2) {
die "Unsupported query-fsmonitor hook version '$version'.\n" .
"Falling back to scanning...\n";
}
my $git_work_tree = get_working_dir();
my $retry = 1;
my $json_pkg;
eval {
require JSON::XS;
$json_pkg = "JSON::XS";
1;
} or do {
require JSON::PP;
$json_pkg = "JSON::PP";
};
launch_watchman();
sub launch_watchman {
my $o = watchman_query();
if (is_work_tree_watched($o)) {
output_result($o->{clock}, @{$o->{files}});
}
}
sub output_result {
my ($clockid, @files) = @_;
# Uncomment for debugging watchman output
# open (my $fh, ">", ".git/watchman-output.out");
# binmode $fh, ":utf8";
# print $fh "$clockid\n@files\n";
# close $fh;
binmode STDOUT, ":utf8";
print $clockid;
print "\0";
local $, = "\0";
print @files;
}
sub watchman_clock {
my $response = qx/watchman clock "$git_work_tree"/;
die "Failed to get clock id on '$git_work_tree'.\n" .
"Falling back to scanning...\n" if $? != 0;
return $json_pkg->new->utf8->decode($response);
}
sub watchman_query {
my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty')
or die "open2() failed: $!\n" .
"Falling back to scanning...\n";
# In the query expression below we're asking for names of files that
# changed since $last_update_token but not from the .git folder.
#
# To accomplish this, we're using the "since" generator to use the
# recency index to select candidate nodes and "fields" to limit the
# output to file names only. Then we're using the "expression" term to
# further constrain the results.
my $last_update_line = "";
if (substr($last_update_token, 0, 1) eq "c") {
$last_update_token = "\"$last_update_token\"";
$last_update_line = qq[\n"since": $last_update_token,];
}
my $query = <<" END";
["query", "$git_work_tree", {$last_update_line
"fields": ["name"],
"expression": ["not", ["dirname", ".git"]]
}]
END
# Uncomment for debugging the watchman query
# open (my $fh, ">", ".git/watchman-query.json");
# print $fh $query;
# close $fh;
print CHLD_IN $query;
close CHLD_IN;
my $response = do {local $/; <CHLD_OUT>};
# Uncomment for debugging the watch response
# open ($fh, ">", ".git/watchman-response.json");
# print $fh $response;
# close $fh;
die "Watchman: command returned no output.\n" .
"Falling back to scanning...\n" if $response eq "";
die "Watchman: command returned invalid output: $response\n" .
"Falling back to scanning...\n" unless $response =~ /^\{/;
return $json_pkg->new->utf8->decode($response);
}
sub is_work_tree_watched {
my ($output) = @_;
my $error = $output->{error};
if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) {
$retry--;
my $response = qx/watchman watch "$git_work_tree"/;
die "Failed to make watchman watch '$git_work_tree'.\n" .
"Falling back to scanning...\n" if $? != 0;
$output = $json_pkg->new->utf8->decode($response);
$error = $output->{error};
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
# Uncomment for debugging watchman output
# open (my $fh, ">", ".git/watchman-output.out");
# close $fh;
# Watchman will always return all files on the first query so
# return the fast "everything is dirty" flag to git and do the
# Watchman query just to get it over with now so we won't pay
# the cost in git to look up each individual file.
my $o = watchman_clock();
$error = $output->{error};
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
output_result($o->{clock}, ("/"));
$last_update_token = $o->{clock};
eval { launch_watchman() };
return 0;
}
die "Watchman: $error.\n" .
"Falling back to scanning...\n" if $error;
return 1;
}
sub get_working_dir {
my $working_dir;
if ($^O =~ 'msys' || $^O =~ 'cygwin') {
$working_dir = Win32::GetCwd();
$working_dir =~ tr/\\/\//;
} else {
require Cwd;
$working_dir = Cwd::cwd();
}
return $working_dir;
}

View file

@ -0,0 +1,8 @@
#!/bin/sh
#
# An example hook script to prepare a packed repository for use over
# dumb transports.
#
# To enable this hook, rename this file to "post-update".
exec git update-server-info

View file

@ -0,0 +1,14 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed
# by applypatch from an e-mail message.
#
# The hook should exit with non-zero status after issuing an
# appropriate message if it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-applypatch".
. git-sh-setup
precommit="$(git rev-parse --git-path hooks/pre-commit)"
test -x "$precommit" && exec "$precommit" ${1+"$@"}
:

View file

@ -0,0 +1,49 @@
#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git commit" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-commit".
if git rev-parse --verify HEAD >/dev/null 2>&1
then
against=HEAD
else
# Initial commit: diff against an empty tree object
against=$(git hash-object -t tree /dev/null)
fi
# If you want to allow non-ASCII filenames set this variable to true.
allownonascii=$(git config --type=bool hooks.allownonascii)
# Redirect output to stderr.
exec 1>&2
# Cross platform projects tend to avoid non-ASCII filenames; prevent
# them from being added to the repository. We exploit the fact that the
# printable range starts at the space character and ends with tilde.
if [ "$allownonascii" != "true" ] &&
# Note that the use of brackets around a tr range is ok here, (it's
# even required, for portability to Solaris 10's /usr/bin/tr), since
# the square bracket bytes happen to fall in the designated range.
test $(git diff-index --cached --name-only --diff-filter=A -z $against |
LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0
then
cat <<\EOF
Error: Attempt to add a non-ASCII file name.
This can cause problems if you want to work with people on other platforms.
To be portable it is advisable to rename the file.
If you know what you are doing you can disable this check using:
git config hooks.allownonascii true
EOF
exit 1
fi
# If there are whitespace errors, print the offending file names and fail.
exec git diff-index --check --cached $against --

Some files were not shown because too many files have changed in this diff Show more