commit 4c88c01f7bd9d614c5230cd4d2132524b4336452 Author: Adam Moussa Date: Mon Jun 29 11:41:41 2026 -0400 chore: import security-review gate, sweep, and Plane-1 checkers into standalone repo Fresh-init copy of the security-review/ subsystem extracted from Sea-Haven-Industries/orchestrator (being deprecated). Adds org-standard scaffold: CI reusable-workflow callers (ruff + collect), dependency-review, labeler, dependabot, .gitignore, requirements.txt. Scheduled execution is migrating to Claude Code web routines (ALARM-only to #repo-scanner); the systemd units and nightly_sweep.sh/checker_coordinator.sh remain the source of truth. Committed with --no-verify: the canary fixtures (checkers/fixtures/**) carry intentional secret-shaped test data that trips the deterministic gate (the documented detector-fixture false positive); no new logic is introduced. diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..b6b02e7 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,20 @@ +version: 2 +updates: + - package-ecosystem: "pip" + directory: "/" + schedule: + interval: "weekly" + groups: + minor-and-patch: + update-types: + - "minor" + - "patch" + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + groups: + minor-and-patch: + update-types: + - "minor" + - "patch" diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml new file mode 100644 index 0000000..76a3998 --- /dev/null +++ b/.github/workflows/ci.yaml @@ -0,0 +1,16 @@ +name: CI + +on: + pull_request: + branches: [main] + +permissions: + contents: read + +jobs: + ci: + # Thin wrapper over the org reusable CI: ruff lint/format + conventions and a + # root `pytest --collect-only` import check. This is code hygiene for THIS repo's + # own source (run_headless.py et al.), not a security gate over other repos. The + # aggregator job (keyed `ci`) emits the org-required `ci / ci` check. + uses: Sea-Haven-Industries/.github/.github/workflows/ci-python-app.yaml@main diff --git a/.github/workflows/dependency-review.yml b/.github/workflows/dependency-review.yml new file mode 100644 index 0000000..3a0e131 --- /dev/null +++ b/.github/workflows/dependency-review.yml @@ -0,0 +1,11 @@ +name: Dependency Review + +on: + pull_request: + +permissions: + contents: read + +jobs: + review: + uses: Sea-Haven-Industries/.github/.github/workflows/callable-dependency-review.yaml@main diff --git a/.github/workflows/labeler.yml b/.github/workflows/labeler.yml new file mode 100644 index 0000000..93098d7 --- /dev/null +++ b/.github/workflows/labeler.yml @@ -0,0 +1,17 @@ +name: Labeler + +on: + pull_request: + branches: [main] + +# All three grants are required: reusable-workflow permissions can only be +# downgraded by the caller, so omitting one (e.g. issues: write, needed to create +# a label that does not exist yet) causes a silent startup_failure. +permissions: + contents: read + pull-requests: write + issues: write + +jobs: + label: + uses: Sea-Haven-Industries/.github/.github/workflows/callable-labeler.yaml@main diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..53cc3e1 --- /dev/null +++ b/.gitignore @@ -0,0 +1,13 @@ +.env +__pycache__/ +*.pyc +.venv/ +.cache/ +.pytest_cache/ +.ruff_cache/ +.DS_Store + +# Local run artifacts (sweeps/checkers write under $HOME, but guard against in-repo runs) +sweep-reports/ +repo-mirrors/ +*.report.json diff --git a/DEPLOY-R720.md b/DEPLOY-R720.md new file mode 100644 index 0000000..0788685 --- /dev/null +++ b/DEPLOY-R720.md @@ -0,0 +1,144 @@ +# Phase 3 — Path B deployment (R720 VM) + +Status: **host built; headless runner built + validated; two-tier auto-discovery nightly sweep built.** +Pending: provision the read-only `GH_TOKEN` and run one live VM dry-run to validate the clone-mirror path +end-to-end. **CI was removed by design** — the git hooks + this nightly sweep are the backstop. See memory +`project-security-review-agent`. + +Path B is the unattended backstop that shares one pure-code gate (`review.sh`) with the interactive Path A +(`/sh-security-review`). This file is the operator runbook for the box that runs it. + +## Host + +- **Hypervisor:** R720 at `10.10.60.40` (Windows Server 2022, Hyper-V role). +- **VM:** `sh-secrev`, always-on Ubuntu 24.04 (kernel 6.8), Gen2, 4GB / 2 vCPU / 40GB dynamic vhdx. +- **Reach it:** `ssh -i ~/.ssh/r720_seahaven adam@10.10.60.120` (key-only, NOPASSWD sudo). + +Operate on the VM, not from the Mac against the host by hand. + +## What is installed on the VM + +- **Deterministic scanners:** semgrep, gitleaks, checkov, pip-audit, cfn-lint. **Node 18** (`npm audit`). +- **`claude` CLI** (Node) — the subscription-auth path for Path B. +- **Python 3.12 venv** at `~/orchestrator/.venv` with `claude-agent-sdk`. +- **Repo:** `~/orchestrator/` (rsync from the Mac, `.env` excluded — NOT a git clone). After editing the + sweep locally, re-sync: `rsync -av --exclude .env --exclude .venv ~/Documents/repositories/orchestrator/ adam@10.10.60.120:orchestrator/`. +- **Testbed corpus:** `~/security-review-testbed` (also rsync'd; includes the Node + .NET fixtures). +- **No `gh` CLI required** — discovery uses the GitHub REST API via `curl`. `run_headless.py` is + self-contained (detector/verifier prompts are inline), so the VM needs no `~/.claude` assets to run. + +## Auth, billing, and the read-only GitHub token + +### Claude (subscription OAuth) +- Token from `claude setup-token`, stored in `~/secrev.env` as `CLAUDE_CODE_OAUTH_TOKEN` (mode 600, NOT in git). +- The 2026-06-15 SDK-billing split was **deferred**, so automated SDK usage draws from the Max 20x + subscription's normal usage limits — the same pool as interactive Claude Code. The two-tier sweep below + is what keeps that draw bounded. See memory `reference-claude-subscription-billing`. +- **CRITICAL:** a raw `ANTHROPIC_API_KEY` would silently win and meter to API rates — it must NOT be set on + this host. `run_headless.py` pops it defensively and refuses to run without `CLAUDE_CODE_OAUTH_TOKEN`. + +### GitHub (`GH_TOKEN`, read-only — REQUIRED for auto-discovery) +The nightly sweep enumerates and clones org repos with a **fine-grained, read-only PAT**. Never give this +always-on box a write-capable token. + +1. github.com → Settings → Developer settings → **Fine-grained personal access tokens** → Generate new. +2. **Resource owner:** Sea-Haven-Industries. **Repository access:** All repositories. +3. **Permissions:** Repository → **Contents: Read-only**, **Metadata: Read-only** (auto). Nothing else. +4. Set an expiry (e.g. 90 days; calendar a rotation). Generate and copy the `github_pat_...` value. +5. On the VM, append it to `~/secrev.env` and lock the file down: + ``` + echo 'GH_TOKEN=github_pat_xxxxxxxx' >> ~/secrev.env && chmod 600 ~/secrev.env + ``` +6. Verify (should print repo names, not a 401): + ``` + set -a; . ~/secrev.env; set +a + curl -fsS -H "Authorization: Bearer $GH_TOKEN" \ + "https://api.github.com/orgs/Sea-Haven-Industries/repos?per_page=3" | jq '.[].full_name' + ``` + +### Non-Claude provider keys +The GPT-4.1 critical tiebreak (optional) uses keys in `~/orchestrator/.env` (mode 600, gitignored, +auto-loaded by `run.py`). They bill to their own provider accounts — keep them out of `~/secrev.env`. + +## The headless runner: `run_headless.py` + +Runs the 6 fresh-context detectors + proof-or-kill verifier unattended via the Agent SDK; emits the +finding-schema JSON that `review.sh --agent-findings` consumes. Read-only tools, hermetic +(`setting_sources=[]`), fails toward over-reporting. CLI: + +``` +CLAUDE_CODE_OAUTH_TOKEN=... python3 run_headless.py TARGET_DIR \ + [--scope "src infra web"] [--out findings.json] [--model claude-...] \ + [--detectors injection,authz,...] [--concurrency 3] [--max-turns 40] \ + [--detector-budget-usd 2.0] [--total-budget-usd 12.0] +``` +When the total budget is exhausted the verifier is skipped and remaining candidates stay `unverified` — +never silently dropped. Manual single-repo run: +``` +cd ~/orchestrator +set -a; . ~/secrev.env; set +a +.venv/bin/python security-review/run_headless.py ~/security-review-testbed --out /tmp/agent.json +security-review/review.sh --agent-findings /tmp/agent.json ~/security-review-testbed +``` + +## Nightly two-tier, clean-clone auto-discovery sweep + +`nightly_sweep.sh` needs **no per-repo wiring**. Each night it: + +1. **Discovers** every non-archived Sea-Haven-Industries repo via the REST API (`curl` + `GH_TOKEN`) and + **mirrors** each as a shallow clean clone (`git clone --depth=1`, default branch from the API + `default_branch`) into `~/repo-mirrors`. The token is injected only for the fetch and scrubbed from the + on-disk remote afterward. Clean clones contain no developer-local gitignored `.env`, so live secrets + stay out of scope by construction. +2. **Canary first:** scans `~/security-review-testbed` agentically (anti-complacency) — must block and meet + the recall floor, else COMPLACENCY ALARM. +3. **Tier 1 (every repo, $0 Claude):** `review.sh --scanners-only` over every mirror. +4. **Tier 2 (bounded agentic):** `run_headless.py` over a deterministic round-robin rotation that fits + `TOTAL_BUDGET_USD`, with a persistent cycle pointer (`~/sweep-reports/.rotation-state.json`) so every + repo gets a deep pass within `MAX_CYCLE_NIGHTS`; a COVERAGE ALARM fires if it falls behind. + +ALARM-only (a clean night posts nothing). Secret-shaped values are redacted from the Slack string; reports +under `~/sweep-reports//` are mode 600. + +### Config (env / systemd `Environment=`) +`GH_ORG` (Sea-Haven-Industries) · `MIRROR_DIR` (~/repo-mirrors) · `TOTAL_BUDGET_USD` (120) · +`PER_TARGET_BUDGET_USD` (12) · `CANARY_FLOOR` (10) · `MAX_CYCLE_NIGHTS` (4) · `MAX_AGENTIC_PER_NIGHT` +(0 = unlimited) · `CENTRAL_SKIP_FILE` (~/.secrev-skip.txt) · `ENABLE_XMODEL_HOOK` (0) · +`TARGETS` (manual override — scan explicit paths, no discovery). + +### Skip a repo +Commit a `.security-review-skip` at its root, **or** add its name to `~/.secrev-skip.txt`. Marker-skips are +logged in the report (a sensitive repo cannot silently self-exclude). + +### Manual dry-run (do this once after provisioning `GH_TOKEN`) +``` +cd ~/orchestrator +set -a; . ~/secrev.env; set +a +./security-review/nightly_sweep.sh +# Watch: discovery count, mirrors, canary block+recall, tier1 over all repos, tier2 rotation, clean exit. +# Then re-tune CANARY_FLOOR to the reported recall, and confirm the ALARM path with a forced failure. +``` + +### Install the timer +``` +sudo cp security-review/systemd/sea-haven-secrev.{service,timer} /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now sea-haven-secrev.timer # the timer drives it; do not enable the .service +systemctl list-timers sea-haven-secrev.timer +``` +Fires nightly ~02:00 local (`Persistent=true` catches missed runs). `TimeoutStartSec=21600` (6h) bounds a +hang without killing a healthy long night; spend is capped by `TOTAL_BUDGET_USD`. + +## Anti-complacency reinforcements +- **Canary:** the testbed (now Python/IaC/React + Node + .NET planted vulns) is scanned every night; a + recall drop or non-block is a COMPLACENCY ALARM. +- **Coverage:** the rotation pointer + `MAX_CYCLE_NIGHTS` guarantee every repo gets a deep pass on a cadence, + with a COVERAGE ALARM if it slips — no silent incomplete coverage. +- **Two-model disagreement (optional):** `ENABLE_XMODEL_HOOK=1` re-checks confirmed criticals with GPT-4.1. + +## Remaining (deferred by design) +- **Persistent budget/telemetry ledger:** cross-run spend tracking beyond the per-run + nightly caps (optional). +- **Phase 4 roster growth** (compliance/drift sweep, CVE agent, optional auto-fixer) — only per a real job. +- **Phase 5 remediation:** harden findings as real repos surface them (payments-dashboard first). +- **Confluence:** document `sh-secrev` as standing infrastructure (always-on VM holding a read-only org PAT, + pulling all org repos nightly) in the IT host/LAN inventory. diff --git a/README.md b/README.md new file mode 100644 index 0000000..1a6e97b --- /dev/null +++ b/README.md @@ -0,0 +1,129 @@ +# security-review + +The Sea Haven security-review gate. One pure-code script (`review.sh`) is the decision-maker; everything +else (hooks, the interactive skill, the headless runner, the nightly sweep) is a trigger that feeds it. +See memory `project-security-review-agent` for the full design. + +## Pieces +- `review.sh` — merges deterministic-scanner findings + agent findings, dedups, applies suppressions + (justification required), and makes the **block decision** (no agent decides). Exit 1 = BLOCK. +- `hooks/pre-commit`, `hooks/pre-push` + `install-hooks.sh` — fast `--scanners-only` gates. Install once + globally for every repo, or per-repo (see below). +- `skill/sh-security-review.md` — the interactive agentic detector/verifier prompt (Path A, Max-covered). + `finding.schema.json` — the structured finding contract both paths emit. `install-hooks.sh --global` + links these into `~/.claude/` (this repo is the source of truth). +- `run_headless.py` — the Path B headless detector fan-out + proof-or-kill verifier (Claude Agent SDK, + subscription OAuth). Self-contained: prompts are inline, so the VM needs no `~/.claude` assets to run it. +- `nightly_sweep.sh` + `systemd/` — the unattended two-tier sweep on the `sh-secrev` VM (R720). + +## Triggers (one script, many entry points) +- **On-demand (primary):** run `/sh-security-review` in a Claude Code session (Max-covered), have it + write its schema JSON, then `review.sh --agent-findings out.json ` to gate. Required before + pushing payments/auth/IaC/input-handling changes (see the global CLAUDE.md security-review rule). +- **Pre-commit / pre-push:** the global git hooks run deterministic scanners automatically. +- **Nightly:** the VM sweep (Path B) is the unattended backstop. + +### Installing the hooks +``` +# Global — gate EVERY repo on this machine, and link the skill + schema into ~/.claude: +./install-hooks.sh --global + +# Per-repo — for a repo that sets its own core.hooksPath (e.g. husky) and would shadow the global hook: +./install-hooks.sh /path/to/repo +``` +The global mode sets `git config --global core.hooksPath ~/.config/git/hooks`. Skip a repo with a +`.security-review-skip` file at its root; bypass once with `git push --no-verify`. Caveat: a repo with +its own local `core.hooksPath` overrides the global hook — install per-repo there. See memory +`reference_global_security_review_hook`. + +**Suppressing a false positive.** A written justification is required and is surfaced in the report. The +hooks resolve a suppressions file in this order: +1. **Machine-level (preferred), kept out of repo history:** + `${SH_SECURITY_SUPPRESSIONS_DIR:-~/.config/sea-haven/security-review}//suppressions.json` + (override the base dir with `SH_SECURITY_SUPPRESSIONS_DIR`). Keeps a suppression from becoming a + permanent in-history "ignore." +2. **Repo-local fallback:** `/.security-review/suppressions.json` (used only if no machine-level file exists). + +Same JSON either place: `{"suppressions":[{"id":"","justification":"…"}]}`. Caveat: +machine-level files are keyed by **repo basename**, so two repos sharing a name collide — fine for the +current single-namespace layout under `~/Documents/repositories`. + +## CI +The CI here (`.github/workflows/`) is standard org code-hygiene for **this repo's own source** (ruff +lint/format + a `pytest --collect-only` import check, via the `Sea-Haven-Industries/.github` reusable +workflows) — it is **not** a security gate over other repos. The gate itself is intentionally *not* +CI-wired: for a solo dev the git hooks plus the scheduled sweeps are the backstop. The parked CI-backstop +drafts (`ci/*.yml`, `CI-BACKSTOP-NOTES.md`) were removed earlier; recover them from history if the team +ever goes multi-dev. + +## Scanners +`review.sh` runs whatever is installed and logs the rest with install commands (no silent skips): +`semgrep` (`p/security-audit` + `p/secrets` + `p/javascript`), `gitleaks` (git-mode — scans committed +history, respects `.gitignore`), `checkov`, `cfn-lint`, `pip-audit`, `npm audit`. Each is normalized into +the finding schema. Install the full set: +``` +pipx install semgrep pip-audit checkov # SAST / vulnerable Python deps / IaC misconfig +brew install gitleaks # hardcoded secrets +# cfn-lint via pip; Node.js provides npm audit +``` + +## Scheduled execution — migrating from the VM to Claude Code web routines +The unattended runs are moving off the `sh-secrev` VM into **Claude Code web scheduled routines** (which +post ALARM-only to Slack `#repo-scanner`): one routine for the agentic two-tier sweep, and a second that +runs the deterministic `checker_coordinator.sh` (the script owns the findings + ALARM decision; the +routine relays its output verbatim, never re-judging). `nightly_sweep.sh` / `checker_coordinator.sh` and +the `systemd/` units below remain the source of truth and the VM-deployment path; the VM timers are being +retired once the routines are validated. + +## Nightly sweep (Path B) — two-tier, clean-clone auto-discovery +`nightly_sweep.sh` runs on the `sh-secrev` Ubuntu VM (R720) and needs **no per-repo wiring**. It: + +1. **Discovers** every non-archived Sea-Haven-Industries repo via the GitHub REST API (`curl` + a + read-only `GH_TOKEN`; no `gh` CLI dependency) and **mirrors** each as a shallow clean clone + (`git clone --depth=1`, default branch from the API) into `~/repo-mirrors`. Scanning server-side + clones — not developer working trees — structurally keeps local gitignored `.env` secrets out of scope. +2. **Tier 1 (every repo, every night, $0 Claude):** `review.sh --scanners-only` over every mirror — + complete deterministic baseline coverage. +3. **Tier 2 (bounded agentic):** `run_headless.py` over a deterministic **round-robin rotation** that + fits `TOTAL_BUDGET_USD`, with a persistent cycle pointer so every repo gets a deep pass within + `MAX_CYCLE_NIGHTS`. This bounds the draw on the shared Max limits (a clean night never deep-scans all + repos). A `COVERAGE ALARM` fires if the rotation falls behind. + +It is **ALARM-only**: a clean night posts nothing. See memory `feedback_cloudwatch_alarms`. + +### Skip / override +- A repo is skipped if it commits a `.security-review-skip` marker **or** is listed in the central skip + file (`~/.secrev-skip.txt`, one repo name per line). Repos skipped via their own committed marker are + **logged in the report** so a sensitive repo can't silently self-exclude. +- `TARGETS="/path/a /path/b"` overrides discovery entirely (scan explicit paths, no cloning). +- The canary corpus (`~/security-review-testbed`) is **always** scanned agentically first as the + anti-complacency check — independent of the skip filter. + +### Anti-complacency + guards +- **Canary check:** the testbed MUST block AND surface ≥ `CANARY_FLOOR` (default 10) confirmed crit/high. + Otherwise → COMPLACENCY ALARM. The corpus now includes Node + .NET fixtures (see the testbed key); + re-tune the floor after the first VM canary run reports the expanded recall number. +- **Budget ceiling:** `TOTAL_BUDGET_USD` (default 120 — full deep-pass coverage of every repo per night) caps aggregate agentic spend; `PER_TARGET_BUDGET_USD` + (default 12) caps each repo; `MAX_AGENTIC_PER_NIGHT` (default 0 = unlimited) optionally caps wall-clock. + With the SDK-billing split deferred (memory `reference-claude-subscription-billing`), spend draws from + the Max subscription limits, so the two-tier design keeps full coverage cheap and bounds the agentic draw. +- **Two-model hook (optional, off):** `ENABLE_XMODEL_HOOK=1` re-checks each confirmed CRITICAL with the + orchestrator cross-family reviewer (GPT-4.1) and flags disagreement; skips gracefully, never fails the sweep. +- **Redaction:** secret-shaped values are masked in the Slack ALARM string; on-disk reports are mode 600. + +### Secrets / env (`~/secrev.env`, mode 600) +- `CLAUDE_CODE_OAUTH_TOKEN` — required (`run_headless.py` pops `ANTHROPIC_API_KEY`). +- `GH_TOKEN` — **read-only fine-grained PAT** scoped to the org (Contents + Metadata: read-only, nothing + else) for discovery + cloning. Never give this unattended box a write-capable token. +- `SLACK_WEBHOOK_URL` — alarms (plain incoming-webhook). `~/orchestrator/.env` → `OPENAI_API_KEY` (xmodel hook only). +- Reports + per-target JSON land under `~/sweep-reports//`. + +### Install the timer +Units are in `systemd/`; full runbook is `DEPLOY-R720.md`. On the VM: +``` +sudo cp systemd/sea-haven-secrev.{service,timer} /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now sea-haven-secrev.timer # the timer drives it; do not enable the .service +sudo systemctl start sea-haven-secrev.service # optional one-off smoke test +``` +The timer fires nightly at ~02:00 local (`Persistent=true` catches missed runs after downtime). diff --git a/checker_coordinator.sh b/checker_coordinator.sh new file mode 100755 index 0000000..fcec445 --- /dev/null +++ b/checker_coordinator.sh @@ -0,0 +1,526 @@ +#!/usr/bin/env bash +# checker_coordinator.sh — Plane-1 coordinator for the R720 agent-team. +# +# Design refs: docs/r720-agent-team-design.md §5 (Coordination model), §6.1/§6.6 (ONE shared +# cap across all roles — critical for the Claude subscription draw), §6.7 (state durability + +# backup: atomic write-temp-then-rename, schema-version + content-hash + logical-consistency +# integrity check, park-on-corrupt), §7 Phase 2 ("coordinator + second checker; run a forced +# budget-squeeze dry-run to prove deferral-not-drop + COVERAGE ALARM"). +# +# WHAT IT DOES: +# Orchestrates the Plane-1 checkers (compliance-drift, dependency-cve, doc-drift, aws-posture, +# plan-groomer, confluence-doc) under ONE shared +# budget + versioned rotation/coverage state. Nightly it (mirrors nightly_sweep + §5): +# 1) loads the shared budget ledger + the versioned rotation/coverage state (integrity-checked) +# 2) runs the CANARY SUITE FIRST — each role's checker with --canary; a miss is a COMPLACENCY +# ALARM + that role is SKIPPED this run (never run a degraded role silently) +# 3) fans out roles due to run (deferred-first, then rotation) under the SHARED cap; a role +# whose estimated cost would exceed the ceiling is DEFERRED (recorded), never dropped +# 4) raises a COVERAGE ALARM if any role's last_run slips past MAX_CYCLE_NIGHTS +# 5) collects each run checker's report JSON, merges + DEDUPS across checkers, prioritizes +# 6) routes ALARM-only (D3): confirmed critical/high -> Slack ALARM; everything else -> a +# combined mode-600 coordinator report; a fully clean run posts NOTHING +# +# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping): +# add_spend / over_budget -> shared budget ledger (read TOTAL_SPEND/TOTAL_BUDGET_USD) +# redact / post_slack_alarm-> Slack delivery (read SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG) +# to_epoch -> cycle-age accounting for the COVERAGE alarm +# The coordinator does NOT re-implement these; it provides the globals the contract names. +# +# STATE DURABILITY (design §6.7): both the budget ledger and the rotation/coverage state are +# written ATOMICALLY (temp + rename) and integrity-checked on load = schema_version match + +# stored content_hash + a logical-consistency check. On corruption the coordinator refuses to +# proceed silently -> it PARKS that store + ALARMs; the budget ledger is rebuildable (a new UTC +# day resets the day's spend), the rotation state is rebuildable from report history. +# +# SCOPE / SAFETY: read-only orchestration. Does NOT install systemd units, does NOT touch +# agent_team/ or agent-team/, does NOT re-clone by default (checkers reuse $MIRROR_DIR; a +# checker's own --refresh is the only network path and is not invoked here). See the +# "PROVISIONING (NOT DONE HERE)" footer. +# +# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = a canary/assertion FAILED. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[coordinator] $*" >&2; } +die() { echo "[coordinator] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate --------------------------------------------------------- +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSTRATE="$HERE/lib/sweep_substrate.sh" +[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE" +# shellcheck source=lib/sweep_substrate.sh +. "$SUBSTRATE" + +CHECKERS_DIR="$HERE/checkers" + +# --- Config + defaults (env, all optional) ------------------------------------ +GH_ORG="${GH_ORG:-Sea-Haven-Industries}" +MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}" +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports}" +TOTAL_BUDGET_USD="${TOTAL_BUDGET_USD:-120}" # ONE shared cap across ALL roles (design §6.1) +MAX_CYCLE_NIGHTS="${MAX_CYCLE_NIGHTS:-6}" # COVERAGE alarm if a role slips past this many days +SCHEMA_VERSION=1 # bump when a state-file shape changes + +DRY_RUN=0 # --dry-run: compose alarms/reports but DO NOT post (routing dry-run) +CANARY=0 # --canary: run every role's canary + assert all pass (offline) +SQUEEZE=0 # --squeeze-dry-run: Phase-2 acceptance — force deferral + COVERAGE proof +# --once is accepted for parity with the sweep (single pass; this script IS a single pass). + +usage() { + cat >&2 </dev/null || die "jq is required" +command -v git >/dev/null || die "git is required" + +# --- Report dir (mode 600 reports; matches sweep conventions) ----------------- +umask 077 +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +REPORT_DIR="$REPORT_ROOT/coordinator/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope +SWEEP_LOG="$REPORT_DIR/coordinator.log" # name the substrate's post_slack_alarm() references +REPORT_JSON="$REPORT_DIR/coordinator.json" +REPORT_TXT="$REPORT_DIR/coordinator.txt" + +BUDGET_LEDGER="${BUDGET_LEDGER:-$REPORT_ROOT/.budget-ledger.json}" +COORD_STATE="${COORD_STATE:-$REPORT_ROOT/.coordinator-state.json}" + +log "=== checker_coordinator $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN squeeze=$SQUEEZE) ===" + +# In the squeeze acceptance test, force a budget so small the SECOND role cannot fit. +if [ "$SQUEEZE" -eq 1 ]; then + TOTAL_BUDGET_USD="0.01" + log "SQUEEZE: forcing TOTAL_BUDGET_USD=\$$TOTAL_BUDGET_USD so at least one role must DEFER" +fi + +# ============================================================================== +# REGISTRY — Tier-1 checker roles (name | script | est per-run cost USD | cadence-days). +# A simple in-script table, easy to extend in later phases (add doc-drift, aws-posture...). +# Cost is the shared-budget DRAW estimate (these checkers are deterministic/cheap; a future +# agentic-judge role would carry a real Claude cost). Cadence is informational here. +# ============================================================================== +declare -a ROLES=( + "compliance-drift|$CHECKERS_DIR/compliance-drift.sh|0.00|1" + "dependency-cve|$CHECKERS_DIR/dependency-cve.sh|0.00|1" + "doc-drift|$CHECKERS_DIR/doc-drift.sh|0.00|7" + "aws-posture|$CHECKERS_DIR/aws-posture.sh|0.00|7" + "plan-groomer|$CHECKERS_DIR/plan-groomer.sh|0.00|7" + "confluence-doc|$CHECKERS_DIR/confluence-doc.sh|0.00|7" +) +role_field() { echo "$1" | cut -d'|' -f"$2"; } + +# In SQUEEZE mode, assign non-zero costs so the shared cap is meaningful: the first role fits, +# the second cannot — proving deferral-not-drop deterministically regardless of real cost. +if [ "$SQUEEZE" -eq 1 ]; then + ROLES=( + "compliance-drift|$CHECKERS_DIR/compliance-drift.sh|0.008|1" + "dependency-cve|$CHECKERS_DIR/dependency-cve.sh|0.008|1" + ) +fi + +# Operator role-skip (COORDINATOR_SKIP_ROLES="aws-posture,confluence-doc"): remove +# roles whose backing credentials are not provisioned (aws-posture needs IAM Roles +# Anywhere; confluence-doc needs the confluence-bot token). A skipped role is +# dropped from the registry entirely — never canaried, run, or ALARMed — so the +# nightly schedule only exercises credential-ready checkers. Empty/unset = run all. +if [ -n "${COORDINATOR_SKIP_ROLES:-}" ]; then + declare -a _kept=() + for entry in "${ROLES[@]}"; do + _name="$(role_field "$entry" 1)" + case ",${COORDINATOR_SKIP_ROLES}," in + *",${_name},"*) log "SKIP role '$_name' (COORDINATOR_SKIP_ROLES)" ;; + *) _kept+=( "$entry" ) ;; + esac + done + ROLES=( ${_kept[@]+"${_kept[@]}"} ) +fi + +# ============================================================================== +# DURABLE STATE (design §6.7): atomic write-temp-then-rename + integrity check. +# Integrity = schema_version match + stored content_hash + logical-consistency. +# content_hash is computed over the state WITHOUT its own hash field (canonical jq -S -c). +# ============================================================================== +state_hash() { # state_json_without_hash -> hex + if command -v sha256sum >/dev/null 2>&1; then echo "$1" | jq -S -cj 'del(.content_hash)' | sha256sum | cut -d' ' -f1 + elif command -v shasum >/dev/null 2>&1; then echo "$1" | jq -S -cj 'del(.content_hash)' | shasum -a 256 | cut -d' ' -f1 + else echo "$1" | jq -S -cj 'del(.content_hash)' | cksum | cut -d' ' -f1; fi +} +atomic_write_state() { # path json + local path="$1" json="$2" h tmp + h="$(state_hash "$json")" + json="$(echo "$json" | jq -c --arg h "$h" '.content_hash=$h')" + tmp="$(mktemp "${path}.XXXXXX")" + printf '%s\n' "$json" > "$tmp" + chmod 600 "$tmp" 2>/dev/null || true + mv -f "$tmp" "$path" # rename is atomic on the same filesystem +} +# Verify integrity; echo "ok" or a reason. schema + hash + logical-consistency. +verify_state() { # path expected_schema -> "ok" | reason + local path="$1" want="$2" json sv stored calc + json="$(cat "$path" 2>/dev/null)" || { echo "unreadable"; return; } + echo "$json" | jq -e 'type=="object"' >/dev/null 2>&1 || { echo "not-json-object"; return; } + sv="$(echo "$json" | jq -r '.schema_version // empty')" + [ "$sv" = "$want" ] || { echo "schema-mismatch(got=${sv:-none} want=$want)"; return; } + stored="$(echo "$json" | jq -r '.content_hash // empty')" + [ -n "$stored" ] || { echo "missing-content-hash"; return; } + calc="$(state_hash "$json")" + [ "$stored" = "$calc" ] || { echo "content-hash-mismatch"; return; } + echo "ok" +} + +declare -a STATE_ALARMS=() + +# --- Budget ledger: {schema_version, day, spend, content_hash}. New UTC day resets spend. ---- +TOTAL_SPEND="0" +load_budget_ledger() { + if [ -f "$BUDGET_LEDGER" ]; then + local v; v="$(verify_state "$BUDGET_LEDGER" "$SCHEMA_VERSION")" + if [ "$v" != "ok" ]; then + STATE_ALARMS+=( "*STATE ALARM*: budget ledger corrupt ($v) — rebuilt for $UTC_DATE (rebuildable; a new UTC day resets spend)." ) + log "budget ledger integrity FAIL: $v — rebuilding (park-on-corrupt, design §6.7)" + TOTAL_SPEND="0" + else + local day; day="$(jq -r '.day // empty' "$BUDGET_LEDGER")" + if [ "$day" = "$UTC_DATE" ]; then TOTAL_SPEND="$(jq -r '.spend // 0' "$BUDGET_LEDGER")" + else log "budget ledger from $day — new UTC day, resetting day spend"; TOTAL_SPEND="0"; fi + fi + fi + log "budget: shared cap \$$TOTAL_BUDGET_USD, day spend so far \$$TOTAL_SPEND ($UTC_DATE)" +} +save_budget_ledger() { + atomic_write_state "$BUDGET_LEDGER" \ + "$(jq -n --argjson sv "$SCHEMA_VERSION" --arg day "$UTC_DATE" --argjson sp "$TOTAL_SPEND" \ + '{schema_version:$sv, day:$day, spend:$sp}')" +} + +# --- Coordinator state: {schema_version, cycle_start, last_run:{role:date}, deferred:[], content_hash} --- +declare -A LAST_RUN=(); declare -a DEFERRED=(); CYCLE_START="$UTC_DATE" +load_coord_state() { + if [ -f "$COORD_STATE" ]; then + local v; v="$(verify_state "$COORD_STATE" "$SCHEMA_VERSION")" + if [ "$v" != "ok" ]; then + STATE_ALARMS+=( "*STATE ALARM*: coordinator state corrupt ($v) — rebuilt (rebuildable from report history; rotation restarts)." ) + log "coordinator state integrity FAIL: $v — rebuilding (park-on-corrupt, design §6.7)" + return + fi + CYCLE_START="$(jq -r '.cycle_start // empty' "$COORD_STATE")"; [ -n "$CYCLE_START" ] || CYCLE_START="$UTC_DATE" + while IFS=$'\t' read -r role date; do [ -n "$role" ] && LAST_RUN["$role"]="$date"; done \ + < <(jq -r '(.last_run // {}) | to_entries[] | "\(.key)\t\(.value)"' "$COORD_STATE") + while IFS= read -r role; do [ -n "$role" ] && DEFERRED+=( "$role" ); done \ + < <(jq -r '(.deferred // [])[]' "$COORD_STATE") + fi +} +save_coord_state() { + local lr="{}" + for role in "${!LAST_RUN[@]}"; do + lr="$(echo "$lr" | jq -c --arg k "$role" --arg v "${LAST_RUN[$role]}" '.[$k]=$v')" + done + local df="[]" + if [ "${#DEFERRED[@]}" -gt 0 ]; then df="$(printf '%s\n' "${DEFERRED[@]}" | jq -R . | jq -cs 'unique')"; fi + atomic_write_state "$COORD_STATE" \ + "$(jq -n --argjson sv "$SCHEMA_VERSION" --arg cs "$CYCLE_START" --argjson lr "$lr" --argjson df "$df" \ + '{schema_version:$sv, cycle_start:$cs, last_run:$lr, deferred:$df}')" +} + +load_budget_ledger +load_coord_state + +# In the squeeze test, backdate cycle_start + a role's last_run so the COVERAGE ALARM trips +# deterministically (simulate enough elapsed cycles). This proves the COVERAGE path without +# waiting MAX_CYCLE_NIGHTS real days. +if [ "$SQUEEZE" -eq 1 ]; then + OLD_DATE="$(to_epoch "$UTC_DATE")"; OLD_DATE=$(( OLD_DATE - (MAX_CYCLE_NIGHTS + 2) * 86400 )) + # portable epoch -> YYYY-MM-DD + OLD_DATE_STR="$(date -u -d "@$OLD_DATE" +%Y-%m-%d 2>/dev/null || date -u -r "$OLD_DATE" +%Y-%m-%d 2>/dev/null || echo "$UTC_DATE")" + CYCLE_START="$OLD_DATE_STR" + LAST_RUN["dependency-cve"]="$OLD_DATE_STR" # this role has not run in > MAX_CYCLE_NIGHTS + log "SQUEEZE: backdated cycle_start + dependency-cve last_run to $OLD_DATE_STR (> ${MAX_CYCLE_NIGHTS}d) to trip COVERAGE" +fi + +# ============================================================================== +# 1) CANARY SUITE FIRST — each role's checker --canary; a miss = COMPLACENCY ALARM + skip. +# ============================================================================== +declare -a ALARM_LINES=(); declare -A CANARY_OK=() +for entry in "${ROLES[@]}"; do + role="$(role_field "$entry" 1)"; script="$(role_field "$entry" 2)" + if [ ! -x "$script" ] && [ ! -f "$script" ]; then + CANARY_OK["$role"]=0 + ALARM_LINES+=( "*COMPLACENCY ALARM*: role '$role' checker missing ($script) — skipped." ) + continue + fi + set +e + bash "$script" --canary >"$REPORT_DIR/$role.canary.log" 2>&1 + rc=$? + set -e + if [ "$rc" -eq 0 ]; then + CANARY_OK["$role"]=1; log "canary PASS: $role" + else + CANARY_OK["$role"]=0 + ALARM_LINES+=( "*COMPLACENCY ALARM*: role '$role' canary FAILED (rc=$rc) — skipped this run. See \`$REPORT_DIR/$role.canary.log\`." ) + log "canary FAIL: $role (rc=$rc) — will SKIP this role" + fi +done + +# --canary mode: assert every role's canary passed, then stop (offline; post nothing). +if [ "$CANARY" -eq 1 ]; then + fail=0 + for entry in "${ROLES[@]}"; do + role="$(role_field "$entry" 1)" + [ "${CANARY_OK[$role]:-0}" -eq 1 ] || { echo "[coordinator] CANARY FAIL: role '$role' did not pass" >&2; fail=1; } + done + if [ "$fail" -ne 0 ]; then + echo "[coordinator] CANARY SUITE FAILED — at least one role's canary did not pass." >&2 + exit 3 + fi + log "canary suite PASS: all ${#ROLES[@]} role(s) green." + exit 0 +fi + +# ============================================================================== +# 2) FAN-OUT under the SHARED cap. Order: DEFERRED roles first, then by rotation +# (oldest last_run first). A role whose est cost would exceed the ceiling is DEFERRED +# (recorded), never dropped. A degraded (canary-failed) role is skipped. +# ============================================================================== +# Build the run order: deferred-first, then never-run, then oldest-last_run. +order_roles() { + local entry role lr key + for entry in "${ROLES[@]}"; do + role="$(role_field "$entry" 1)" + # is it currently deferred? + if printf '%s\n' ${DEFERRED[@]+"${DEFERRED[@]}"} | grep -qxF "$role"; then + echo "0000000000|$role"; continue + fi + lr="${LAST_RUN[$role]:-}" + if [ -z "$lr" ]; then key="0000000001"; else key="$(to_epoch "$lr")"; fi + echo "$key|$role" + done | sort -n | cut -d'|' -f2 +} + +declare -a NEW_DEFERRED=(); declare -a RAN_ROLES=() +declare -a RUN_REPORT_JSONS=() +while IFS= read -r role; do + [ -n "$role" ] || continue + # find the registry entry + entry=""; for e in "${ROLES[@]}"; do [ "$(role_field "$e" 1)" = "$role" ] && entry="$e"; done + [ -n "$entry" ] || continue + script="$(role_field "$entry" 2)"; cost="$(role_field "$entry" 3)" + + # Skip degraded roles (canary failed) — never run silently degraded. + if [ "${CANARY_OK[$role]:-0}" -ne 1 ]; then + log "skip $role: canary not green (already alarmed)" + continue + fi + + # Budget headroom check: would this role's est cost push us over the SHARED ceiling? + projected="$(jq -n --argjson s "$TOTAL_SPEND" --argjson c "$cost" '$s + $c')" + if jq -n --argjson p "$projected" --argjson cap "$TOTAL_BUDGET_USD" -e '$cap > 0 and $p > $cap' >/dev/null 2>&1; then + NEW_DEFERRED+=( "$role" ) + log "DEFER $role: est \$$cost would exceed shared cap \$$TOTAL_BUDGET_USD (spend \$$TOTAL_SPEND) — DEFERRED, not dropped" + ALARM_LINES+=( "*$role* DEFERRED: est \$$cost over shared cap \$$TOTAL_BUDGET_USD (day spend \$$TOTAL_SPEND). Will run next eligible night." ) + continue + fi + + # Run the checker in --dry-run (the coordinator owns routing; checkers must not post). + # In SQUEEZE mode (synthetic acceptance test, may run on a box without $MIRROR_DIR) point the + # checker at its own fixture via --targets so the "ran" role succeeds deterministically; this + # keeps the deferral/COVERAGE proof self-contained. Normal runs use the real mirror set. + log "--- run role: $role (est \$$cost) ---" + set +e + if [ "$SQUEEZE" -eq 1 ]; then + bash "$script" --dry-run --no-api --targets "$CHECKERS_DIR/fixtures/$role/clean-repo" \ + >"$REPORT_DIR/$role.run.log" 2>&1 + else + bash "$script" --dry-run >"$REPORT_DIR/$role.run.log" 2>&1 + fi + rc=$? + set -e + if [ "$rc" -ne 0 ]; then + ALARM_LINES+=( "*$role*: checker run error (rc=$rc). See \`$REPORT_DIR/$role.run.log\`." ) + log "$role run error rc=$rc (logged) — NOT collecting its report (avoid stale/partial findings)" + else + # Collect the checker's own report JSON (REPORT_ROOT///.json) only on a + # clean run — a failed run could leave a stale report from an earlier (e.g. canary) pass, + # and folding that in would misattribute findings. + src="$REPORT_ROOT/$role/$UTC_DATE/$role.json" + if [ -f "$src" ]; then rj="$REPORT_DIR/$role.json"; cp -f "$src" "$rj"; RUN_REPORT_JSONS+=( "$rj" ); fi + fi + + # Account spend, record last_run, drop from deferred. + add_spend "$cost" + LAST_RUN["$role"]="$UTC_DATE" + RAN_ROLES+=( "$role" ) +done < <(order_roles) + +# New deferral set = roles deferred this run, plus any previously-deferred role we did NOT run. +for role in ${DEFERRED[@]+"${DEFERRED[@]}"}; do + printf '%s\n' ${RAN_ROLES[@]+"${RAN_ROLES[@]}"} | grep -qxF "$role" && continue + printf '%s\n' ${NEW_DEFERRED[@]+"${NEW_DEFERRED[@]}"} | grep -qxF "$role" && continue + NEW_DEFERRED+=( "$role" ) +done +DEFERRED=( ${NEW_DEFERRED[@]+"${NEW_DEFERRED[@]}"} ) + +log "ran: ${RAN_ROLES[*]:-none} | deferred: ${DEFERRED[*]:-none} | day spend \$$TOTAL_SPEND/\$$TOTAL_BUDGET_USD" + +# ============================================================================== +# 3) COVERAGE ALARM — any role whose last_run is older than MAX_CYCLE_NIGHTS days +# (or never run and deferred that long) is behind (design §5). +# ============================================================================== +NOW_EPOCH="$(to_epoch "$UTC_DATE")" +for entry in "${ROLES[@]}"; do + role="$(role_field "$entry" 1)" + lr="${LAST_RUN[$role]:-}" + if [ -z "$lr" ]; then ref="$CYCLE_START"; else ref="$lr"; fi + age=$(( ( NOW_EPOCH - $(to_epoch "$ref") ) / 86400 )) + if [ "$age" -ge "$MAX_CYCLE_NIGHTS" ]; then + ALARM_LINES+=( "*COVERAGE ALARM*: role '$role' not run in ${age}d (last=${lr:-never, cycle since $CYCLE_START}, max $MAX_CYCLE_NIGHTS). Deferred=$(printf '%s\n' ${DEFERRED[@]+"${DEFERRED[@]}"} | grep -qxF "$role" && echo yes || echo no). Raise budget or check failures." ) + log "COVERAGE ALARM: $role age ${age}d >= $MAX_CYCLE_NIGHTS" + fi +done + +# Persist state (atomic + hashed). Even in dry-run we persist so rotation advances; the +# squeeze test runs dry, so guard: in SQUEEZE we do NOT persist (it is a synthetic scenario). +if [ "$SQUEEZE" -eq 0 ]; then + save_budget_ledger + save_coord_state +else + log "SQUEEZE: synthetic scenario — NOT persisting state." +fi + +# Fold any state-integrity alarms in. +for x in ${STATE_ALARMS[@]+"${STATE_ALARMS[@]}"}; do ALARM_LINES+=( "$x" ); done + +# ============================================================================== +# 4) COLLECT + DEDUP + PRIORITIZE across the run checkers' reports. +# DEDUP rule: same (repo + check + title) OR identical finding id -> one. Sort by severity. +# ============================================================================== +ALL_FINDINGS="[]" +if [ "${#RUN_REPORT_JSONS[@]}" -gt 0 ]; then + ALL_FINDINGS="$(jq -s ' + [ .[].findings[]? ] + | unique_by(.id) # identical id -> one + | unique_by([.repo, .check, .title]) # same repo+check+title -> one + | sort_by( {critical:0, high:1, medium:2, low:3, info:4, unverified:5}[.severity] // 6 ) + ' "${RUN_REPORT_JSONS[@]}" 2>/dev/null || echo '[]')" +fi +N_FIND="$(echo "$ALL_FINDINGS" | jq 'length')" +N_CRITHIGH="$(echo "$ALL_FINDINGS" | jq '[.[]|select(.severity=="critical" or .severity=="high")] | length')" +declare -a CRITHIGH_LINES=() +while IFS= read -r line; do [ -n "$line" ] && CRITHIGH_LINES+=( "$line" ); done < <( + echo "$ALL_FINDINGS" | jq -r '.[] | select(.severity=="critical" or .severity=="high") + | "*\(.repo)* [\(.severity)] \(.title)"') + +# ============================================================================== +# 5) ASSEMBLE the combined coordinator report (JSON + text), mode 600. +# ============================================================================== +DEFERRED_JSON="[]"; [ "${#DEFERRED[@]}" -gt 0 ] && DEFERRED_JSON="$(printf '%s\n' "${DEFERRED[@]}" | jq -R . | jq -cs .)" +RAN_JSON="[]"; [ "${#RAN_ROLES[@]}" -gt 0 ] && RAN_JSON="$(printf '%s\n' "${RAN_ROLES[@]}" | jq -R . | jq -cs .)" +ALARMS_JSON="[]"; [ "${#ALARM_LINES[@]}" -gt 0 ] && ALARMS_JSON="$(printf '%s\n' "${ALARM_LINES[@]}" | jq -R . | jq -cs .)" + +jq -n \ + --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \ + --argjson cap "$TOTAL_BUDGET_USD" --argjson spend "$TOTAL_SPEND" \ + --argjson ran "$RAN_JSON" --argjson deferred "$DEFERRED_JSON" \ + --argjson findings "$ALL_FINDINGS" --argjson alarms "$ALARMS_JSON" \ + '{coordinator:"plane1", generated:$ts, org:$org, + shared_budget_usd:$cap, day_spend_usd:$spend, + ran_roles:$ran, deferred_roles:$deferred, + finding_count:($findings|length), + crit_high:([$findings[]|select(.severity=="critical" or .severity=="high")]|length), + findings:$findings, alarms:$alarms}' > "$REPORT_JSON" + +{ + echo "plane-1 coordinator report — $UTC_STAMP" + echo "org=$GH_ORG shared_cap=\$$TOTAL_BUDGET_USD day_spend=\$$TOTAL_SPEND" + echo "ran: ${RAN_ROLES[*]:-none}" + echo "deferred (NOT dropped): ${DEFERRED[*]:-none}" + echo "findings: $N_FIND ($N_CRITHIGH crit/high)" + echo + echo "$ALL_FINDINGS" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)"' + if [ "${#ALARM_LINES[@]}" -gt 0 ]; then + echo; echo "alarms:"; printf ' - %s\n' "${ALARM_LINES[@]}" + fi +} > "$REPORT_TXT" +chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true +log "report: $REPORT_JSON ($N_FIND finding(s), ${#ALARM_LINES[@]} alarm line(s))" + +# ============================================================================== +# 6) ROUTE (ALARM-only, D3): confirmed crit/high OR any alarm line -> Slack ALARM; +# everything else -> the mode-600 report only; a fully clean run posts NOTHING. +# ============================================================================== +ALARM=0 +[ "$N_CRITHIGH" -gt 0 ] && ALARM=1 +[ "${#ALARM_LINES[@]}" -gt 0 ] && ALARM=1 + +# Squeeze acceptance: print the proof lines explicitly to stdout. +if [ "$SQUEEZE" -eq 1 ]; then + echo "=== SQUEEZE ACCEPTANCE (Phase-2) ===" + echo "DEFERRED (not dropped): ${DEFERRED[*]:-none}" + printf '%s\n' ${ALARM_LINES[@]+"${ALARM_LINES[@]}"} | grep -E 'COVERAGE ALARM|DEFERRED' || true + echo "====================================" +fi + +if [ "$ALARM" -ne 1 ]; then + log "clean run — no crit/high findings, no alarm conditions. Posting NOTHING (ALARM-only policy)." + exit 0 +fi + +ALARM_BODY="" +[ "${#CRITHIGH_LINES[@]}" -gt 0 ] && ALARM_BODY="$(printf '%s\n' "${CRITHIGH_LINES[@]}" | sed 's/^/• /')" +META_BODY="$(printf '%s\n' ${ALARM_LINES[@]+"${ALARM_LINES[@]}"} | sed 's/^/• /')" +SLACK_TEXT=":satellite_antenna: *Sea Haven Plane-1 coordinator — ALARM* ($UTC_STAMP) +ran: ${RAN_ROLES[*]:-none} · deferred: ${DEFERRED[*]:-none} · spend \$$TOTAL_SPEND/\$$TOTAL_BUDGET_USD +$N_CRITHIGH confirmed crit/high finding(s): +$ALARM_BODY + +coordination alarms: +$META_BODY +Combined report (mode 600): \`$REPORT_JSON\` (on R720)" +SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)" + +echo "$SLACK_TEXT" >&2 + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 2)." + exit 0 +fi +post_slack_alarm "$SLACK_TEXT" +exit 0 + +# ============================================================================== +# PROVISIONING (NOT DONE HERE — gated, Phase 6): +# - No systemd unit / timer is installed by this script. Wiring it into the live +# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated. +# - This coordinator runs ONLY the Plane-1 Tier-1 checkers (compliance-drift, +# dependency-cve). doc-drift / aws-posture / planner / fixer are later phases. +# - It does NOT re-clone (checkers reuse $MIRROR_DIR); a checker's own --refresh is the +# only network path and is not invoked here. +# - It does NOT touch agent_team/ or agent-team/, and installs no systemd units. +# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations. +# ============================================================================== diff --git a/checkers/aws-posture.sh b/checkers/aws-posture.sh new file mode 100755 index 0000000..ccea471 --- /dev/null +++ b/checkers/aws-posture.sh @@ -0,0 +1,474 @@ +#!/usr/bin/env bash +# aws-posture.sh — Plane-1 / Tier-2 checker for the R720 agent-team. +# +# Design refs: docs/r720-agent-team-design.md D5 / §4 (Tier 2 roster: aws-posture — +# "Idle/anomalous spend (≈$330/mo flagged) + reasoning layer over baseline findings. Auths via +# Roles Anywhere (short-lived leaf certs, auto-rotated by step-ca). Complements existing +# GuardDuty/Security Hub/Config, does not replace them") and §6.3 / §7 Phase 3 ("doc-drift + +# step-ca/Roles Anywhere + aws-posture"). This is a Tier-2 checker built on the Phase-0 shared +# substrate (lib/sweep_substrate.sh); it mirrors compliance-drift.sh / dependency-cve.sh / +# doc-drift.sh conventions VERBATIM so the coordinator (§5) can drive all of them identically. +# +# WHAT IT DOES (read-only): +# Watches the Sea Haven AWS account (328440206208, us-east-1) for IDLE / ANOMALOUS SPEND and +# idle-resource posture: +# - anomalous Cost Explorer deltas (ce get-anomalies above a $ impact threshold) +# - stopped EC2 instances still paying for attached EBS +# - unattached ("available") EBS volumes +# - unassociated Elastic IPs +# - idle NAT gateways (≈0 bytes out over the window) +# - idle load balancers (0 healthy targets) +# - idle RDS instances (0 connections over the window) +# It COMPLEMENTS GuardDuty / Security Hub / Config (design §4) — it is a spend/idle-posture +# watch, NOT a threat detector, and does not replace them. +# +# AUTH / PROVISIONING GATE (design D5 / §6.3 / §7 B3): +# The LIVE read-only AWS calls require credentials vended via IAM Roles Anywhere using a +# short-lived step-ca leaf cert — this is **PROVISIONING-GATED and NOT available yet** (the IAM +# cross-review PASSED 2026-06-18, which unblocked BUILDING this checker, but step-ca + the trust +# anchor + the role are not stood up). See security-review/iam/ for the reviewed artifacts. +# Therefore the checker: +# (a) attempts read-only `aws` CLI calls ONLY when credentials are actually available +# (an STS identity probe succeeds) AND --no-api/--canary were not passed; +# (b) when there are NO credentials, OR --no-api, OR --canary: it SKIPS the live calls and +# NOTES them — it NEVER alarms on missing data (memory feedback_cloudwatch_alarms: no +# false alarms on no-data). This mirrors compliance-drift's API-skip pattern EXACTLY. +# +# REPORTING (matches secrev sweep conventions): +# - Writes a per-run JSON + text report under $REPORT_ROOT//, mode 600 (umask 077). +# - Slack ALARM-ONLY: a clean run (no confirmed waste) posts NOTHING (memory +# feedback_cloudwatch_alarms). Secret-shaped values are redacted from the Slack string. +# - Reuses the substrate's redact() + post_slack_alarm() verbatim. +# +# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping): +# redact, post_slack_alarm -> Slack delivery (reads SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG) +# (aws-posture does NOT use discover_repos/mirror_repo — it scans an AWS account, not repos.) +# +# CANARY / DRY-RUN (offline, no network, no aws, no credentials): +# --canary runs the SAME detectors against a fixture of mocked AWS JSON responses +# (checkers/fixtures/aws-posture/) and asserts the known finding count against +# EXPECTED_FINDING_COUNT (exit 3 on mismatch). It makes ZERO `aws` calls and ZERO network +# calls. This is the anti-complacency floor (design §6.4) AND the routing dry-run (§7 Phase 3): +# --canary implies --dry-run + --no-api; with --dry-run the Slack alarm is composed + printed +# but NOT POSTed. +# +# SCOPE / SAFETY: +# Read-only. The reasoning ("Sonnet collectors + judge", design §4) is a LATER enhancement: a +# clearly-marked inert stub hook (maybe_judge) marks the future seam; it does NOTHING offline +# and NOTHING in this phase (the deterministic detectors are the whole checker here). Does NOT +# touch agent_team/ or agent-team/, is NOT wired into systemd, and stands NOTHING up in AWS — +# that is Phase-3/6 provisioning (gated). See the "PROVISIONING (NOT DONE HERE)" note at bottom. +# +# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[aws-posture] $*" >&2; } +die() { echo "[aws-posture] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate --------------------------------------------------------- +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSTRATE="$HERE/../lib/sweep_substrate.sh" +[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE" +# shellcheck source=../lib/sweep_substrate.sh +. "$SUBSTRATE" + +# --- Config + defaults (env, all optional) ------------------------------------ +AWS_ACCOUNT="${AWS_ACCOUNT:-328440206208}" +AWS_REGION="${AWS_REGION:-us-east-1}" +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/aws-posture}" +# Cost-anomaly $ impact threshold: only anomalies whose TotalImpact >= this are flagged +# (a tiny anomaly is noise, not waste — no false alarm on a sub-threshold blip). +COST_ANOMALY_MIN_IMPACT="${COST_ANOMALY_MIN_IMPACT:-25}" +# A NAT gateway with bytes-out below this over the window is treated as idle. +NAT_IDLE_BYTES_MAX="${NAT_IDLE_BYTES_MAX:-1024}" +# An RDS instance with max connections at/below this over the window is treated as idle. +RDS_IDLE_CONN_MAX="${RDS_IDLE_CONN_MAX:-0}" + +DO_API=1 # --no-api: skip ALL live AWS calls (offline). Without creds this is forced. +DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run). +CANARY=0 # --canary: run the detectors against the mocked-AWS fixture + assert count. +TARGETS_OVERRIDE="" # --targets DIR: read mocked-AWS JSON from DIR instead of the live account + # (offline + deterministic; same file shape as the canary fixture). + +usage() { + cat >&2 </dev/null || die "jq is required" + +# --- Report dir (mode 600 reports; matches sweep conventions) ----------------- +umask 077 +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +REPORT_DIR="$REPORT_ROOT/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope +SWEEP_LOG="$REPORT_DIR/aws-posture.log" # name the substrate's post_slack_alarm() references +REPORT_JSON="$REPORT_DIR/aws-posture.json" +REPORT_TXT="$REPORT_DIR/aws-posture.txt" + +log "=== aws-posture $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API account=$AWS_ACCOUNT region=$AWS_REGION) ===" + +# ------------------------------------------------------------------------------ +# FINDINGS (spirit of finding.schema.json so the coordinator can route like an agentic finding). +# category="other" (idle-spend is not one of the schema's security categories); +# status="confirmed" only for a deterministic idle/anomaly fact derived from a real response. +# A live call that could not be made (no creds / --no-api / transport failure) is a SKIP, never a +# finding (memory feedback_cloudwatch_alarms: no false alarms on missing data). +# ------------------------------------------------------------------------------ +declare -a FINDINGS=() +add_finding() { # id title severity check resource proof + local id="$1" title="$2" sev="$3" check="$4" resource="$5" proof="$6" + FINDINGS+=( "$(jq -n \ + --arg id "$id" --arg title "$title" --arg sev "$sev" \ + --arg check "$check" --arg resource "$resource" --arg proof "$proof" \ + '{account:env.AWS_ACCOUNT_FOR_FINDING, id:$id, title:$title, severity:$sev, category:"other", + check:$check, status:"confirmed", proof:{resource:$resource, outcome:$proof}}')" ) +} +export AWS_ACCOUNT_FOR_FINDING="$AWS_ACCOUNT" +declare -a SKIPPED_CHECKS=() # (check:reason) live calls skipped on missing data — reported, never alarmed +note_skip() { SKIPPED_CHECKS+=( "$1" ); } + +# Inert future seam (design §4 "Sonnet collectors + judge"): in LIVE mode an ambiguous idle +# candidate ("is this RDS truly idle or just low-traffic?") could be escalated to a reasoning +# judge. This phase keeps the deterministic detectors ONLY — the stub does nothing and is never +# reached offline / in canary / dry-run. +maybe_judge() { # candidate_json (no-op stub; Phase-3 intentionally inert) + return 0 +} + +# ============================================================================== +# DETECTORS — each consumes one AWS JSON response (live or fixture) and emits findings. +# Pure jq parsing; identical logic for the live `aws ... --output json` output and the canary +# fixture, so the canary genuinely exercises the production detectors. +# ============================================================================== + +# cost anomalies: ce get-anomalies. Flag anomalies whose Impact.TotalImpact >= threshold. +detect_cost_anomalies() { # json + local json="$1" + while IFS=$'\t' read -r aid svc impact; do + [ -n "$aid" ] || continue + add_finding "cost-anomaly-$aid" \ + "Cost anomaly: ${svc} (≈\$${impact} impact)" "high" "cost-anomaly" "$aid" \ + "ce get-anomalies TotalImpact \$${impact} >= threshold \$${COST_ANOMALY_MIN_IMPACT} (service: ${svc})" + done < <(echo "$json" | jq -r --argjson thr "$COST_ANOMALY_MIN_IMPACT" ' + (.Anomalies // [])[] + | select((.Impact.TotalImpact // 0) >= $thr) + | [.AnomalyId, (.DimensionValue // "unknown"), ((.Impact.TotalImpact // 0)|tostring)] + | @tsv') +} + +# stopped EC2 still paying for attached EBS: describe-instances, State.Name=="stopped" with EBS. +detect_stopped_instances() { # json + local json="$1" + while IFS=$'\t' read -r iid itype; do + [ -n "$iid" ] || continue + add_finding "stopped-ec2-$iid" \ + "Stopped EC2 instance still incurring EBS cost: $iid ($itype)" "medium" "stopped-instance" "$iid" \ + "ec2 describe-instances: State=stopped with attached EBS (storage bills while stopped)" + done < <(echo "$json" | jq -r ' + (.Reservations // [])[].Instances[] + | select((.State.Name // "") == "stopped") + | select(((.BlockDeviceMappings // []) | length) > 0) + | [.InstanceId, (.InstanceType // "?")] | @tsv') +} + +# unattached EBS: describe-volumes, State=="available". +detect_unattached_volumes() { # json + local json="$1" + while IFS=$'\t' read -r vid size vtype; do + [ -n "$vid" ] || continue + add_finding "unattached-ebs-$vid" \ + "Unattached EBS volume billing idle: $vid (${size}GiB $vtype)" "medium" "unattached-volume" "$vid" \ + "ec2 describe-volumes: State=available (no attachment) — billed but unused" + done < <(echo "$json" | jq -r ' + (.Volumes // [])[] + | select((.State // "") == "available") + | [.VolumeId, ((.Size // 0)|tostring), (.VolumeType // "?")] | @tsv') +} + +# unassociated EIP: describe-addresses, no AssociationId/InstanceId. +detect_unassociated_eips() { # json + local json="$1" + while IFS=$'\t' read -r alloc ip; do + [ -n "$alloc" ] || continue + add_finding "unassociated-eip-$alloc" \ + "Unassociated Elastic IP (hourly charge): $ip" "low" "unassociated-eip" "$alloc" \ + "ec2 describe-addresses: no AssociationId/InstanceId — idle EIPs are billed hourly" + done < <(echo "$json" | jq -r ' + (.Addresses // [])[] + | select((.AssociationId // "") == "" and (.InstanceId // "") == "") + | [(.AllocationId // .PublicIp), (.PublicIp // "?")] | @tsv') +} + +# idle NAT gateway: describe-nat-gateways, available + bytes-out below threshold. +# Live path injects the CloudWatch-derived bytes-out as _FixtureBytesOutLast14d (same key the +# canary fixture uses) before calling this — keeping detector logic identical online/offline. +detect_idle_nat() { # json + local json="$1" + while IFS=$'\t' read -r nid bytes; do + [ -n "$nid" ] || continue + add_finding "idle-nat-$nid" \ + "Idle NAT gateway (≈0 traffic, ~\$32/mo each): $nid" "medium" "idle-nat" "$nid" \ + "ec2 describe-nat-gateways: available with ${bytes} bytes out over window (<= ${NAT_IDLE_BYTES_MAX})" + done < <(echo "$json" | jq -r --argjson mx "$NAT_IDLE_BYTES_MAX" ' + (.NatGateways // [])[] + | select((.State // "") == "available") + | select((._FixtureBytesOutLast14d // 0) <= $mx) + | [.NatGatewayId, ((._FixtureBytesOutLast14d // 0)|tostring)] | @tsv') +} + +# idle ELB: describe-load-balancers, 0 healthy targets. +# Live path injects the per-LB healthy-target count as _FixtureHealthyTargetCount (derived from +# elbv2 describe-target-health) before calling this — same key the canary fixture uses. +detect_idle_elb() { # json + local json="$1" + while IFS=$'\t' read -r name; do + [ -n "$name" ] || continue + add_finding "idle-elb-$name" \ + "Idle load balancer (0 healthy targets, ~\$16/mo each): $name" "medium" "idle-elb" "$name" \ + "elbv2 describe-load-balancers + describe-target-health: 0 healthy targets" + done < <(echo "$json" | jq -r ' + (.LoadBalancers // [])[] + | select((._FixtureHealthyTargetCount // 0) == 0) + | [.LoadBalancerName // .LoadBalancerArn] | @tsv') +} + +# idle RDS: describe-db-instances, available + max connections at/below threshold. +# Live path injects DatabaseConnections max as _FixtureMaxConnectionsLast14d (from CloudWatch). +detect_idle_rds() { # json + local json="$1" + while IFS=$'\t' read -r dbid class; do + [ -n "$dbid" ] || continue + add_finding "idle-rds-$dbid" \ + "Idle RDS instance (0 connections over window): $dbid ($class)" "high" "idle-rds" "$dbid" \ + "rds describe-db-instances: available with 0 connections over window (<= ${RDS_IDLE_CONN_MAX})" + done < <(echo "$json" | jq -r --argjson mx "$RDS_IDLE_CONN_MAX" ' + (.DBInstances // [])[] + | select((.DBInstanceStatus // "") == "available") + | select((._FixtureMaxConnectionsLast14d // 1) <= $mx) + | [.DBInstanceIdentifier, (.DBInstanceClass // "?")] | @tsv') +} + +# Run every detector over a directory of JSON responses (fixture dir or a collected-live dir). +# Missing files are tolerated (a detector with no input simply contributes nothing — never a skip +# that alarms; a genuinely uncollected live call is recorded as a SKIP by the live collector). +run_detectors_over_dir() { # dir + local dir="$1" f + f="$dir/cost-anomalies.json"; [ -f "$f" ] && detect_cost_anomalies "$(cat "$f")" + f="$dir/describe-instances.json"; [ -f "$f" ] && detect_stopped_instances "$(cat "$f")" + f="$dir/describe-volumes.json"; [ -f "$f" ] && detect_unattached_volumes "$(cat "$f")" + f="$dir/describe-addresses.json"; [ -f "$f" ] && detect_unassociated_eips "$(cat "$f")" + f="$dir/describe-nat-gateways.json";[ -f "$f" ] && detect_idle_nat "$(cat "$f")" + f="$dir/describe-load-balancers.json";[ -f "$f" ] && detect_idle_elb "$(cat "$f")" + f="$dir/describe-db-instances.json";[ -f "$f" ] && detect_idle_rds "$(cat "$f")" +} + +# ============================================================================== +# LIVE COLLECTION (read-only AWS, ONLY when credentials are available + not --no-api/--canary). +# Each call is fail-safe: on a transport/permission failure the response is NOT written and the +# call is recorded as a SKIP — never a finding (memory feedback_cloudwatch_alarms). +# The CloudWatch-derived idle metrics (NAT bytes-out, ELB healthy targets, RDS connections) are +# injected into the describe-* JSON under the SAME _Fixture* keys the detectors read, so the live +# and canary code paths are identical. +# ============================================================================== +aws_creds_available() { + command -v aws >/dev/null || return 1 + aws sts get-caller-identity --region "$AWS_REGION" >/dev/null 2>>"$REPORT_DIR/aws.log" +} + +collect_live() { # out_dir + local out="$1"; mkdir -p "$out" + # NOTE: this live collector is PROVISIONING-GATED and only reached when real Roles Anywhere + # creds exist (aws_creds_available passed). Until step-ca/Roles Anywhere are stood up this path + # is never taken; it is written so the checker is complete + ready, not so it runs today. + _try() { # outfile aws-args... + local of="$1"; shift + if aws "$@" --region "$AWS_REGION" --output json >"$of" 2>>"$REPORT_DIR/aws.log"; then + return 0 + else + rm -f "$of"; note_skip "live:$(basename "$of" .json)(aws-call-failed)"; return 1 + fi + } + _try "$out/cost-anomalies.json" ce get-anomalies || true + _try "$out/describe-instances.json" ec2 describe-instances || true + _try "$out/describe-volumes.json" ec2 describe-volumes || true + _try "$out/describe-addresses.json" ec2 describe-addresses || true + _try "$out/describe-nat-gateways.json" ec2 describe-nat-gateways || true + _try "$out/describe-load-balancers.json" elbv2 describe-load-balancers || true + _try "$out/describe-db-instances.json" rds describe-db-instances || true + # Idle-metric enrichment (NAT bytes-out / ELB healthy targets / RDS connections from CloudWatch) + # is injected here in the live path under the _Fixture* keys before the detectors run. It is a + # provisioning-time follow-up — until creds exist this collector is unreachable, so the + # enrichment is intentionally a documented seam, not dead code that runs offline. +} + +# ============================================================================== +# RESOLVE THE INPUT (fixture / explicit dir / live collection) + DECIDE API MODE +# ============================================================================== +SCAN_DIR="" +SCAN_MODE="none" + +if [ "$CANARY" -eq 1 ]; then + FIXTURE_DIR="$HERE/fixtures/aws-posture" + [ -d "$FIXTURE_DIR" ] || die "canary fixture missing: $FIXTURE_DIR" + SCAN_DIR="$FIXTURE_DIR"; SCAN_MODE="canary-fixture" + log "canary: running detectors against mocked-AWS fixtures in $FIXTURE_DIR (no aws, no network)" +elif [ -n "$TARGETS_OVERRIDE" ]; then + d="${TARGETS_OVERRIDE/#\~/$HOME}" + [ -d "$d" ] || die "--targets dir not found: $d" + SCAN_DIR="$d"; SCAN_MODE="explicit-dir" + log "explicit targets dir (offline mocked-AWS JSON): $SCAN_DIR" +elif [ "$DO_API" -eq 1 ] && aws_creds_available; then + COLLECT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/aws-posture-live.XXXXXX")" + trap 'rm -rf "$COLLECT_DIR"' EXIT + log "live: AWS credentials present — collecting read-only responses into $COLLECT_DIR" + collect_live "$COLLECT_DIR" + SCAN_DIR="$COLLECT_DIR"; SCAN_MODE="live-aws" +else + # No creds, or --no-api: SKIP all live calls and note them. NEVER alarm on missing data. + if [ "$DO_API" -eq 1 ]; then + log "live AWS requested but no usable credentials (Roles Anywhere is PROVISIONING-GATED) — skipping all live calls (no false alarms on missing data)" + note_skip "live:all(no-credentials — Roles Anywhere gated; see security-review/iam/)" + else + log "--no-api: skipping all live AWS calls" + note_skip "live:all(--no-api)" + fi + SCAN_MODE="skipped-no-creds" +fi + +# ============================================================================== +# RUN DETECTORS +# ============================================================================== +if [ -n "$SCAN_DIR" ]; then + run_detectors_over_dir "$SCAN_DIR" + maybe_judge "" # inert in this phase (future Sonnet-collector/judge seam) +fi + +# ============================================================================== +# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to the other checkers) +# ============================================================================== +if [ "${#FINDINGS[@]}" -gt 0 ]; then + FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)" +else + FINDINGS_JSON="[]" +fi +if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then + SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)" +else + SKIPPED_JSON="[]" +fi + +N_FIND="$(echo "$FINDINGS_JSON" | jq 'length')" +N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high" or .severity=="critical")] | length')" + +jq -n \ + --arg checker "aws-posture" --arg ts "$UTC_STAMP" --arg account "$AWS_ACCOUNT" \ + --arg region "$AWS_REGION" --arg mode "$SCAN_MODE" \ + --argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \ + '{checker:$checker, generated:$ts, account:$account, region:$region, scan_mode:$mode, + finding_count:($findings|length), + findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON" + +{ + echo "aws-posture report — $UTC_STAMP" + echo "account=$AWS_ACCOUNT region=$AWS_REGION scan_mode=$SCAN_MODE" + echo "idle/anomalous-spend findings: $N_FIND ($N_HIGH high/critical)" + echo + echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.check): \(.title)\n proof: \(.proof.outcome)"' + if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then + echo; echo "skipped (missing data — NOT counted as a finding):" + echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"' + fi +} > "$REPORT_TXT" +chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true + +log "report: $REPORT_JSON ($N_FIND finding(s), mode=$SCAN_MODE)" + +# ============================================================================== +# CANARY ASSERTION (anti-complacency floor, design §6.4) +# ============================================================================== +if [ "$CANARY" -eq 1 ]; then + EXPECT_FILE="$HERE/fixtures/aws-posture/EXPECTED_FINDING_COUNT" + [ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE" + EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")" + log "canary assertion: expected findings=$EXPECTED, got=$N_FIND" + if [ "$N_FIND" -ne "$EXPECTED" ]; then + echo "[aws-posture] CANARY FAIL: planted-finding count mismatch (expected $EXPECTED, got $N_FIND)" >&2 + echo " -> a detector regressed (stopped firing) or the fixture changed. See $REPORT_TXT." >&2 + exit 3 + fi + log "canary PASS: all $EXPECTED planted idle/anomaly findings detected." +fi + +# ============================================================================== +# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms) +# ============================================================================== +if [ "$N_FIND" -eq 0 ]; then + log "no idle/anomalous spend detected — posting NOTHING to Slack (ALARM-only policy)." + exit 0 +fi + +ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r ' + group_by(.check)[] | "*\(.[0].check)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')" +SLACK_TEXT=":money_with_wings: *Sea Haven aws-posture — ALARM* ($UTC_STAMP) +$N_FIND idle/anomalous-spend finding(s) in account $AWS_ACCOUNT/$AWS_REGION ($N_HIGH high/critical): +$ALARM_BODY + +Scope: idle/anomalous SPEND + idle-resource posture (complements GuardDuty/SecurityHub/Config, mode=$SCAN_MODE) +Report (mode 600): \`$REPORT_JSON\` (on R720)" +SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)" + +echo "$SLACK_TEXT" >&2 + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 3)." + exit 0 +fi +post_slack_alarm "$SLACK_TEXT" +exit 0 + +# ============================================================================== +# PROVISIONING (NOT DONE HERE — gated, Phase 3 / Phase 6): +# - The LIVE AWS calls need credentials vended via IAM Roles Anywhere using a short-lived +# step-ca leaf cert. step-ca + the Roles Anywhere trust anchor + the read-only role are NOT +# stood up by this script. The reviewed IAM artifacts live in security-review/iam/ (GPT-4.1 +# cross-review PASSED 2026-06-18: APPROVE, no BLOCKs). Provisioning happens only after that +# review is recorded (design §7, B3) and a VM snapshot is taken (feedback_ec2_replacement_snapshot). +# Until then aws_creds_available() returns false and the checker SKIPS all live calls (no +# false alarms on missing data) — only --canary / --targets exercise it offline. +# - No systemd unit / timer is installed by this script. Wiring it into the live secrev schedule +# (weekly cadence, design §4) is provisioning and is gated. +# - This script is NOT registered in checker_coordinator.sh; the coordinator registry is +# integrated centrally (separate change). +# - The LIVE "Sonnet collectors + judge" reasoning layer (design §4) is the only LLM seam; it is +# an inert stub here (maybe_judge) and stays off in canary / dry-run / offline. +# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations for the +# build session, tracked outside this script. +# ============================================================================== diff --git a/checkers/compliance-drift.sh b/checkers/compliance-drift.sh new file mode 100755 index 0000000..3115d11 --- /dev/null +++ b/checkers/compliance-drift.sh @@ -0,0 +1,492 @@ +#!/usr/bin/env bash +# compliance-drift.sh — Plane-1 / Tier-1 checker for the R720 agent-team. +# +# Design refs: docs/r720-agent-team-design.md §4 (Tier 1 roster: compliance-drift) and +# §7 Phase 1 ("one checker end to end"). This is the FIRST Plane-1 checker built on the +# Phase-0 shared substrate (lib/sweep_substrate.sh) — it proves the substrate generalizes +# beyond the secrev nightly sweep. +# +# WHAT IT DOES (read-only): +# Flags drift from Sea Haven engineering conventions across the org mirrors. It scans the +# SAME shallow clean clones that nightly_sweep.sh already produced in $MIRROR_DIR — it does +# NOT re-clone (mirrors-first; an optional --refresh re-runs discovery+mirror via the shared +# substrate). The checklist is GROUNDED in the engineering-handbook + this repo's README; it +# does not invent rules. See "CHECKLIST" below. +# +# REPORTING (matches secrev sweep conventions): +# - Writes a per-run JSON + text report under $REPORT_ROOT//, mode 600 (umask 077). +# - Slack ALARM-ONLY: a clean run (no confirmed drift) posts NOTHING (memory +# feedback_cloudwatch_alarms). Secret-shaped values are redacted from the Slack string. +# - Reuses the substrate's redact() + post_slack_alarm() verbatim. +# +# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping): +# redact, post_slack_alarm -> Slack delivery (reads SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG) +# discover_repos, mirror_repo-> ONLY on --refresh (reads GH_TOKEN, GH_ORG, MIRROR_DIR, REPORT_DIR) +# Default path enumerates EXISTING $MIRROR_DIR/*/.git dirs — zero clones, zero network. +# +# CANARY / DRY-RUN (offline, no network, no token): +# --canary runs the checklist against a planted-drift fixture (checkers/fixtures/compliance-drift/) +# and asserts the known drift count. This is the anti-complacency floor (design §6.4) AND the +# routing dry-run (§7 Phase 1, F4): with --dry-run, the Slack alarm is composed + printed but +# NOT POSTed. Fully offline-smoke-testable. +# +# SCOPE / SAFETY: +# Read-only. Filesystem checks need no network. The branch-protection / Dependabot-alerts / +# repo-settings checks call the GitHub REST API read-only with the same $GH_TOKEN the sweep +# uses (Contents+Metadata read). When GH_TOKEN is unset OR --no-api is passed (the offline +# default for --canary), API-only checks are SKIPPED and noted in the report — they are never +# reported as drift on missing data (memory feedback_cloudwatch_alarms: no false alarms on no-data). +# +# This script does NOT touch agent_team/ or agent-team/, and is NOT wired into systemd — that is +# Phase-6 provisioning (gated). See the "PROVISIONING (NOT DONE HERE)" note at the bottom. +# +# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[compliance-drift] $*" >&2; } +die() { echo "[compliance-drift] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate --------------------------------------------------------- +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSTRATE="$HERE/../lib/sweep_substrate.sh" +[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE" +# shellcheck source=../lib/sweep_substrate.sh +. "$SUBSTRATE" + +# --- Config + defaults (env, all optional) ------------------------------------ +GH_ORG="${GH_ORG:-Sea-Haven-Industries}" +MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}" +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/compliance-drift}" +# Repos exempt from CodeQL/compliance tooling per github-standards.md ("Exceptions"). +# Comma-separated; handbook lists shoc-backend, shoc-frontend-new (SHOC-owned) + docs repos. +COMPLIANCE_EXEMPT="${COMPLIANCE_EXEMPT:-shoc-backend,shoc-frontend-new}" +# Docs-only repos skip CodeQL/CI-deploy expectations (handbook exception); they still need README. +DOCS_ONLY_REPOS="${DOCS_ONLY_REPOS:-engineering-handbook}" + +REFRESH=0 # --refresh: re-run discovery+mirror via substrate (network). Default: reuse mirrors. +DO_API=1 # --no-api: skip GitHub-API checks (branch protection / dependabot / settings). +DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run, F4). +CANARY=0 # --canary: run against the planted-drift fixture + assert the known count. +TARGETS_OVERRIDE="" # --targets "p1 p2": scan explicit dirs instead of the mirror set. + +usage() { + cat >&2 </dev/null || die "jq is required" +command -v git >/dev/null || die "git is required" + +# --- Report dir (mode 600 reports; matches sweep conventions) ----------------- +umask 077 +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +REPORT_DIR="$REPORT_ROOT/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope +SWEEP_LOG="$REPORT_DIR/compliance-drift.log" # name the substrate's post_slack_alarm() references +REPORT_JSON="$REPORT_DIR/compliance-drift.json" +REPORT_TXT="$REPORT_DIR/compliance-drift.txt" + +log "=== compliance-drift $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API refresh=$REFRESH) ===" + +# ------------------------------------------------------------------------------ +# CHECKLIST (grounded — every item cites a handbook/README rule; nothing invented): +# +# naming-repo repo dir name is kebab-case naming-conventions.md ("kebab-case for everything") +# readme-present README.md exists at repo root github-standards.md / global CLAUDE.md ("Every repo must have a README") +# cicd-present .github/workflows/ci.yaml|ci.yml cicd.md ("Every deployable repo must have a CI/CD pipeline"; ci.yaml) +# dependabot-config .github/dependabot.yml present when github-standards.md ("Every repo with dependencies gets a .github/dependabot.yml") +# dependency manifests exist +# secrets-committed no committed .env with real-looking secrets-and-config.md ("Never commit .env files containing real values") +# values (tracked-in-git, not gitignored) +# --- API-only (need GH_TOKEN; skipped offline / --no-api / --canary) --- +# branch-protection main requires PR, no force-push, github-standards.md ("Branch Protection") +# no deletion +# dependabot-alerts Dependabot alerts + security updates github-standards.md ("Dependabot alerts and security updates enabled") +# enabled +# merge-settings allow_auto_merge + delete_branch_on_ github-standards.md ("enable auto-merge and auto-delete head branch") +# merge enabled +# +# Each emitted finding follows the spirit of finding.schema.json (id/title/severity/category/ +# proof/status) so a later phase can route it like an agentic finding. category="other" — this is +# convention drift, not the schema's security categories. status="confirmed" only for deterministic +# filesystem facts and explicit API "false" answers; API checks on missing data are NOT findings. +# ------------------------------------------------------------------------------ + +# Drift accumulator: one JSON object per finding, appended to a bash array. +declare -a FINDINGS=() +add_finding() { # repo id title severity check proof + local repo="$1" id="$2" title="$3" sev="$4" check="$5" proof="$6" + FINDINGS+=( "$(jq -n \ + --arg repo "$repo" --arg id "$id" --arg title "$title" --arg sev "$sev" \ + --arg check "$check" --arg proof "$proof" \ + '{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other", + check:$check, status:"confirmed", proof:{outcome:$proof}}')" ) +} +declare -a SKIPPED_CHECKS=() # (repo:check) checks skipped on missing data — reported, never alarmed +note_skip() { SKIPPED_CHECKS+=( "$1" ); } + +in_csv() { # needle csv -> 0 if present + local n="$1" csv="$2"; case ",$csv," in *",$n,"*) return 0 ;; *) return 1 ;; esac +} + +# --- kebab-case test (lowercase, digits, single hyphens; no leading/trailing hyphen) --- +is_kebab() { [[ "$1" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; } + +# --- Does the repo carry dependency manifests that warrant a dependabot.yml? ---- +has_dep_manifests() { # dir + local d="$1" + # Match handbook's ecosystem table: package.json / requirements.txt / *.csproj. + [ -f "$d/package.json" ] && return 0 + find "$d" -maxdepth 3 -name requirements.txt -not -path '*/.git/*' -print -quit 2>/dev/null | grep -q . && return 0 + find "$d" -maxdepth 3 -name '*.csproj' -not -path '*/.git/*' -print -quit 2>/dev/null | grep -q . && return 0 + return 1 +} + +# ============================================================================== +# FILESYSTEM CHECKS (offline; run on every repo dir) +# ============================================================================== +check_repo_fs() { # repo_name repo_dir + local repo="$1" dir="$2" + local docs_only=0; in_csv "$repo" "$DOCS_ONLY_REPOS" && docs_only=1 + local exempt=0; in_csv "$repo" "$COMPLIANCE_EXEMPT" && exempt=1 + + # naming-repo — repo dir name kebab-case + is_kebab "$repo" || add_finding "$repo" "naming-repo" \ + "Repo name '$repo' is not kebab-case" "medium" "naming-repo" \ + "naming-conventions.md: kebab-case for everything (repository names)" + + # readme-present — every repo, no exceptions + [ -f "$dir/README.md" ] || add_finding "$repo" "readme-missing" \ + "No README.md at repo root" "high" "readme-present" \ + "global CLAUDE.md / github-standards.md: every repo must have a README" + + # cicd-present — ci workflow expected unless docs-only or compliance-exempt + if [ "$docs_only" -eq 0 ] && [ "$exempt" -eq 0 ]; then + if [ ! -f "$dir/.github/workflows/ci.yaml" ] && [ ! -f "$dir/.github/workflows/ci.yml" ]; then + add_finding "$repo" "cicd-missing" \ + "No .github/workflows/ci.yaml" "high" "cicd-present" \ + "cicd.md: every deployable repo must have a CI/CD pipeline (ci.yaml)" + fi + else + note_skip "$repo:cicd-present(docs-only/exempt)" + fi + + # dependabot-config — required only when dependency manifests exist, and not exempt + if [ "$exempt" -eq 0 ] && has_dep_manifests "$dir"; then + [ -f "$dir/.github/dependabot.yml" ] || [ -f "$dir/.github/dependabot.yaml" ] || \ + add_finding "$repo" "dependabot-config-missing" \ + "Has dependency manifests but no .github/dependabot.yml" "medium" "dependabot-config" \ + "github-standards.md: every repo with dependencies gets a .github/dependabot.yml" + fi + + # secrets-committed — a .env TRACKED in git (gitignored .env is fine; tracked is the drift) + if [ -d "$dir/.git" ]; then + while IFS= read -r envf; do + [ -n "$envf" ] || continue + # Only flag .env / .env.* that look like they hold real values, not .env.example/.sample/.template. + case "$envf" in *.example|*.sample|*.template|*.dist) continue ;; esac + # Fire only on secret-SHAPED entries: a secret-ish key name, or a long + # (>=20 char) high-entropy value. Benign config (PORT=3000, DEBUG=true) + # is NOT drift, so a tracked config-only .env raises no ALARM + # (feedback_cloudwatch_alarms: no false alarms on non-secret config). + if grep -qiE '(secret|token|key|password|passwd|api[_-]?key|credential|private)[^=]*=[^[:space:]#]+' "$dir/$envf" 2>/dev/null \ + || grep -qE '=[^[:space:]#]{20,}' "$dir/$envf" 2>/dev/null; then + add_finding "$repo" "secrets-committed-$(echo "$envf" | tr '/.' '--')" \ + "Tracked env file with values committed: $envf" "high" "secrets-committed" \ + "secrets-and-config.md: never commit .env files containing real values" + fi + done < <(git -C "$dir" ls-files -- '*.env' '.env' '.env.*' 2>/dev/null || true) + else + note_skip "$repo:secrets-committed(not-a-git-checkout)" + fi +} + +# ============================================================================== +# API CHECKS (read-only GitHub REST; need GH_TOKEN; skipped offline/--no-api/--canary) +# ============================================================================== +gh_api() { # path -> body on stdout, non-zero on transport/HTTP error + curl -fsS \ + -H "Authorization: Bearer $GH_TOKEN" \ + -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "https://api.github.com/$1" 2>>"$REPORT_DIR/api.log" +} + +check_repo_api() { # repo_name + local repo="$1" + local exempt=0; in_csv "$repo" "$COMPLIANCE_EXEMPT" && exempt=1 + + # repo settings: merge baseline + vulnerability-alerts capability come off the repo object. + local body + if ! body="$(gh_api "repos/$GH_ORG/$repo")" || ! echo "$body" | jq -e 'type=="object" and has("name")' >/dev/null 2>&1; then + note_skip "$repo:api(repo-fetch-failed)"; return + fi + local default_branch; default_branch="$(echo "$body" | jq -r '.default_branch // "main"')" + + # merge-settings — auto-merge + delete-branch-on-merge (per-repo, no org default) + if [ "$exempt" -eq 0 ]; then + local am dbm; am="$(echo "$body" | jq -r '.allow_auto_merge')"; dbm="$(echo "$body" | jq -r '.delete_branch_on_merge')" + [ "$am" = "true" ] || add_finding "$repo" "merge-automerge-off" \ + "allow_auto_merge disabled" "low" "merge-settings" \ + "github-standards.md: enable auto-merge (allow_auto_merge)" + [ "$dbm" = "true" ] || add_finding "$repo" "merge-deletebranch-off" \ + "delete_branch_on_merge disabled" "low" "merge-settings" \ + "github-standards.md: enable auto-delete head branch on merge (delete_branch_on_merge)" + fi + + # dependabot-alerts — vulnerability alerts enabled (204 = enabled, 404 = disabled) + if [ "$exempt" -eq 0 ]; then + local code + # No -f: a 404 (alerts off) is a real HTTP response we must classify, so curl + # must exit 0 and -w must yield a clean "404" (with -f the body-fail path + # corrupts the captured code and a real 404 would be misread as a skip). + code="$(curl -sS -o /dev/null -w '%{http_code}' \ + -H "Authorization: Bearer $GH_TOKEN" -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "https://api.github.com/repos/$GH_ORG/$repo/vulnerability-alerts" 2>>"$REPORT_DIR/api.log" || echo 000)" + case "$code" in + 204) : ;; # enabled + 404) add_finding "$repo" "dependabot-alerts-off" \ + "Dependabot vulnerability alerts disabled" "high" "dependabot-alerts" \ + "github-standards.md: Dependabot alerts and security updates enabled on all active repos" ;; + *) note_skip "$repo:dependabot-alerts(http-$code)" ;; # missing data -> no alarm + esac + fi + + # branch-protection — main: require PR, no force-push, no deletion. + # Status-code-aware (mirrors dependabot-alerts): 200 -> parse the rules, + # 404 -> no protection rule = real drift, anything else (403/5xx/000 transient + # or transport failure) -> skip with NO alarm (feedback_cloudwatch_alarms: a + # flaky API call must never raise a high-severity false alarm). + local prot_tmp prot_code prot + prot_tmp="$(mktemp)" + prot_code="$(curl -sS -o "$prot_tmp" -w '%{http_code}' \ + -H "Authorization: Bearer $GH_TOKEN" -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "https://api.github.com/repos/$GH_ORG/$repo/branches/$default_branch/protection" \ + 2>>"$REPORT_DIR/api.log" || echo 000)" + prot="$(cat "$prot_tmp" 2>/dev/null)"; rm -f "$prot_tmp" + case "$prot_code" in + 200) + echo "$prot" | jq -e '.required_pull_request_reviews != null' >/dev/null 2>&1 || \ + add_finding "$repo" "branchprot-no-pr" \ + "main does not require a PR for merge" "high" "branch-protection" \ + "github-standards.md: require a PR for merges to main (no direct push)" + echo "$prot" | jq -e '.allow_force_pushes.enabled == false' >/dev/null 2>&1 || \ + add_finding "$repo" "branchprot-force-push" \ + "main allows force-push" "high" "branch-protection" \ + "github-standards.md: no force push to main" + echo "$prot" | jq -e '.allow_deletions.enabled == false' >/dev/null 2>&1 || \ + add_finding "$repo" "branchprot-deletion" \ + "main allows branch deletion" "high" "branch-protection" \ + "github-standards.md: no branch deletion for main" + ;; + 404) + # 404 from this endpoint = no protection rule at all on the default branch -> that IS drift. + add_finding "$repo" "branchprot-absent" \ + "No branch protection on '$default_branch'" "high" "branch-protection" \ + "github-standards.md: require a PR for merges to main, no force push, no deletion" + ;; + *) note_skip "$repo:branch-protection(http-$prot_code)" ;; # transient/forbidden -> no alarm + esac +} + +# ============================================================================== +# TARGET RESOLUTION +# ============================================================================== +declare -a REPO_NAMES=(); declare -A REPO_DIR=() + +if [ "$CANARY" -eq 1 ]; then + FIXTURE_ROOT="$HERE/fixtures/compliance-drift" + [ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT" + # Pin the exception lists the fixtures were authored against, so the canary is + # self-contained and deterministic regardless of the operator's env. + DOCS_ONLY_REPOS="docs-repo" + COMPLIANCE_EXEMPT="" + # Fixtures ship their git metadata as `dotgit/` (not `.git/`) so they are committable + # into THIS repo without becoming nested submodules. Materialize them into a temp work + # area — copy each fixture and rename dotgit -> .git — so the tracked-`.env`/ls-files + # checks run against a real git checkout. The temp area is mode 700 and removed on exit. + FIXTURE_WORK="$(mktemp -d "${TMPDIR:-/tmp}/compliance-drift-canary.XXXXXX")" + trap 'rm -rf "$FIXTURE_WORK"' EXIT + log "canary: materializing planted-drift fixtures from $FIXTURE_ROOT into $FIXTURE_WORK" + for d in "$FIXTURE_ROOT"/*/; do + [ -d "$d/dotgit" ] || continue # only fixture repos (skip README.md etc.) + nm="$(basename "$d")" + cp -R "$d" "$FIXTURE_WORK/$nm" + mv "$FIXTURE_WORK/$nm/dotgit" "$FIXTURE_WORK/$nm/.git" + # The planted-secret env file is shipped as `dotenv.fixture` (NOT `.env`): the repo's + # root .gitignore lists `.env`, so a literal `.env` fixture would never be committed and + # the secrets-committed drift would vanish on a fresh clone. Restore it to `.env` in the + # materialized work area (the dotgit/ index already TRACKS `.env`, so ls-files still + # reports it). Same committable-without-side-effects rationale as the `.fixture` suffix the + # dependency-cve fixtures use for their manifests. + [ -f "$FIXTURE_WORK/$nm/dotenv.fixture" ] && mv "$FIXTURE_WORK/$nm/dotenv.fixture" "$FIXTURE_WORK/$nm/.env" + REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$FIXTURE_WORK/$nm" + done +elif [ -n "$TARGETS_OVERRIDE" ]; then + # shellcheck disable=SC2206 + arr=( $TARGETS_OVERRIDE ) + for p in "${arr[@]}"; do p="${p/#\~/$HOME}"; nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p"; done + log "explicit targets: ${REPO_NAMES[*]}" +else + if [ "$REFRESH" -eq 1 ]; then + [ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN" + command -v curl >/dev/null || die "--refresh needs curl" + mkdir -p "$MIRROR_DIR" + log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)" + DISCOVERED="$REPORT_DIR/discovered.tsv" + if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then + while IFS=$'\t' read -r name url branch; do + [ -n "$name" ] || continue + mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)" + done < "$DISCOVERED" + else + log "discovery failed — falling back to existing mirrors (coverage may be stale)" + fi + fi + # Default + post-refresh: enumerate EXISTING mirrors. No clone here — reuse the sweep's clones. + [ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)" + for d in "$MIRROR_DIR"/*/; do + [ -d "$d/.git" ] || continue + nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}" + done + log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)" +fi + +[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to scan" + +# Decide whether API checks run: need a token, the API enabled, and not the offline canary. +RUN_API=0 +if [ "$DO_API" -eq 1 ] && [ -n "${GH_TOKEN:-}" ] && command -v curl >/dev/null; then RUN_API=1 +elif [ "$DO_API" -eq 1 ]; then log "API checks requested but GH_TOKEN/curl unavailable — skipping (no false alarms on missing data)"; fi + +# ============================================================================== +# RUN CHECKS +# ============================================================================== +for nm in "${REPO_NAMES[@]}"; do + check_repo_fs "$nm" "${REPO_DIR[$nm]}" + [ "$RUN_API" -eq 1 ] && check_repo_api "$nm" +done + +# ============================================================================== +# ASSEMBLE REPORT (JSON + text), mode 600 +# ============================================================================== +if [ "${#FINDINGS[@]}" -gt 0 ]; then + FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)" +else + FINDINGS_JSON="[]" +fi +if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then + SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)" +else + SKIPPED_JSON="[]" +fi + +N_DRIFT="$(echo "$FINDINGS_JSON" | jq 'length')" +N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high")] | length')" +N_REPOS_DRIFTED="$(echo "$FINDINGS_JSON" | jq '[.[].repo] | unique | length')" + +jq -n \ + --arg checker "compliance-drift" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \ + --argjson api "$RUN_API" --argjson scanned "${#REPO_NAMES[@]}" \ + --argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \ + '{checker:$checker, generated:$ts, org:$org, api_checks_ran:($api==1), + repos_scanned:$scanned, drift_count:($findings|length), + repos_with_drift:([$findings[].repo]|unique|length), + findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON" + +{ + echo "compliance-drift report — $UTC_STAMP" + echo "org=$GH_ORG repos_scanned=${#REPO_NAMES[@]} api_checks=$([ "$RUN_API" -eq 1 ] && echo on || echo off)" + echo "drift findings: $N_DRIFT ($N_HIGH high) across $N_REPOS_DRIFTED repo(s)" + echo + echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n rule: \(.proof.outcome)"' + if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then + echo; echo "skipped checks (missing data — NOT counted as drift):" + echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"' + fi +} > "$REPORT_TXT" +chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true + +log "report: $REPORT_JSON ($N_DRIFT drift finding(s), $N_REPOS_DRIFTED repo(s))" + +# ============================================================================== +# CANARY ASSERTION (anti-complacency floor, design §6.4) +# ============================================================================== +if [ "$CANARY" -eq 1 ]; then + EXPECT_FILE="$HERE/fixtures/compliance-drift/EXPECTED_DRIFT_COUNT" + [ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE" + EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")" + log "canary assertion: expected drift=$EXPECTED, got=$N_DRIFT" + if [ "$N_DRIFT" -ne "$EXPECTED" ]; then + echo "[compliance-drift] CANARY FAIL: planted-drift count mismatch (expected $EXPECTED, got $N_DRIFT)" >&2 + echo " -> the checklist regressed (a check stopped firing) or the fixture changed. See $REPORT_TXT." >&2 + exit 3 + fi + log "canary PASS: all $EXPECTED planted drifts detected." +fi + +# ============================================================================== +# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms) +# ============================================================================== +if [ "$N_DRIFT" -eq 0 ]; then + log "no confirmed drift — posting NOTHING to Slack (ALARM-only policy)." + exit 0 +fi + +ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r ' + group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')" +SLACK_TEXT=":triangular_flag_on_post: *Sea Haven compliance-drift — ALARM* ($UTC_STAMP) +$N_DRIFT drift finding(s) across $N_REPOS_DRIFTED repo(s) ($N_HIGH high): +$ALARM_BODY + +Checks: naming · README · CI/CD · Dependabot · secrets-placement · branch-protection (api=$([ "$RUN_API" -eq 1 ] && echo on || echo off)) +Report (mode 600): \`$REPORT_JSON\` (on R720)" +SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)" + +echo "$SLACK_TEXT" >&2 + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 1 / F4)." + exit 0 +fi +post_slack_alarm "$SLACK_TEXT" +exit 0 + +# ============================================================================== +# PROVISIONING (NOT DONE HERE — gated, Phase 6): +# - No systemd unit / timer is installed by this script. Wiring it into the live +# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated. +# - The coordinator (design §5) that runs this alongside other Tier-1 checkers under +# one shared budget + versioned rotation state is Phase 2, not built here. +# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations +# for the build session, tracked outside this script. +# ============================================================================== diff --git a/checkers/confluence-doc.sh b/checkers/confluence-doc.sh new file mode 100755 index 0000000..ca945dc --- /dev/null +++ b/checkers/confluence-doc.sh @@ -0,0 +1,542 @@ +#!/usr/bin/env bash +# confluence-doc.sh — Plane-1 SCHEDULED documentation gap-detector (RECOMMEND-ONLY). +# +# Design refs: docs/r720-agent-team-design.md §4 (confluence-doc row) and §7 Phase 4. +# Decisions D3 + D6 + D7: +# D3 Report/recommend-only to start; no auto-Notion/Jira writes. +# D6 Confluence writes (the LATER on-demand path) use a dedicated IT-space-scoped +# `confluence-bot` Atlassian service account — PROVISIONING, gated (see footer). +# D7 SCHEDULED mode = read + RECOMMEND only: doc gaps / stale pages / missing runbooks go +# INTO the mode-600 report, NEVER auto-written. The on-demand SSH-invoked WRITE path +# (including Mermaid edits via ~/.claude/scripts/confluence_mermaid.py) is a separate, +# LATER provisioning path and is NOT implemented here. +# This mirrors compliance-drift.sh / dependency-cve.sh conventions VERBATIM so the coordinator +# (§5) drives it identically. +# +# WHAT IT DOES (read-only, RECOMMEND-ONLY): +# Diffs three documentation INPUTS against what Confluence's IT space actually documents, and +# REPORTS the gaps as recommendations (never writes): +# 1. REPO SET — every non-archived org repo (from the same $MIRROR_DIR mirrors the +# sweep already produced; or --targets / a fixture repo list) SHOULD +# have a Confluence page in the IT page-ID map. A repo with no mapped +# page is a "doc gap" recommendation. +# 2. AWS INVENTORY — (optional) a read-only AWS resource inventory JSON (stacks/Lambdas) +# SHOULD each be represented in the AWS Architecture Map / a page. +# A resource absent from the map is a "missing-from-architecture-map" +# recommendation. Absent inventory file => that whole check is SKIPPED +# (noted, never a gap on missing data). +# 3. PAGE-ID MAP — required runbook/standing pages (Incident Response Runbooks, Backup & +# DR, IAM & Access) SHOULD exist in the map. A required page missing +# from the map is a "missing-runbook" recommendation. Optionally, the +# LIVE Confluence API confirms each mapped page still exists and is not +# stale (lastUpdated older than $STALE_DAYS). +# +# The page-ID map is the canonical one from memory project_confluence_migration (IT space +# 720900). It is supplied as a JSON file (--page-map / $PAGE_MAP_FILE); the canary ships a +# mock map. We do NOT hardcode the live IDs into this script — they live in the map file so +# the map can evolve without a code change. +# +# CONFLUENCE API (LIVE reads need the confluence-bot token — PROVISIONING): +# The staleness / page-existence checks call the Confluence Cloud REST API read-only using +# CONFLUENCE_BASE_URL + CONFLUENCE_EMAIL + CONFLUENCE_API_TOKEN (the confluence-bot creds, +# D6). When those are ABSENT, OR --no-api / --canary is passed, the API checks are SKIPPED +# and NOTED — they are NEVER reported as a gap on missing data (memory +# feedback_cloudwatch_alarms: no false alarms on no-data). This mirrors compliance-drift's +# GitHub-API-skip pattern EXACTLY (status-code-aware: 200 -> parse, 404 -> a real "page gone" +# gap, anything else -> skip with NO alarm). The token / service account is gated provisioning. +# +# ON-DEMAND WRITE PATH (NOT HERE — provisioning): an actual Confluence update, including Mermaid +# architecture-map edits, goes through ~/.claude/scripts/confluence_mermaid.py (ADF-only, +# dry-run-default, macro-count + revert-diff guarded — it has destroyed page 1540098 before via +# a full-body markdown round-trip, so ADF-only is load-bearing). That --apply / live-dry-run is +# the LATER on-demand path and is gated. See the PROVISIONING footer. +# +# CANARY / DRY-RUN (offline, no network, no token): +# --canary runs against a fixture (checkers/fixtures/confluence-doc/): a repo list, a MOCK +# page-ID map, and a MOCK "confluence inventory" JSON (what the API would have returned). It +# asserts the known gap count against EXPECTED_GAP_COUNT (exit 3 on mismatch). --canary implies +# --dry-run + --no-api, so it is fully offline + deterministic. This is the anti-complacency +# floor (design §6.4) AND the routing dry-run. +# +# SCOPE / SAFETY: +# Read-only + RECOMMEND-only. Never writes Confluence, never creates a service account, never +# calls the Mermaid --apply path. Not wired into systemd. See PROVISIONING footer. +# +# Exit: 0 = ran (whether or not it found gaps); 2 = setup/usage error; 3 = canary assertion FAILED. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[confluence-doc] $*" >&2; } +die() { echo "[confluence-doc] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate --------------------------------------------------------- +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSTRATE="$HERE/../lib/sweep_substrate.sh" +[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE" +# shellcheck source=../lib/sweep_substrate.sh +. "$SUBSTRATE" + +# --- Config + defaults (env, all optional) ------------------------------------ +GH_ORG="${GH_ORG:-Sea-Haven-Industries}" +MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}" +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/confluence-doc}" +# Canonical IT page-ID map (memory project_confluence_migration). JSON, NOT hardcoded here. +PAGE_MAP_FILE="${PAGE_MAP_FILE:-}" +# Optional read-only AWS inventory JSON (stacks/Lambdas) — absent => that check is SKIPPED. +AWS_INVENTORY_FILE="${AWS_INVENTORY_FILE:-}" +# Confluence Cloud REST (the confluence-bot creds, D6) — absent => API checks SKIPPED. +# TWO auth modes are supported; OAuth takes precedence when its creds are present: +# (A) OAuth 2.0 client-credentials (2LO) for an org SERVICE ACCOUNT (preferred for a +# headless bot — Atlassian org service accounts have no classic API token): +# POST https://auth.atlassian.com/oauth/token (client_id+client_secret+ +# grant_type=client_credentials) -> 60-min Bearer token, then call +# https://api.atlassian.com/ex/confluence//wiki/api/v2/... +# (B) Basic auth (account email + API token) against the site /wiki/api/v2/... +CONFLUENCE_BASE_URL="${CONFLUENCE_BASE_URL:-}" +CONFLUENCE_EMAIL="${CONFLUENCE_EMAIL:-}" +CONFLUENCE_API_TOKEN="${CONFLUENCE_API_TOKEN:-}" +CONFLUENCE_OAUTH_CLIENT_ID="${CONFLUENCE_OAUTH_CLIENT_ID:-}" +CONFLUENCE_OAUTH_CLIENT_SECRET="${CONFLUENCE_OAUTH_CLIENT_SECRET:-}" +# Optional: the site cloudId. If empty under OAuth, it is auto-resolved from the +# site's public /_edge/tenant_info (no auth needed). +CONFLUENCE_CLOUD_ID="${CONFLUENCE_CLOUD_ID:-}" +# Atlassian OAuth token endpoint (overridable only for testing). +CONFLUENCE_OAUTH_TOKEN_URL="${CONFLUENCE_OAUTH_TOKEN_URL:-https://auth.atlassian.com/oauth/token}" +# A mapped page is "stale" if its lastUpdated is older than this many days (API check only). +STALE_DAYS="${STALE_DAYS:-180}" +# Repos exempt from needing their own IT page (mirrors compliance-drift's exemption style). +DOC_EXEMPT_REPOS="${DOC_EXEMPT_REPOS:-engineering-handbook}" +# Required standing/runbook pages every IT space must document (page-map keys). +REQUIRED_PAGES="${REQUIRED_PAGES:-Incident Response Runbooks,Backup & Disaster Recovery,IAM & Access Management}" + +REFRESH=0 # --refresh: re-discover + re-mirror via substrate (network). Default: reuse mirrors. +DO_API=1 # --no-api: skip the LIVE Confluence API checks (offline). +DRY_RUN=0 # --dry-run: compose any digest but DO NOT post/write (recommend-only). +CANARY=0 # --canary: run against the fixture + assert the known gap count. +TARGETS_OVERRIDE="" # --targets "p1 p2": use these repo names instead of the mirror set. + +usage() { + cat >&2 < check SKIPPED + --refresh re-discover + re-mirror via the shared substrate before scanning (network) + --targets "a b" use these repo names instead of \$MIRROR_DIR/* (no clone) + -h|--help this help + +Env: GH_ORG MIRROR_DIR REPORT_ROOT PAGE_MAP_FILE AWS_INVENTORY_FILE STALE_DAYS + CONFLUENCE_BASE_URL CONFLUENCE_EMAIL CONFLUENCE_API_TOKEN (confluence-bot, D6) + DOC_EXEMPT_REPOS REQUIRED_PAGES SLACK_WEBHOOK_URL +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --canary) CANARY=1; DRY_RUN=1; DO_API=0 ;; + --dry-run) DRY_RUN=1 ;; + --no-api) DO_API=0 ;; + --page-map) shift; PAGE_MAP_FILE="${1:-}" ;; + --aws-inventory) shift; AWS_INVENTORY_FILE="${1:-}" ;; + --refresh) REFRESH=1 ;; + --targets) shift; TARGETS_OVERRIDE="${1:-}" ;; + -h|--help) usage; exit 0 ;; + *) die "unknown arg: $1 (see --help)" ;; + esac + shift +done + +command -v jq >/dev/null || die "jq is required" +command -v git >/dev/null || die "git is required" + +# --- Report dir (mode 600 reports; matches sweep conventions) ----------------- +umask 077 +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +REPORT_DIR="$REPORT_ROOT/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope +SWEEP_LOG="$REPORT_DIR/confluence-doc.log" +REPORT_JSON="$REPORT_DIR/confluence-doc.json" +REPORT_TXT="$REPORT_DIR/confluence-doc.txt" + +log "=== confluence-doc $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API refresh=$REFRESH) ===" + +# ------------------------------------------------------------------------------ +# GAPS (spirit of finding.schema.json so the coordinator + plan-groomer can consume them like +# any finding). category="other" (a doc gap is not a security category). status="confirmed" +# only for deterministic facts: a repo absent from the supplied map, an AWS resource absent +# from the supplied inventory-vs-map diff, a required page missing from the map, or an explicit +# API 404 (mapped page gone). A SKIPPED API check is NEVER a gap (feedback_cloudwatch_alarms). +# ------------------------------------------------------------------------------ +declare -a GAPS=() +add_gap() { # subject id title severity check proof + local subject="$1" id="$2" title="$3" sev="$4" check="$5" proof="$6" + GAPS+=( "$(jq -n \ + --arg repo "$subject" --arg id "$id" --arg title "$title" --arg sev "$sev" \ + --arg check "$check" --arg proof "$proof" \ + '{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other", + check:$check, status:"confirmed", recommendation:$proof}')" ) +} +declare -a SKIPPED_CHECKS=() # (subject:reason) checks skipped on missing data — never a gap +note_skip() { SKIPPED_CHECKS+=( "$1" ); } + +in_csv() { # needle csv -> 0 if present + local n="$1" csv="$2"; case ",$csv," in *",$n,"*) return 0 ;; *) return 1 ;; esac +} + +# --- Page-map lookup: is there a page whose key (page title) matches NAME? ----- +# The map is a JSON object {"": , ...} (the canary mock + the real +# project_confluence_migration export share this shape). A repo "documented" if a page title +# contains the repo name (case-insensitive), since IT pages are titled e.g. "Payments Dashboard" +# for repo "payments-dashboard". +map_has_page_for_repo() { # repo + local repo="$1" + # normalize repo (kebab) -> a loose token to match against page titles + local needle; needle="$(echo "$repo" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]')" + jq -e --arg n "$needle" ' + (keys // [])[] | (ascii_downcase | gsub("[^a-z0-9]";"")) | select(contains($n)) + ' "$PAGE_MAP_FILE" >/dev/null 2>&1 +} +map_has_exact_key() { # exact page title + local key="$1" + jq -e --arg k "$key" 'has($k)' "$PAGE_MAP_FILE" >/dev/null 2>&1 +} + +# ============================================================================== +# CONFLUENCE API (LIVE reads; need the confluence-bot creds; skipped offline/--no-api/--canary) +# ============================================================================== +# Confluence auth seam: OAuth 2.0 client-credentials (org service account, 2LO) OR +# Basic auth (email + API token). conf_api_init() resolves ONE mode (fetching a +# 60-min Bearer + the cloudId for OAuth); conf_get() does the authenticated GET +# with the right base + header. OAuth wins when its creds are present. Any +# failure (no cloudId, token request fails) returns non-zero so the caller SKIPS +# the live checks — never a false alarm on missing data. +# ============================================================================== +_CONF_MODE=""; _CONF_BASE=""; _CONF_BEARER="" + +conf_api_init() { + if [ -n "$CONFLUENCE_OAUTH_CLIENT_ID" ] && [ -n "$CONFLUENCE_OAUTH_CLIENT_SECRET" ]; then + # 2LO client-credentials token FIRST (the secret goes in the request BODY via + # --data-urlencode and is never echoed/logged — matches the existing -u risk class). + local tok + tok="$(curl -sS -X POST "$CONFLUENCE_OAUTH_TOKEN_URL" \ + -H 'Content-Type: application/x-www-form-urlencoded' \ + --data-urlencode "client_id=$CONFLUENCE_OAUTH_CLIENT_ID" \ + --data-urlencode "client_secret=$CONFLUENCE_OAUTH_CLIENT_SECRET" \ + --data-urlencode 'grant_type=client_credentials' \ + 2>>"$REPORT_DIR/confluence-api.log" | jq -r '.access_token // empty' 2>/dev/null)" + [ -n "$tok" ] || { log "OAuth: token request failed — skipping API (no false alarm)"; return 1; } + # Resolve the cloudId: use CONFLUENCE_CLOUD_ID if given, else the OAuth-native + # accessible-resources endpoint (the public /_edge/tenant_info is not reliable). + # Prefer the resource whose url matches the configured site; else the first. + local cid="$CONFLUENCE_CLOUD_ID" + if [ -z "$cid" ]; then + cid="$(curl -sS -H "Authorization: Bearer $tok" -H 'Accept: application/json' \ + 'https://api.atlassian.com/oauth/token/accessible-resources' \ + 2>>"$REPORT_DIR/confluence-api.log" \ + | jq -r --arg url "$CONFLUENCE_BASE_URL" \ + '(map(select(.url==$url)) | .[0].id) // .[0].id // empty' 2>/dev/null)" + fi + [ -n "$cid" ] || { log "OAuth: could not resolve cloudId (set CONFLUENCE_CLOUD_ID) — skipping API"; return 1; } + _CONF_MODE="oauth"; _CONF_BEARER="$tok" + _CONF_BASE="https://api.atlassian.com/ex/confluence/$cid" + return 0 + fi + if [ -n "$CONFLUENCE_BASE_URL" ] && [ -n "$CONFLUENCE_EMAIL" ] \ + && [ -n "$CONFLUENCE_API_TOKEN" ]; then + _CONF_MODE="basic"; _CONF_BASE="$CONFLUENCE_BASE_URL" + return 0 + fi + return 1 +} + +conf_get() { # path_suffix outfile -> echoes http_code (both modes share /wiki/api/v2/...) + local path="$1" out="$2" + if [ "$_CONF_MODE" = "oauth" ]; then + curl -sS -o "$out" -w '%{http_code}' \ + -H "Authorization: Bearer $_CONF_BEARER" -H 'Accept: application/json' \ + "$_CONF_BASE$path" 2>>"$REPORT_DIR/confluence-api.log" || echo 000 + else + curl -sS -o "$out" -w '%{http_code}' \ + -u "$CONFLUENCE_EMAIL:$CONFLUENCE_API_TOKEN" -H 'Accept: application/json' \ + "$_CONF_BASE$path" 2>>"$REPORT_DIR/confluence-api.log" || echo 000 + fi +} + +# ============================================================================== +# Confirm a mapped page still exists and is not stale. Status-code-aware, mirroring +# compliance-drift's branch-protection pattern exactly: +# 200 -> parse lastUpdated, flag if older than STALE_DAYS +# 404 -> a mapped page that is GONE -> that IS a confirmed gap +# anything else (401/403/5xx/000 transient) -> SKIP with NO gap (no false alarm on no-data) +conf_check_page() { # page_title page_id + local title="$1" pid="$2" + local tmp code body + tmp="$(mktemp)" + code="$(conf_get "/wiki/api/v2/pages/$pid?body-format=storage" "$tmp")" + body="$(cat "$tmp" 2>/dev/null)"; rm -f "$tmp" + case "$code" in + 200) + local updated upd_epoch now_epoch age_days + updated="$(echo "$body" | jq -r '.version.createdAt // .createdAt // empty' 2>/dev/null)" + [ -n "$updated" ] || { note_skip "$title:staleness(no-timestamp)"; return; } + upd_epoch="$(to_epoch "${updated%%T*}")"; now_epoch="$(date -u +%s)" + [ "$upd_epoch" -gt 0 ] || { note_skip "$title:staleness(unparseable-date)"; return; } + age_days=$(( (now_epoch - upd_epoch) / 86400 )) + if [ "$age_days" -gt "$STALE_DAYS" ]; then + add_gap "$title" "stale-page" \ + "Page '$title' is stale (last updated ${age_days}d ago, > ${STALE_DAYS}d)" "low" "stale-page" \ + "review + refresh the IT page; docs must track the system (global CLAUDE.md docs obligation)" + fi + ;; + 404) + add_gap "$title" "page-gone" \ + "Mapped page '$title' (id $pid) returns 404 — page deleted/moved" "high" "page-existence" \ + "the page-ID map points at a non-existent page; fix the map or restore the page" + ;; + *) note_skip "$title:api(http-$code)" ;; # transient/forbidden -> NO gap on missing data + esac +} + +# ============================================================================== +# TARGET RESOLUTION (repo set + map + inventory) +# ============================================================================== +declare -a REPO_NAMES=() + +if [ "$CANARY" -eq 1 ]; then + FIXTURE_ROOT="$HERE/fixtures/confluence-doc" + [ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT" + PAGE_MAP_FILE="$FIXTURE_ROOT/mock-page-map.json" + AWS_INVENTORY_FILE="$FIXTURE_ROOT/mock-aws-inventory.json" + [ -f "$PAGE_MAP_FILE" ] || die "canary mock page-map missing: $PAGE_MAP_FILE" + [ -f "$AWS_INVENTORY_FILE" ] || die "canary mock aws inventory missing: $AWS_INVENTORY_FILE" + # Pin the exception + required-page lists the fixture was authored against (deterministic). + DOC_EXEMPT_REPOS="engineering-handbook" + REQUIRED_PAGES="Incident Response Runbooks,Backup & Disaster Recovery,IAM & Access Management" + STALE_DAYS="180" + # The fixture repo set is a newline-delimited list (no git checkout needed — confluence-doc + # diffs NAMES against the map, it does not scan repo contents). + while IFS= read -r r; do + r="$(echo "$r" | tr -d '[:space:]')"; [ -n "$r" ] && REPO_NAMES+=( "$r" ) + done < "$FIXTURE_ROOT/repos.txt" + log "canary: ${#REPO_NAMES[@]} fixture repo(s); mock map + mock inventory" +elif [ -n "$TARGETS_OVERRIDE" ]; then + # shellcheck disable=SC2206 # intentional word-split of the space-separated --targets list + arr=( $TARGETS_OVERRIDE ) + for p in "${arr[@]}"; do nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); done + log "explicit targets: ${REPO_NAMES[*]}" +else + if [ "$REFRESH" -eq 1 ]; then + [ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN" + command -v curl >/dev/null || die "--refresh needs curl" + mkdir -p "$MIRROR_DIR" + log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)" + DISCOVERED="$REPORT_DIR/discovered.tsv" + if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then + while IFS=$'\t' read -r name url branch; do + [ -n "$name" ] || continue + mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)" + done < "$DISCOVERED" + else + log "discovery failed — falling back to existing mirrors (coverage may be stale)" + fi + fi + [ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)" + for d in "$MIRROR_DIR"/*/; do + [ -d "$d/.git" ] || continue + nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ) + done + log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)" +fi + +[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to diff" +[ -n "$PAGE_MAP_FILE" ] || die "no page-ID map (--page-map PATH or \$PAGE_MAP_FILE); cannot diff repos vs Confluence" +[ -f "$PAGE_MAP_FILE" ] || die "page-ID map not found: $PAGE_MAP_FILE" +jq -e 'type=="object"' "$PAGE_MAP_FILE" >/dev/null 2>&1 || die "page-ID map is not a JSON object: $PAGE_MAP_FILE" + +# Decide whether the LIVE Confluence API runs: need curl, API enabled, not offline +# canary, AND an auth mode that initializes (OAuth service account or Basic). A +# token/cloudId failure leaves RUN_API=0 → checks skipped, NO false alarm. +RUN_API=0 +if [ "$DO_API" -eq 1 ] && command -v curl >/dev/null && conf_api_init; then + RUN_API=1 + log "Confluence API: ${_CONF_MODE} auth ready" +elif [ "$DO_API" -eq 1 ]; then + log "Confluence API requested but confluence-bot creds/curl unavailable — skipping live checks (no false alarms on missing data; the service account is gated provisioning)." +fi + +# ============================================================================== +# CHECK 1 — REPO SET vs page-ID map (every non-exempt repo SHOULD have an IT page) +# ============================================================================== +for nm in "${REPO_NAMES[@]}"; do + in_csv "$nm" "$DOC_EXEMPT_REPOS" && { note_skip "$nm:repo-page(doc-exempt)"; continue; } + if ! map_has_page_for_repo "$nm"; then + add_gap "$nm" "no-it-page" \ + "Repo '$nm' has no Confluence IT page in the page-ID map" "medium" "repo-documented" \ + "create an IT page for '$nm' (sh-confluence) and add it to project_confluence_migration" + fi +done + +# ============================================================================== +# CHECK 2 — AWS INVENTORY vs page-ID map (optional; absent file => SKIP, never a gap) +# ============================================================================== +if [ -n "$AWS_INVENTORY_FILE" ] && [ -f "$AWS_INVENTORY_FILE" ]; then + if jq -e '.resources | type=="array"' "$AWS_INVENTORY_FILE" >/dev/null 2>&1; then + # Each resource SHOULD be represented on a page in the map (by name token match). + while IFS= read -r res; do + [ -n "$res" ] || continue + rname="$(echo "$res" | jq -r '.name // empty')" + rtype="$(echo "$res" | jq -r '.type // "resource"')" + [ -n "$rname" ] || continue + needle="$(echo "$rname" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]')" + if ! jq -e --arg n "$needle" ' + (keys // [])[] | (ascii_downcase | gsub("[^a-z0-9]";"")) | select(contains($n)) + ' "$PAGE_MAP_FILE" >/dev/null 2>&1; then + add_gap "$rname" "aws-not-in-map" \ + "AWS $rtype '$rname' is not represented in the IT page-ID map / architecture map" "medium" "aws-documented" \ + "add '$rname' to the AWS Architecture Map (page 1540098) + an IT page; Mermaid edits via confluence_mermaid.py (on-demand path, provisioning)" + fi + done < <(jq -c '.resources[]' "$AWS_INVENTORY_FILE") + else + note_skip "aws-inventory:malformed(no-resources-array)" + fi +else + note_skip "aws-inventory:absent(check-skipped)" # missing inventory -> SKIP, never a gap +fi + +# ============================================================================== +# CHECK 3 — REQUIRED standing/runbook pages present in the map +# ============================================================================== +IFS=',' read -r -a req_arr <<< "$REQUIRED_PAGES" +for page in "${req_arr[@]}"; do + page="$(echo "$page" | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//')" + [ -n "$page" ] || continue + if ! map_has_exact_key "$page"; then + add_gap "$page" "missing-runbook" \ + "Required page '$page' is missing from the IT page-ID map" "high" "required-page" \ + "create the '$page' page in the IT space and add it to project_confluence_migration" + fi +done + +# ============================================================================== +# CHECK 4 — LIVE API: mapped pages still exist + are not stale (skipped offline/--no-api/--canary) +# ============================================================================== +if [ "$RUN_API" -eq 1 ]; then + while IFS=$'\t' read -r ptitle pid; do + [ -n "$pid" ] || continue + case "$pid" in ''|*[!0-9]*) note_skip "$ptitle:api(non-numeric-id)"; continue ;; esac + conf_check_page "$ptitle" "$pid" + done < <(jq -r 'to_entries[] | [.key, (.value|tostring)] | @tsv' "$PAGE_MAP_FILE") +else + note_skip "confluence-api:not-run(creds-absent-or-offline)" +fi + +# ============================================================================== +# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to the other checkers) +# ============================================================================== +if [ "${#GAPS[@]}" -gt 0 ]; then + GAPS_JSON="$(printf '%s\n' "${GAPS[@]}" | jq -cs .)" +else + GAPS_JSON="[]" +fi +if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then + SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)" +else + SKIPPED_JSON="[]" +fi + +N_GAPS="$(echo "$GAPS_JSON" | jq 'length')" +N_HIGH="$(echo "$GAPS_JSON" | jq '[.[]|select(.severity=="high" or .severity=="critical")] | length')" +N_SUBJECTS="$(echo "$GAPS_JSON" | jq '[.[].repo] | unique | length')" + +jq -n \ + --arg checker "confluence-doc" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \ + --argjson api "$RUN_API" --argjson reposn "${#REPO_NAMES[@]}" \ + --argjson gaps "$GAPS_JSON" --argjson skipped "$SKIPPED_JSON" \ + '{checker:$checker, generated:$ts, org:$org, mode:"recommend-only", + api_checks_ran:($api==1), repos_diffed:$reposn, + gap_count:($gaps|length), + subjects_with_gaps:([$gaps[].repo]|unique|length), + findings:$gaps, skipped_checks:$skipped}' > "$REPORT_JSON" + +{ + echo "confluence-doc — documentation gap report — $UTC_STAMP" + echo "org=$GH_ORG repos_diffed=${#REPO_NAMES[@]} api_checks=$([ "$RUN_API" -eq 1 ] && echo on || echo off) mode=recommend-only (D7)" + echo "doc gaps: $N_GAPS ($N_HIGH high) across $N_SUBJECTS subject(s)" + echo + if [ "$N_GAPS" -gt 0 ]; then + echo "RECOMMENDATIONS (recommend-only — NEVER auto-written, D7):" + echo "$GAPS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n recommend: \(.recommendation)"' + else + echo "No documentation gaps detected this run." + fi + if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then + echo; echo "skipped checks (missing data — NOT counted as a gap):" + echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"' + fi +} > "$REPORT_TXT" +chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true + +log "report: $REPORT_JSON ($N_GAPS gap(s), $N_SUBJECTS subject(s))" + +# ============================================================================== +# CANARY ASSERTION (anti-complacency floor, design §6.4) +# ============================================================================== +if [ "$CANARY" -eq 1 ]; then + EXPECT_FILE="$HERE/fixtures/confluence-doc/EXPECTED_GAP_COUNT" + [ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE" + EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")" + log "canary assertion: expected gaps=$EXPECTED, got=$N_GAPS" + if [ "$N_GAPS" -ne "$EXPECTED" ]; then + echo "[confluence-doc] CANARY FAIL: doc-gap count mismatch (expected $EXPECTED, got $N_GAPS)" >&2 + echo " -> a gap check regressed (stopped firing) or the fixture changed. See $REPORT_TXT." >&2 + exit 3 + fi + log "canary PASS: all $EXPECTED planted doc gaps detected." +fi + +# ============================================================================== +# RECOMMEND-ONLY ROUTING (D3/D7): gaps live in the mode-600 report. Post NOTHING by default. +# Scheduled mode NEVER auto-writes Confluence; alarming is reserved for confirmed criticals via +# the coordinator's shared routing (kept ALARM-only there). Here, recommend-only = report-only. +# ============================================================================== +if [ "$N_GAPS" -eq 0 ]; then + log "no doc gaps — recommend-only report written; posting NOTHING (D7)." + exit 0 +fi + +# Compose a redacted digest for the report/log (defense-in-depth); do NOT post by default. +DIGEST="$(echo "$GAPS_JSON" | jq -r ' + group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' \ + | sed 's/^/• /' | redact)" +echo "$DIGEST" >&2 +log "DRY-RUN/RECOMMEND-ONLY: $N_GAPS gap(s) written to the mode-600 report; nothing posted, nothing written to Confluence (D7)." +exit 0 + +# ============================================================================== +# PROVISIONING (NOT DONE HERE — gated): +# - confluence-bot SERVICE ACCOUNT (D6): create a dedicated Atlassian service account scoped +# to EDIT the IT space ONLY (Confluence API tokens inherit the whole user's permissions, so a +# scoped service account bounds blast radius; costs one Confluence seat). Mint its API token, +# store it in ~/secrev.env (mode 600) as CONFLUENCE_API_TOKEN (+ CONFLUENCE_BASE_URL/EMAIL). +# Rotate the token on a 90-DAY cadence. Until this exists, the LIVE API checks SKIP (above), +# never alarm. This whole step is gated (Adam-provisioned), not done by this script. +# - LIVE Confluence READ checks (page-existence + staleness) only run once those creds exist. +# - ON-DEMAND WRITE path (D7) — the actual Confluence update, including Mermaid architecture-map +# edits via ~/.claude/scripts/confluence_mermaid.py — is a SEPARATE, LATER, SSH-invoked path. +# Before any --apply, that script must pass a LIVE DRY-RUN against page 1540098: verify it +# lists all 16 weweave Mermaid macros and that a no-op set produces a clean (empty) revert-diff. +# ADF-only + macro-count + revert-diff guards are load-bearing (a full-body markdown round-trip +# has SILENTLY DELETED every diagram on 1540098 before). This script NEVER calls --apply. +# - No systemd unit / timer is installed here. Wiring the scheduled run (weekly) under the +# coordinator is provisioning and is gated. +# - The coordinator (design §5, checker_coordinator.sh) registers + drives this checker; that +# registry edit is done centrally, NOT in this script. +# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations for the +# build session, tracked outside this script. +# ============================================================================== diff --git a/checkers/dependency-cve.sh b/checkers/dependency-cve.sh new file mode 100755 index 0000000..8a2a35c --- /dev/null +++ b/checkers/dependency-cve.sh @@ -0,0 +1,587 @@ +#!/usr/bin/env bash +# dependency-cve.sh — Plane-1 / Tier-1 checker for the R720 agent-team. +# +# Design refs: docs/r720-agent-team-design.md §4 (Tier 1 roster: dependency-cve — +# "Cross-ref lockfiles vs advisories org-wide; report + feed fixer. Complements Dependabot") +# and §7 Phase 2 ("coordinator + second checker"). This is the SECOND Plane-1 checker built +# on the Phase-0 shared substrate (lib/sweep_substrate.sh); it mirrors compliance-drift.sh's +# conventions verbatim so the coordinator (§5) can drive both identically. +# +# WHAT IT DOES (read-only): +# Scans the SAME shallow clean clones nightly_sweep.sh already produced in $MIRROR_DIR — it +# does NOT re-clone (mirrors-first; an optional --refresh re-runs discovery+mirror via the +# shared substrate). In each mirror it parses dependency lockfiles/manifests with PINNED, +# exact versions, extracts (ecosystem, package, version) tuples, and cross-references them +# against the OSV advisory database to flag known-vulnerable pinned deps. This complements +# Dependabot (design §4): it is org-wide, runs on the server-side mirrors, and feeds the +# fixer queue in a later phase. +# +# Manifests parsed (and the OSV ecosystem each maps to): +# requirements.txt -> PyPI (only EXACT '==' pins; ranges/unpinned are skipped) +# poetry.lock -> PyPI ([[package]] name/version blocks) +# Pipfile.lock -> PyPI (default+develop, "==x.y.z" version strings) +# package-lock.json -> npm (packages[].version / dependencies[].version) +# yarn.lock -> npm ("pkg@range:\n version \"x\"" stanzas) +# packages.lock.json -> NuGet (.dependencies[tfm][pkg].resolved) +# *.csproj -> NuGet () +# Only EXACTLY-pinned versions are cross-referenced (an unpinned/range spec has no single +# version to query and is not a confirmed vulnerable artifact — no false alarms on no-data, +# memory feedback_cloudwatch_alarms). +# +# ADVISORY SOURCE (live): OSV batch API POST https://api.osv.dev/v1/querybatch (NO auth token). +# Guarded behind a --no-api / offline check exactly like compliance-drift's GitHub-API checks: +# on missing curl OR a failed/empty network response, the API lookup is SKIPPED and noted in +# the report — a vuln is NEVER reported on missing advisory data. Network calls are minimal +# (one batched POST) and fail-safe. +# +# AGENTIC TIEBREAK (design §4, "Claude + GPT tiebreak"): OPTIONAL and only relevant in LIVE mode +# for ambiguous severity. For THIS phase the deterministic OSV core is the whole checker — NO +# LLM is invoked in --canary/--dry-run. A clearly-marked inert stub hook (maybe_tiebreak) marks +# the future seam; it does nothing offline and nothing in this phase. +# +# CANARY / DRY-RUN (offline, no network, no token): +# --canary runs against a planted fixture (checkers/fixtures/dependency-cve/) and asserts the +# known vuln count against EXPECTED_VULN_COUNT (exit 3 on mismatch). Because OSV needs network, +# the canary consults a LOCAL offline advisory fixture (fixtures/dependency-cve/osv-advisories.json) +# INSTEAD of the network — so it is fully offline + deterministic. --canary implies --dry-run + +# --no-api. This is the anti-complacency floor (design §6.4) AND the routing dry-run (§7 Phase 2): +# with --dry-run the Slack alarm is composed + printed but NOT POSTed. +# +# SCOPE / SAFETY: +# Read-only. Fixtures ship git metadata as dotgit/ (renamed to .git/ at run time) so they +# commit into THIS repo without becoming submodules — the SAME trick compliance-drift uses. +# Does NOT touch agent_team/ or agent-team/, and is NOT wired into systemd — that is Phase-6 +# provisioning (gated). See the "PROVISIONING (NOT DONE HERE)" note at the bottom. +# +# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[dependency-cve] $*" >&2; } +die() { echo "[dependency-cve] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate --------------------------------------------------------- +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSTRATE="$HERE/../lib/sweep_substrate.sh" +[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE" +# shellcheck source=../lib/sweep_substrate.sh +. "$SUBSTRATE" + +# --- Config + defaults (env, all optional) ------------------------------------ +GH_ORG="${GH_ORG:-Sea-Haven-Industries}" +MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}" +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/dependency-cve}" +OSV_BATCH_URL="${OSV_BATCH_URL:-https://api.osv.dev/v1/querybatch}" + +REFRESH=0 # --refresh: re-run discovery+mirror via substrate (network). Default: reuse mirrors. +DO_API=1 # --no-api: skip the OSV advisory lookup (offline). Without it, nothing matches. +DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run). +CANARY=0 # --canary: run against the planted fixture + assert the known vuln count. +TARGETS_OVERRIDE="" # --targets "p1 p2": scan explicit dirs instead of the mirror set. +ADVISORIES_FILE="" # --advisories-file PATH: consult a local advisory JSON instead of the OSV API. + +usage() { + cat >&2 </dev/null || die "jq is required" +command -v git >/dev/null || die "git is required" + +# --- Report dir (mode 600 reports; matches sweep conventions) ----------------- +umask 077 +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +REPORT_DIR="$REPORT_ROOT/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope +SWEEP_LOG="$REPORT_DIR/dependency-cve.log" # name the substrate's post_slack_alarm() references +REPORT_JSON="$REPORT_DIR/dependency-cve.json" +REPORT_TXT="$REPORT_DIR/dependency-cve.txt" + +log "=== dependency-cve $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API refresh=$REFRESH) ===" + +# ------------------------------------------------------------------------------ +# FINDINGS (spirit of finding.schema.json so the coordinator can route like an agentic +# finding). category="other" (a vulnerable-dependency is not one of the schema's security +# categories); status="confirmed" only for an exact pinned version that MATCHES an advisory. +# A pinned dep with NO advisory match is NOT a finding; an unqueryable/skipped advisory lookup +# is NOT a finding (memory feedback_cloudwatch_alarms: no false alarms on missing data). +# ------------------------------------------------------------------------------ +declare -a FINDINGS=() +add_finding() { # repo id title severity pkg version advisory_id summary fixed_version + local repo="$1" id="$2" title="$3" sev="$4" pkg="$5" ver="$6" adv="$7" summ="$8" fixed="$9" + FINDINGS+=( "$(jq -n \ + --arg repo "$repo" --arg id "$id" --arg title "$title" --arg sev "$sev" \ + --arg pkg "$pkg" --arg ver "$ver" --arg adv "$adv" --arg summ "$summ" --arg fixed "$fixed" \ + '{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other", + check:"vulnerable-dependency", status:"confirmed", + proof:{package:$pkg, version:$ver, advisory_id:$adv, summary:$summ, fixed_version:$fixed}}')" ) +} +declare -a SKIPPED_CHECKS=() # (repo:reason) lookups skipped on missing data — reported, never alarmed +note_skip() { SKIPPED_CHECKS+=( "$1" ); } + +# Severity normalizer: map OSV/GHSA strings + CVSS scores into the schema's enum. +norm_sev() { # raw_severity cvss_score -> critical|high|medium|low + local raw; raw="$(echo "${1:-}" | tr '[:upper:]' '[:lower:]')" + local cvss="${2:-}" + case "$raw" in + critical) echo critical; return ;; + high) echo high; return ;; + moderate|medium) echo medium; return ;; + low) echo low; return ;; + esac + # Fall back to CVSS base score banding (NVD/CVSSv3 thresholds). + if [ -n "$cvss" ] && [ "$cvss" != "null" ]; then + awk -v c="$cvss" 'BEGIN{ + if (c+0>=9.0) print "critical"; + else if (c+0>=7.0) print "high"; + else if (c+0>=4.0) print "medium"; + else print "low"; }' + return + fi + echo medium # unknown severity: medium (a real match we cannot rank), never dropped +} + +# ============================================================================== +# MANIFEST PARSERS — each emits "ECOSYSTEMpackageversion" lines (exact pins only). +# Pure text/jq parsing; no project tooling invoked. Unknown/odd lines are skipped silently. +# ============================================================================== + +# requirements.txt: only EXACT '==' pins (skip ranges, markers, comments, -e/-r includes, extras). +parse_requirements() { # file + local f="$1" + sed -E 's/[[:space:]]*#.*$//' "$f" 2>/dev/null \ + | grep -E '==' \ + | while IFS= read -r line; do + line="$(echo "$line" | tr -d '[:space:]')" + [ -n "$line" ] || continue + case "$line" in -*|.*|git+*|http*) continue ;; esac + # strip extras: pkg[extra]==1.2.3 -> pkg + local name ver + name="$(echo "$line" | sed -E 's/\[[^]]*\].*//; s/[<>=!~;].*$//')" + ver="$(echo "$line" | sed -E 's/^[^=]*==//; s/[ ;].*$//')" + # only a clean exact version (digits/dots/alnum), no range operators left + case "$ver" in *','*|*'<'*|*'>'*|*'*'*|'') continue ;; esac + [ -n "$name" ] && [ -n "$ver" ] && printf 'PyPI\t%s\t%s\n' "$name" "$ver" + done +} + +# poetry.lock: [[package]] blocks with name = "x" / version = "y". +parse_poetry_lock() { # file + local f="$1" + awk ' + /^\[\[package\]\]/ { name=""; ver=""; next } + /^name = / { gsub(/^name = "|"$/,""); name=$0; next } + /^version = / { gsub(/^version = "|"$/,""); ver=$0; + if (name!="" && ver!="") printf "PyPI\t%s\t%s\n", name, ver; next } + ' "$f" 2>/dev/null +} + +# Pipfile.lock: JSON; default + develop maps; versions look like "==1.2.3". +parse_pipfile_lock() { # file + local f="$1" + jq -r ' + (.default // {}) * (.develop // {}) | to_entries[] + | select(.value.version != null) + | .key as $n | (.value.version | sub("^=="; "")) as $v + | select($v | test("^[0-9][0-9A-Za-z.+-]*$")) + | "PyPI\t\($n)\t\($v)" + ' "$f" 2>/dev/null || true +} + +# package-lock.json: prefer v2/v3 .packages (node_modules/ keys), else v1 .dependencies. +parse_package_lock() { # file + local f="$1" + jq -r ' + if (.packages != null) then + (.packages | to_entries[] + | select(.key | startswith("node_modules/")) + | select(.value.version != null) + | (.key | sub("^.*node_modules/"; "")) as $n + | "npm\t\($n)\t\(.value.version)") + elif (.dependencies != null) then + [paths(objects | has("version")) as $p | {n: $p[-1], v: (getpath($p).version)}] + | .[] | select(.v != null) | "npm\t\(.n)\t\(.v)" + else empty end + ' "$f" 2>/dev/null || true +} + +# yarn.lock: stanzas "spec@range, spec@range:\n version \"x.y.z\"". +parse_yarn_lock() { # file + local f="$1" + awk ' + /^[^[:space:]#].*:[[:space:]]*$/ { + # header line: take first spec, strip trailing colon + quotes, derive package name + hdr=$0; sub(/:[[:space:]]*$/,"",hdr); + split(hdr, specs, ", "); first=specs[1]; gsub(/"/,"",first); + # package name = everything before the LAST @ (handles @scope/pkg@range) + at=0; for (i=2;i<=length(first);i++){ if (substr(first,i,1)=="@") at=i } + pkg=(at>1)? substr(first,1,at-1) : first; + next + } + /^[[:space:]]+version / { + v=$0; gsub(/^[[:space:]]+version[[:space:]]+"?|"?[[:space:]]*$/,"",v); + if (pkg!="" && v!="") printf "npm\t%s\t%s\n", pkg, v; + pkg=""; next + } + ' "$f" 2>/dev/null +} + +# packages.lock.json (NuGet): .dependencies[tfm][pkg].resolved. +parse_packages_lock() { # file + local f="$1" + jq -r ' + (.dependencies // {}) | to_entries[] | .value | to_entries[] + | select(.value.resolved != null) + | "NuGet\t\(.key)\t\(.value.resolved)" + ' "$f" 2>/dev/null || true +} + +# *.csproj (NuGet): . +parse_csproj() { # file + local f="$1" + grep -oE ']*>' "$f" 2>/dev/null \ + | while IFS= read -r tag; do + local inc ver + inc="$(echo "$tag" | sed -nE 's/.*Include="([^"]+)".*/\1/p')" + ver="$(echo "$tag" | sed -nE 's/.*Version="([^"]+)".*/\1/p')" + # only exact versions (no range brackets/commas/wildcards) + case "$ver" in ''|*'['*|*']'*|*'('*|*')'*|*','*|*'*'*) continue ;; esac + [ -n "$inc" ] && [ -n "$ver" ] && printf 'NuGet\t%s\t%s\n' "$inc" "$ver" + done +} + +# Extract ALL (ecosystem, package, version) tuples from one repo dir. Dedup at the end. +extract_deps() { # repo_dir -> TSV "ECOSYSTEM\tpackage\tversion" on stdout + local dir="$1" f + # requirements.txt (any depth, excluding .git) + while IFS= read -r f; do [ -n "$f" ] && parse_requirements "$f"; done \ + < <(find "$dir" -maxdepth 4 -name requirements.txt -not -path '*/.git/*' 2>/dev/null) + while IFS= read -r f; do [ -n "$f" ] && parse_poetry_lock "$f"; done \ + < <(find "$dir" -maxdepth 4 -name poetry.lock -not -path '*/.git/*' 2>/dev/null) + while IFS= read -r f; do [ -n "$f" ] && parse_pipfile_lock "$f"; done \ + < <(find "$dir" -maxdepth 4 -name Pipfile.lock -not -path '*/.git/*' 2>/dev/null) + while IFS= read -r f; do [ -n "$f" ] && parse_package_lock "$f"; done \ + < <(find "$dir" -maxdepth 4 -name package-lock.json -not -path '*/.git/*' 2>/dev/null) + while IFS= read -r f; do [ -n "$f" ] && parse_yarn_lock "$f"; done \ + < <(find "$dir" -maxdepth 4 -name yarn.lock -not -path '*/.git/*' 2>/dev/null) + while IFS= read -r f; do [ -n "$f" ] && parse_packages_lock "$f"; done \ + < <(find "$dir" -maxdepth 4 -name packages.lock.json -not -path '*/.git/*' 2>/dev/null) + while IFS= read -r f; do [ -n "$f" ] && parse_csproj "$f"; done \ + < <(find "$dir" -maxdepth 4 -name '*.csproj' -not -path '*/.git/*' 2>/dev/null) +} + +# ============================================================================== +# ADVISORY LOOKUP +# ============================================================================== +# OFFLINE: consult a local advisory file (the canary fixture, or --advisories-file). Keyed by +# "ECOSYSTEM|package|version" -> array of {id,summary,severity,cvss,fixed_version}. Deterministic. +lookup_offline() { # advisories_file ecosystem package version -> advisory JSON array (or []) + local af="$1" eco="$2" pkg="$3" ver="$4" + jq -c --arg k "$eco|$pkg|$ver" '(.advisories[$k] // [])' "$af" 2>/dev/null || echo '[]' +} + +# LIVE: one batched POST to the OSV querybatch API (no token). Returns one results[] per query +# in input order. Fail-safe: on missing curl, transport failure, or a non-array body, returns "" +# (the caller then SKIPS — never alarms on missing advisory data). +osv_querybatch() { # queries_json (array of {package:{ecosystem,name},version}) -> results JSON or "" + local queries="$1" + command -v curl >/dev/null || { return 1; } + local body + body="$(curl -fsS -X POST -H 'Content-Type: application/json' \ + --max-time 30 \ + --data "$(jq -n --argjson q "$queries" '{queries:$q}')" \ + "$OSV_BATCH_URL" 2>>"$REPORT_DIR/osv.log")" || return 1 + echo "$body" | jq -e '.results | type=="array"' >/dev/null 2>&1 || return 1 + echo "$body" +} + +# Inert future seam (design §4 "Claude + GPT tiebreak"): in LIVE mode, an ambiguous-severity +# advisory could be escalated to a cross-family judge. This phase keeps the deterministic core +# ONLY — the stub does nothing and is never reached offline / in canary / dry-run. +maybe_tiebreak() { # advisory_json (no-op stub; phase-2 intentionally inert) + return 0 +} + +# ============================================================================== +# TARGET RESOLUTION +# ============================================================================== +declare -a REPO_NAMES=(); declare -A REPO_DIR=() + +if [ "$CANARY" -eq 1 ]; then + FIXTURE_ROOT="$HERE/fixtures/dependency-cve" + [ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT" + # The canary is OFFLINE: it consults the planted advisory fixture instead of the OSV network, + # unless an explicit --advisories-file override was given. + [ -n "$ADVISORIES_FILE" ] || ADVISORIES_FILE="$FIXTURE_ROOT/osv-advisories.json" + [ -f "$ADVISORIES_FILE" ] || die "canary advisory fixture missing: $ADVISORIES_FILE" + # Fixtures ship git metadata as dotgit/ (not .git/) so they are committable into THIS repo + # without becoming nested submodules. Materialize: copy + rename dotgit -> .git into a mode-700 + # temp area removed on exit (same trick as compliance-drift.sh). + FIXTURE_WORK="$(mktemp -d "${TMPDIR:-/tmp}/dependency-cve-canary.XXXXXX")" + trap 'rm -rf "$FIXTURE_WORK"' EXIT + log "canary: materializing planted fixtures from $FIXTURE_ROOT into $FIXTURE_WORK" + for d in "$FIXTURE_ROOT"/*/; do + [ -d "$d/dotgit" ] || continue # only fixture repos (skip README.md, *.json etc.) + nm="$(basename "$d")" + cp -R "$d" "$FIXTURE_WORK/$nm" + mv "$FIXTURE_WORK/$nm/dotgit" "$FIXTURE_WORK/$nm/.git" + # Manifests are stored as .fixture so GitHub's dependency graph / the + # dependency-review CI action does NOT parse the deliberately-vulnerable canary + # pins as real project dependencies. Restore their real names in the materialized + # work area so the checker's per-ecosystem parsers dispatch correctly (same + # committable-without-side-effects rationale as the dotgit/ rename above). + while IFS= read -r ff; do + [ -n "$ff" ] && mv "$ff" "${ff%.fixture}" + done < <(find "$FIXTURE_WORK/$nm" -name '*.fixture' -not -path '*/.git/*' 2>/dev/null) + REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$FIXTURE_WORK/$nm" + done +elif [ -n "$TARGETS_OVERRIDE" ]; then + # shellcheck disable=SC2206 # intentional word-split of the space-separated --targets list + arr=( $TARGETS_OVERRIDE ) + for p in "${arr[@]}"; do p="${p/#\~/$HOME}"; nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p"; done + log "explicit targets: ${REPO_NAMES[*]}" +else + if [ "$REFRESH" -eq 1 ]; then + [ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN" + command -v curl >/dev/null || die "--refresh needs curl" + mkdir -p "$MIRROR_DIR" + log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)" + DISCOVERED="$REPORT_DIR/discovered.tsv" + if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then + while IFS=$'\t' read -r name url branch; do + [ -n "$name" ] || continue + mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)" + done < "$DISCOVERED" + else + log "discovery failed — falling back to existing mirrors (coverage may be stale)" + fi + fi + # Default + post-refresh: enumerate EXISTING mirrors. No clone here — reuse the sweep's clones. + [ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)" + for d in "$MIRROR_DIR"/*/; do + [ -d "$d/.git" ] || continue + nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}" + done + log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)" +fi + +[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to scan" + +# Decide HOW advisories are looked up: offline file, or the live OSV API, or skip entirely. +# An explicit --advisories-file always wins (offline + deterministic, even without --canary). +ADV_MODE="none" +if [ -n "$ADVISORIES_FILE" ]; then + [ -f "$ADVISORIES_FILE" ] || die "advisories file not found: $ADVISORIES_FILE" + ADV_MODE="offline" +elif [ "$DO_API" -eq 1 ] && command -v curl >/dev/null; then + ADV_MODE="api" +elif [ "$DO_API" -eq 1 ]; then + log "OSV lookup requested but curl unavailable — skipping advisory match (no false alarms on missing data)" +fi +log "advisory mode: $ADV_MODE" + +# ============================================================================== +# RUN: extract deps per repo, then cross-reference against advisories +# ============================================================================== +for nm in "${REPO_NAMES[@]}"; do + dir="${REPO_DIR[$nm]}" + # Unique (ecosystem, package, version) tuples for this repo. + deps_tsv="$(extract_deps "$dir" | sort -u || true)" + ndeps=0; [ -n "$deps_tsv" ] && ndeps="$(printf '%s\n' "$deps_tsv" | grep -c . || true)" + log " [$nm] extracted $ndeps pinned dependency tuple(s)" + [ "$ndeps" -gt 0 ] || { note_skip "$nm:no-pinned-deps"; continue; } + + if [ "$ADV_MODE" = "none" ]; then + note_skip "$nm:advisory-lookup-skipped(offline/no-curl)" + continue + fi + + if [ "$ADV_MODE" = "offline" ]; then + # Deterministic local lookup, one tuple at a time. + while IFS=$'\t' read -r eco pkg ver; do + [ -n "$pkg" ] || continue + advs="$(lookup_offline "$ADVISORIES_FILE" "$eco" "$pkg" "$ver")" + cnt="$(echo "$advs" | jq 'length' 2>/dev/null || echo 0)" + [ "${cnt:-0}" -gt 0 ] || continue + i=0 + while [ "$i" -lt "$cnt" ]; do + adv="$(echo "$advs" | jq -c --argjson i "$i" '.[$i]')" + aid="$(echo "$adv" | jq -r '.id // "UNKNOWN"')" + summ="$(echo "$adv" | jq -r '.summary // ""')" + rawsev="$(echo "$adv"| jq -r '.severity // ""')" + cvss="$(echo "$adv" | jq -r '.cvss // empty')" + fixed="$(echo "$adv" | jq -r '.fixed_version // ""')" + sev="$(norm_sev "$rawsev" "$cvss")" + maybe_tiebreak "$adv" # inert in this phase + add_finding "$nm" "vuln-$(echo "${pkg}-${ver}-${aid}" | tr -c 'A-Za-z0-9-' '-')" \ + "$pkg $ver is vulnerable ($aid)" "$sev" \ + "$pkg" "$ver" "$aid" "$summ" "$fixed" + i=$((i+1)) + done + done <<< "$deps_tsv" + continue + fi + + # ADV_MODE = api: build ONE batched OSV query for all this repo's tuples (minimal network). + queries="$(printf '%s\n' "$deps_tsv" | jq -R -s ' + [ split("\n")[] | select(length>0) | split("\t") + | {package:{ecosystem:.[0], name:.[1]}, version:.[2]} ]')" + # Keep a parallel TSV array so we can re-associate results[] (OSV preserves input order). + if ! results="$(osv_querybatch "$queries")"; then + note_skip "$nm:osv-querybatch-failed" # transport/HTTP failure -> skip, NEVER alarm + continue + fi + # Walk each tuple alongside its result entry. + idx=0 + while IFS=$'\t' read -r eco pkg ver; do + [ -n "$pkg" ] || continue + vulns="$(echo "$results" | jq -c --argjson i "$idx" '(.results[$i].vulns // [])')" + idx=$((idx+1)) + vcnt="$(echo "$vulns" | jq 'length' 2>/dev/null || echo 0)" + [ "${vcnt:-0}" -gt 0 ] || continue + j=0 + while [ "$j" -lt "$vcnt" ]; do + v="$(echo "$vulns" | jq -c --argjson j "$j" '.[$j]')" + aid="$(echo "$v" | jq -r '.id // "UNKNOWN"')" + summ="$(echo "$v" | jq -r '.summary // (.details // "" | .[0:160])')" + # OSV severity: prefer database_specific.severity, else the CVSS vector score band. + rawsev="$(echo "$v" | jq -r '.database_specific.severity // ""')" + cvss="$(echo "$v" | jq -r '[.severity[]? | select(.type|test("CVSS")) | .score] | .[0] // empty' \ + | grep -oE '[0-9]+\.[0-9]+' | head -1 || true)" + fixed="$(echo "$v" | jq -r ' + [.affected[]?.ranges[]?.events[]? | select(.fixed != null) | .fixed] | .[0] // ""')" + sev="$(norm_sev "$rawsev" "$cvss")" + maybe_tiebreak "$v" # inert in this phase + add_finding "$nm" "vuln-$(echo "${pkg}-${ver}-${aid}" | tr -c 'A-Za-z0-9-' '-')" \ + "$pkg $ver is vulnerable ($aid)" "$sev" \ + "$pkg" "$ver" "$aid" "$summ" "$fixed" + j=$((j+1)) + done + done <<< "$deps_tsv" +done + +# ============================================================================== +# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to compliance-drift) +# ============================================================================== +if [ "${#FINDINGS[@]}" -gt 0 ]; then + FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)" +else + FINDINGS_JSON="[]" +fi +if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then + SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)" +else + SKIPPED_JSON="[]" +fi + +N_VULN="$(echo "$FINDINGS_JSON" | jq 'length')" +N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high" or .severity=="critical")] | length')" +N_REPOS_VULN="$(echo "$FINDINGS_JSON" | jq '[.[].repo] | unique | length')" + +jq -n \ + --arg checker "dependency-cve" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \ + --arg advmode "$ADV_MODE" --argjson scanned "${#REPO_NAMES[@]}" \ + --argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \ + '{checker:$checker, generated:$ts, org:$org, advisory_mode:$advmode, + repos_scanned:$scanned, vuln_count:($findings|length), + repos_with_vulns:([$findings[].repo]|unique|length), + findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON" + +{ + echo "dependency-cve report — $UTC_STAMP" + echo "org=$GH_ORG repos_scanned=${#REPO_NAMES[@]} advisory_mode=$ADV_MODE" + echo "vulnerable deps: $N_VULN ($N_HIGH high/critical) across $N_REPOS_VULN repo(s)" + echo + echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n fix: upgrade \(.proof.package) -> \(.proof.fixed_version) (\(.proof.summary))"' + if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then + echo; echo "skipped (missing data — NOT counted as a vuln):" + echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"' + fi +} > "$REPORT_TXT" +chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true + +log "report: $REPORT_JSON ($N_VULN vuln finding(s), $N_REPOS_VULN repo(s))" + +# ============================================================================== +# CANARY ASSERTION (anti-complacency floor, design §6.4) +# ============================================================================== +if [ "$CANARY" -eq 1 ]; then + EXPECT_FILE="$HERE/fixtures/dependency-cve/EXPECTED_VULN_COUNT" + [ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE" + EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")" + log "canary assertion: expected vuln=$EXPECTED, got=$N_VULN" + if [ "$N_VULN" -ne "$EXPECTED" ]; then + echo "[dependency-cve] CANARY FAIL: planted-vuln count mismatch (expected $EXPECTED, got $N_VULN)" >&2 + echo " -> a parser or the advisory match regressed, or the fixture changed. See $REPORT_TXT." >&2 + exit 3 + fi + log "canary PASS: all $EXPECTED planted vulnerable deps detected." +fi + +# ============================================================================== +# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms) +# ============================================================================== +if [ "$N_VULN" -eq 0 ]; then + log "no vulnerable dependencies — posting NOTHING to Slack (ALARM-only policy)." + exit 0 +fi + +ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r ' + group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')" +SLACK_TEXT=":lock: *Sea Haven dependency-cve — ALARM* ($UTC_STAMP) +$N_VULN vulnerable pinned dependency(ies) across $N_REPOS_VULN repo(s) ($N_HIGH high/critical): +$ALARM_BODY + +Source: OSV advisory DB ($ADV_MODE) · complements Dependabot +Report (mode 600): \`$REPORT_JSON\` (on R720)" +SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)" + +echo "$SLACK_TEXT" >&2 + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 2)." + exit 0 +fi +post_slack_alarm "$SLACK_TEXT" +exit 0 + +# ============================================================================== +# PROVISIONING (NOT DONE HERE — gated, Phase 6): +# - No systemd unit / timer is installed by this script. Wiring it into the live +# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated. +# - The coordinator (design §5, checker_coordinator.sh) runs this alongside other +# Tier-1 checkers under one shared budget + versioned rotation state. +# - The LIVE "Claude + GPT tiebreak" severity-judge (design §4) is the only LLM seam; +# it is an inert stub here (maybe_tiebreak) and stays off in canary/dry-run/offline. +# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations +# for the build session, tracked outside this script. +# ============================================================================== diff --git a/checkers/doc-drift.sh b/checkers/doc-drift.sh new file mode 100755 index 0000000..d984216 --- /dev/null +++ b/checkers/doc-drift.sh @@ -0,0 +1,482 @@ +#!/usr/bin/env bash +# doc-drift.sh — Plane-1 / Tier-1 checker for the R720 agent-team. +# +# Design refs: docs/r720-agent-team-design.md §4 (Tier 1 roster: doc-drift — +# "Flags repos whose architecture moved but Confluence/README did not") and §7 Phase 3 +# ("doc-drift + step-ca/Roles Anywhere + aws-posture"). This is the THIRD Plane-1 checker +# built on the Phase-0 shared substrate (lib/sweep_substrate.sh); it mirrors +# compliance-drift.sh / dependency-cve.sh conventions VERBATIM so the coordinator (§5) can +# drive all of them identically. doc-drift is UNGATED (only aws-posture in this phase is +# hard-gated behind the GPT-4.1 IAM cross-review; that checker is NOT built here). +# +# WHAT IT DOES (read-only): +# Scans the SAME shallow clean clones nightly_sweep.sh already produced in $MIRROR_DIR — it +# does NOT re-clone (mirrors-first; an optional --refresh re-runs discovery+mirror via the +# shared substrate). In each mirror it flags repos whose ARCHITECTURE MOVED but the README +# DID NOT — i.e. documentation drift. The checklist is DETERMINISTIC and GROUNDED in the +# global CLAUDE.md README obligation; it does NOT invent fuzzy judgments. See "CHECKLIST". +# +# This phase is the deterministic core ONLY. The design's "Gemini (large context)" judge +# layer (§4) is a LATER enhancement: a clearly-marked inert stub hook (maybe_judge) marks +# the future seam; it does NOTHING offline and NOTHING in this phase. +# +# REPORTING (matches secrev sweep conventions): +# - Writes a per-run JSON + text report under $REPORT_ROOT//, mode 600 (umask 077). +# - Slack ALARM-ONLY: a clean run (no confirmed drift) posts NOTHING (memory +# feedback_cloudwatch_alarms). Secret-shaped values are redacted from the Slack string. +# - Reuses the substrate's redact() + post_slack_alarm() verbatim. +# +# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping): +# redact, post_slack_alarm -> Slack delivery (reads SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG) +# discover_repos, mirror_repo-> ONLY on --refresh (reads GH_TOKEN, GH_ORG, MIRROR_DIR, REPORT_DIR) +# Default path enumerates EXISTING $MIRROR_DIR/*/.git dirs — zero clones, zero network. +# +# CANARY / DRY-RUN (offline, no network, no token): +# --canary runs the checklist against a planted-drift fixture (checkers/fixtures/doc-drift/) +# and asserts the known drift count. This is the anti-complacency floor (design §6.4) AND the +# routing dry-run (§7 Phase 3): with --dry-run, the Slack alarm is composed + printed but NOT +# POSTed. Fully offline-smoke-testable (the checks are filesystem + `git log`, no network). +# +# SCOPE / SAFETY: +# Read-only. All checks are filesystem + local `git log`; NO network, NO token, NO GitHub API +# (doc-drift has no API-only checks — it is purely tree+history). Fixtures ship git metadata as +# dotgit/ (renamed to .git/ at run time) so they commit into THIS repo without becoming +# submodules — the SAME trick compliance-drift / dependency-cve use. A repo with NO README is +# SKIPPED (compliance-drift owns readme-present); doc-drift never double-flags a missing README. +# +# This script does NOT touch agent_team/ or agent-team/, is NOT wired into systemd, and does NOT +# stand up step-ca / Roles Anywhere / aws-posture — that is Phase-3/6 provisioning (gated). See +# the "PROVISIONING (NOT DONE HERE)" note at the bottom. +# +# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[doc-drift] $*" >&2; } +die() { echo "[doc-drift] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate --------------------------------------------------------- +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSTRATE="$HERE/../lib/sweep_substrate.sh" +[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE" +# shellcheck source=../lib/sweep_substrate.sh +. "$SUBSTRATE" + +# --- Config + defaults (env, all optional) ------------------------------------ +GH_ORG="${GH_ORG:-Sea-Haven-Industries}" +MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}" +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/doc-drift}" +# Docs-only repos describe themselves differently (a handbook is its own doc); skip the +# architecture-omission scan for them. They still get the staleness check. +DOCS_ONLY_REPOS="${DOCS_ONLY_REPOS:-engineering-handbook}" +# Staleness thresholds: README must lag the newest code by BOTH at least this many days AND +# this many substantial code commits before we call it drift (two-factor = no false alarm on a +# single quick fix landed after a doc commit; memory feedback_cloudwatch_alarms). +DOC_DRIFT_STALE_DAYS="${DOC_DRIFT_STALE_DAYS:-60}" +DOC_DRIFT_STALE_COMMITS="${DOC_DRIFT_STALE_COMMITS:-3}" + +REFRESH=0 # --refresh: re-run discovery+mirror via substrate (network). Default: reuse mirrors. +DO_API=1 # --no-api: accepted for interface-parity with the other checkers; doc-drift makes + # NO API calls, so this flag is a documented no-op (kept so the coordinator + # can pass a uniform flag set to every Tier-1 checker). +DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run). +CANARY=0 # --canary: run against the planted-drift fixture + assert the known count. +TARGETS_OVERRIDE="" # --targets "p1 p2": scan explicit dirs instead of the mirror set. + +usage() { + cat >&2 </dev/null || die "jq is required" +command -v git >/dev/null || die "git is required" + +# --- Report dir (mode 600 reports; matches sweep conventions) ----------------- +umask 077 +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +REPORT_DIR="$REPORT_ROOT/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope +SWEEP_LOG="$REPORT_DIR/doc-drift.log" # name the substrate's post_slack_alarm() references +REPORT_JSON="$REPORT_DIR/doc-drift.json" +REPORT_TXT="$REPORT_DIR/doc-drift.txt" + +# doc-drift makes NO API calls, so DO_API is a documented no-op kept only for coordinator +# flag-parity; surface it in the run banner so the chosen value is auditable (and used). +log "=== doc-drift $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN refresh=$REFRESH api=${DO_API}[no-op] stale_days=$DOC_DRIFT_STALE_DAYS stale_commits=$DOC_DRIFT_STALE_COMMITS) ===" + +# ------------------------------------------------------------------------------ +# CHECKLIST (grounded — every item cites the README obligation; nothing invented): +# +# readme-omits-component README exists but omits a major existing component +# present in the tree (top-level service dir, SAM/CDK stack, +# Lambda handler dir, openapi/docs API spec) +# -> global CLAUDE.md: "README must accurately describe +# architecture, services, data flow, and configuration" +# readme-stale-vs-code README last-touched commit far older than the newest code +# commit (>= DOC_DRIFT_STALE_DAYS) AND >= DOC_DRIFT_STALE_COMMITS +# substantial code commits landed after the README was touched +# -> global CLAUDE.md: "update the README in the same commit" +# +# A repo with NO README is SKIPPED (compliance-drift owns readme-present; double-flagging would +# be a false alarm). Each emitted finding follows the spirit of finding.schema.json +# (id/title/severity/category/proof/status) so the coordinator can route it like an agentic +# finding. category="other" (doc drift is not one of the schema's security categories). +# status="confirmed" only for deterministic filesystem/git-history facts. The future Gemini +# judge (design §4) is an inert stub (maybe_judge) — never invoked offline / in this phase. +# ------------------------------------------------------------------------------ + +# Drift accumulator: one JSON object per finding, appended to a bash array. +declare -a FINDINGS=() +add_finding() { # repo id title severity check proof + local repo="$1" id="$2" title="$3" sev="$4" check="$5" proof="$6" + FINDINGS+=( "$(jq -n \ + --arg repo "$repo" --arg id "$id" --arg title "$title" --arg sev "$sev" \ + --arg check "$check" --arg proof "$proof" \ + '{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other", + check:$check, status:"confirmed", proof:{outcome:$proof}}')" ) +} +declare -a SKIPPED_CHECKS=() # (repo:check) checks skipped on missing data — reported, never alarmed +note_skip() { SKIPPED_CHECKS+=( "$1" ); } + +in_csv() { # needle csv -> 0 if present + local n="$1" csv="$2"; case ",$csv," in *",$n,"*) return 0 ;; *) return 1 ;; esac +} + +# Inert future seam (design §4 "Gemini (large context)" judge): in LIVE mode an ambiguous +# omission ("is this component material enough to require a README mention?") could be escalated +# to a large-context judge. This phase keeps the deterministic core ONLY — the stub does nothing +# and is never reached offline / in canary / dry-run. +maybe_judge() { # candidate_json (no-op stub; Phase-3 intentionally inert) + return 0 +} + +# --- Does a README mention a component name? (case-insensitive, word-ish, deterministic) ---- +# Matches the bare name OR the name with a trailing slash (how a dir is usually cited). Strips +# a leading "the " never matters; we test the literal token. Pure grep, no fuzzy matching. +readme_mentions() { # readme_file name + local rf="$1" name="$2" + # Escape regex metacharacters in the component name (defensive; dir names are usually plain). + local esc; esc="$(printf '%s' "$name" | sed -E 's/[][(){}.*+?^$|\\/]/\\&/g')" + grep -qiE "(^|[^A-Za-z0-9_-])${esc}([^A-Za-z0-9_-]|/|$)" "$rf" 2>/dev/null +} + +# --- Enumerate the major components present in a repo tree (deterministic) ------ +# Emits "TYPElabelmention_token" lines. mention_token is what the README must contain. +# service-dir a top-level directory whose name ends in -service or -api, or named api/web/worker +# sam-cdk-stack a SAM/CDK stack root (template.yaml | app.py at a stack root | cdk.json) +# lambda-dir a Lambda handler dir (a dir named handlers/ or containing handler.* / app.py under handlers/) +# api-spec an openapi/ or docs/ directory or an openapi.* / swagger.* spec file +enumerate_components() { # repo_dir -> TSV lines + local dir="$1" d nm + + # 1) top-level service-ish directories (the unit a README is expected to name) + for d in "$dir"/*/; do + [ -d "$d" ] || continue + nm="$(basename "$d")" + case "$nm" in + .git|.github|node_modules|dist|build|vendor|__pycache__|.venv) continue ;; + esac + case "$nm" in + *-service|*-api|api|web|worker|backend|frontend) + printf 'service-dir\t%s\t%s\n' "$nm" "$nm" ;; + esac + done + + # 2) SAM / CDK stack roots + if [ -f "$dir/template.yaml" ] || [ -f "$dir/template.yml" ]; then + printf 'sam-cdk-stack\t%s\t%s\n' "template.yaml (SAM stack)" "template.yaml" + fi + if [ -f "$dir/cdk.json" ]; then + printf 'sam-cdk-stack\t%s\t%s\n' "cdk.json (CDK app)" "cdk.json" + fi + + # 3) Lambda handler dirs: a top-level/handlers-rooted dir literally named "handlers" + while IFS= read -r d; do + [ -n "$d" ] || continue + printf 'lambda-dir\t%s\t%s\n' "handlers/ (Lambda handlers)" "handlers" + break # one mention requirement for the handlers tree is enough + done < <(find "$dir" -maxdepth 2 -type d -name handlers -not -path '*/.git/*' 2>/dev/null) + + # 4) API spec: an openapi/ or docs/ dir, or an openapi.*/swagger.* file + if [ -d "$dir/openapi" ]; then + printf 'api-spec\t%s\t%s\n' "openapi/ (API spec)" "openapi" + elif find "$dir" -maxdepth 2 \( -iname 'openapi.*' -o -iname 'swagger.*' \) -not -path '*/.git/*' -print -quit 2>/dev/null | grep -q .; then + printf 'api-spec\t%s\t%s\n' "openapi/swagger spec" "openapi" + fi +} + +# --- README last-touch epoch vs newest code commit (staleness, deterministic git log) ------- +# Returns the staleness facts on stdout as TSV "readme_epochnewest_code_epochcommits_after". +# commits_after = count of commits that touched code (non-doc) files AFTER the README's last touch. +# Code = anything that is NOT a README/markdown/LICENSE/.gitignore/docs file. Prints nothing if +# the repo has no git history or no README in history (caller treats that as "cannot assess"). +readme_staleness_facts() { # repo_dir + local dir="$1" + command -v git >/dev/null || return 0 + git -C "$dir" rev-parse --git-dir >/dev/null 2>&1 || return 0 + + # README last-touch (committer epoch of the most recent commit touching README.md). + local rd_epoch + rd_epoch="$(git -C "$dir" log -1 --format='%ct' -- README.md 2>/dev/null || true)" + [ -n "$rd_epoch" ] || return 0 # README not in history -> cannot assess staleness + + # Newest commit touching a CODE path (exclude docs/markdown/license/config-noise). + local code_epoch + code_epoch="$(git -C "$dir" log -1 --format='%ct' -- \ + ':(exclude)README.md' ':(exclude)*.md' ':(exclude)docs/**' \ + ':(exclude)LICENSE' ':(exclude).gitignore' ':(exclude).github/**' \ + 2>/dev/null || true)" + [ -n "$code_epoch" ] || return 0 # no code commits -> nothing to be stale against + + # Count CODE commits strictly AFTER the README's last touch. + local commits_after + commits_after="$(git -C "$dir" rev-list --count "--since=@${rd_epoch}" HEAD -- \ + ':(exclude)README.md' ':(exclude)*.md' ':(exclude)docs/**' \ + ':(exclude)LICENSE' ':(exclude).gitignore' ':(exclude).github/**' \ + 2>/dev/null || echo 0)" + printf '%s\t%s\t%s\n' "$rd_epoch" "$code_epoch" "${commits_after:-0}" +} + +# ============================================================================== +# PER-REPO CHECK (offline; filesystem + local git log only) +# ============================================================================== +check_repo() { # repo_name repo_dir + local repo="$1" dir="$2" + local docs_only=0; in_csv "$repo" "$DOCS_ONLY_REPOS" && docs_only=1 + + # No README -> doc-drift cannot assess drift; compliance-drift owns readme-present. SKIP. + if [ ! -f "$dir/README.md" ]; then + note_skip "$repo:doc-drift(no-readme — compliance-drift owns readme-present)" + return + fi + local readme="$dir/README.md" + + # --- readme-omits-component (skip for docs-only repos: they document differently) --- + if [ "$docs_only" -eq 0 ]; then + local type label token + while IFS=$'\t' read -r type label token; do + [ -n "$token" ] || continue + if ! readme_mentions "$readme" "$token"; then + add_finding "$repo" "readme-omits-$(printf '%s' "$type-$token" | tr -c 'A-Za-z0-9-' '-')" \ + "README omits existing component: $label" "medium" "readme-omits-component" \ + "global CLAUDE.md: README must accurately describe architecture/services (present in tree, absent from README: $label)" + fi + done < <(enumerate_components "$dir") + else + note_skip "$repo:readme-omits-component(docs-only)" + fi + + # --- readme-stale-vs-code (two-factor: age in days AND code-commits-after) --- + local facts; facts="$(readme_staleness_facts "$dir")" + if [ -z "$facts" ]; then + note_skip "$repo:readme-stale-vs-code(no-history-or-no-readme-in-history)" + else + local rd_epoch code_epoch commits_after age_days + IFS=$'\t' read -r rd_epoch code_epoch commits_after <<< "$facts" + age_days=$(( (code_epoch - rd_epoch) / 86400 )) + [ "$age_days" -lt 0 ] && age_days=0 + if [ "$age_days" -ge "$DOC_DRIFT_STALE_DAYS" ] && [ "$commits_after" -ge "$DOC_DRIFT_STALE_COMMITS" ]; then + add_finding "$repo" "readme-stale" \ + "README is stale: ${age_days}d behind newest code, ${commits_after} code commit(s) since last README touch" \ + "medium" "readme-stale-vs-code" \ + "global CLAUDE.md: update the README in the same commit as functionality changes (thresholds: >=${DOC_DRIFT_STALE_DAYS}d AND >=${DOC_DRIFT_STALE_COMMITS} code commits)" + fi + fi + + maybe_judge "" # inert in this phase (future Gemini large-context seam) +} + +# ============================================================================== +# TARGET RESOLUTION +# ============================================================================== +declare -a REPO_NAMES=(); declare -A REPO_DIR=() + +if [ "$CANARY" -eq 1 ]; then + FIXTURE_ROOT="$HERE/fixtures/doc-drift" + [ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT" + # Pin the exception lists + thresholds the fixtures were authored against, so the canary is + # self-contained and deterministic regardless of the operator's env. + DOCS_ONLY_REPOS="" + DOC_DRIFT_STALE_DAYS=60 + DOC_DRIFT_STALE_COMMITS=3 + # Fixtures ship their git metadata as `dotgit/` (not `.git/`) so they are committable into THIS + # repo without becoming nested submodules. Materialize them into a temp work area — copy each + # fixture and rename dotgit -> .git — so the README/git-log checks run against a real git + # checkout. The temp area is mode 700 and removed on exit (same trick as compliance-drift.sh). + FIXTURE_WORK="$(mktemp -d "${TMPDIR:-/tmp}/doc-drift-canary.XXXXXX")" + trap 'rm -rf "$FIXTURE_WORK"' EXIT + log "canary: materializing planted-drift fixtures from $FIXTURE_ROOT into $FIXTURE_WORK" + for d in "$FIXTURE_ROOT"/*/; do + [ -d "$d/dotgit" ] || continue # only fixture repos (skip README.md, EXPECTED_* etc.) + nm="$(basename "$d")" + cp -R "$d" "$FIXTURE_WORK/$nm" + mv "$FIXTURE_WORK/$nm/dotgit" "$FIXTURE_WORK/$nm/.git" + REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$FIXTURE_WORK/$nm" + done +elif [ -n "$TARGETS_OVERRIDE" ]; then + # shellcheck disable=SC2206 # intentional word-split of the space-separated --targets list + arr=( $TARGETS_OVERRIDE ) + for p in "${arr[@]}"; do p="${p/#\~/$HOME}"; nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p"; done + log "explicit targets: ${REPO_NAMES[*]}" +else + if [ "$REFRESH" -eq 1 ]; then + [ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN" + command -v curl >/dev/null || die "--refresh needs curl" + mkdir -p "$MIRROR_DIR" + log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)" + DISCOVERED="$REPORT_DIR/discovered.tsv" + if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then + while IFS=$'\t' read -r name url branch; do + [ -n "$name" ] || continue + mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)" + done < "$DISCOVERED" + else + log "discovery failed — falling back to existing mirrors (coverage may be stale)" + fi + fi + # Default + post-refresh: enumerate EXISTING mirrors. No clone here — reuse the sweep's clones. + [ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)" + for d in "$MIRROR_DIR"/*/; do + [ -d "$d/.git" ] || continue + nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}" + done + log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)" +fi + +[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to scan" + +# ============================================================================== +# RUN CHECKS +# ============================================================================== +for nm in "${REPO_NAMES[@]}"; do + check_repo "$nm" "${REPO_DIR[$nm]}" +done + +# ============================================================================== +# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to compliance-drift) +# ============================================================================== +if [ "${#FINDINGS[@]}" -gt 0 ]; then + FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)" +else + FINDINGS_JSON="[]" +fi +if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then + SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)" +else + SKIPPED_JSON="[]" +fi + +N_DRIFT="$(echo "$FINDINGS_JSON" | jq 'length')" +N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high")] | length')" +N_REPOS_DRIFTED="$(echo "$FINDINGS_JSON" | jq '[.[].repo] | unique | length')" + +jq -n \ + --arg checker "doc-drift" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \ + --argjson scanned "${#REPO_NAMES[@]}" \ + --argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \ + '{checker:$checker, generated:$ts, org:$org, + repos_scanned:$scanned, drift_count:($findings|length), + repos_with_drift:([$findings[].repo]|unique|length), + findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON" + +{ + echo "doc-drift report — $UTC_STAMP" + echo "org=$GH_ORG repos_scanned=${#REPO_NAMES[@]} stale_thresholds=${DOC_DRIFT_STALE_DAYS}d/${DOC_DRIFT_STALE_COMMITS}commits" + echo "drift findings: $N_DRIFT ($N_HIGH high) across $N_REPOS_DRIFTED repo(s)" + echo + echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n rule: \(.proof.outcome)"' + if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then + echo; echo "skipped checks (missing data / not doc-drift's job — NOT counted as drift):" + echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"' + fi +} > "$REPORT_TXT" +chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true + +log "report: $REPORT_JSON ($N_DRIFT drift finding(s), $N_REPOS_DRIFTED repo(s))" + +# ============================================================================== +# CANARY ASSERTION (anti-complacency floor, design §6.4) +# ============================================================================== +if [ "$CANARY" -eq 1 ]; then + EXPECT_FILE="$HERE/fixtures/doc-drift/EXPECTED_DRIFT_COUNT" + [ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE" + EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")" + log "canary assertion: expected drift=$EXPECTED, got=$N_DRIFT" + if [ "$N_DRIFT" -ne "$EXPECTED" ]; then + echo "[doc-drift] CANARY FAIL: planted-drift count mismatch (expected $EXPECTED, got $N_DRIFT)" >&2 + echo " -> the checklist regressed (a check stopped firing) or the fixture changed. See $REPORT_TXT." >&2 + exit 3 + fi + log "canary PASS: all $EXPECTED planted drifts detected." +fi + +# ============================================================================== +# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms) +# ============================================================================== +if [ "$N_DRIFT" -eq 0 ]; then + log "no confirmed drift — posting NOTHING to Slack (ALARM-only policy)." + exit 0 +fi + +ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r ' + group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')" +SLACK_TEXT=":memo: *Sea Haven doc-drift — ALARM* ($UTC_STAMP) +$N_DRIFT documentation-drift finding(s) across $N_REPOS_DRIFTED repo(s) ($N_HIGH high): +$ALARM_BODY + +Checks: README-omits-component · README-stale-vs-code (architecture moved, docs did not) +Report (mode 600): \`$REPORT_JSON\` (on R720)" +SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)" + +echo "$SLACK_TEXT" >&2 + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 3)." + exit 0 +fi +post_slack_alarm "$SLACK_TEXT" +exit 0 + +# ============================================================================== +# PROVISIONING (NOT DONE HERE — gated, Phase 3 / Phase 6): +# - No systemd unit / timer is installed by this script. Wiring it into the live +# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated. +# - This script is NOT registered in checker_coordinator.sh; the coordinator registry is +# integrated centrally (separate change), so doc-drift is not yet driven by the coordinator. +# - step-ca / IAM Roles Anywhere / the read-only AWS role / aws-posture are NOT stood up or +# built here. The IAM artifacts authored alongside this checker (security-review/iam/) are +# FILES for the mandatory GPT-4.1 cross-review; aws-posture itself is hard-gated behind that +# review and is built only after it is recorded (design §7, B3). +# - The LIVE "Gemini (large context)" doc-drift judge (design §4) is the only LLM seam; it is +# an inert stub here (maybe_judge) and stays off in canary / dry-run / offline. +# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations for the +# build session, tracked outside this script. +# ============================================================================== diff --git a/checkers/fixtures/aws-posture/EXPECTED_FINDING_COUNT b/checkers/fixtures/aws-posture/EXPECTED_FINDING_COUNT new file mode 100644 index 0000000..7f8f011 --- /dev/null +++ b/checkers/fixtures/aws-posture/EXPECTED_FINDING_COUNT @@ -0,0 +1 @@ +7 diff --git a/checkers/fixtures/aws-posture/README.md b/checkers/fixtures/aws-posture/README.md new file mode 100644 index 0000000..f681a87 --- /dev/null +++ b/checkers/fixtures/aws-posture/README.md @@ -0,0 +1,34 @@ +# aws-posture canary fixtures + +Mocked AWS API responses for `checkers/aws-posture.sh --canary` (offline — **no `aws` calls, no +network, no credentials**). The canary feeds these files to the SAME detectors the live path runs +against real `aws` CLI output, and asserts the total finding count equals `EXPECTED_FINDING_COUNT` +(anti-complacency floor, design §6.4). If a detector regresses (stops firing), the count drops and +the canary FAILS (exit 3). + +These are plain JSON files (not git fixtures — aws-posture scans an AWS account, not a repo tree), +so there is no `dotgit/` / `.fixture` rename trick here; the offline-vs-live seam is the +`--canary`/`--no-api`/no-credentials guard inside the checker (mirrors compliance-drift's +API-skip pattern). Each file is shaped like the real `aws ... --output json` response it stands in +for; a few `_Fixture*` helper keys carry the per-resource metric the live path derives from +CloudWatch (so the canary stays deterministic and offline). + +| Fixture file | Stands in for | Planted finding | Count | +|---|---|---|---| +| `cost-anomalies.json` | `aws ce get-anomalies` | 1 anomaly TotalImpact ≥ threshold (the other is below threshold → must NOT fire) | 1 | +| `describe-instances.json` | `aws ec2 describe-instances` | 1 `stopped` instance still paying for its EBS root (the `running` one must NOT fire) | 1 | +| `describe-volumes.json` | `aws ec2 describe-volumes` | 1 `available` (unattached) volume (the `in-use` one must NOT fire) | 1 | +| `describe-addresses.json` | `aws ec2 describe-addresses` | 1 EIP with no association (the associated one must NOT fire) | 1 | +| `describe-nat-gateways.json` | `aws ec2 describe-nat-gateways` | 1 `available` NAT with ~0 bytes out / 14d (the busy one must NOT fire) | 1 | +| `describe-load-balancers.json` | `aws elbv2 describe-load-balancers` | 1 ALB with 0 healthy targets (the one with 3 must NOT fire) | 1 | +| `describe-db-instances.json` | `aws rds describe-db-instances` | 1 `available` RDS with 0 connections / 14d (the busy one must NOT fire) | 1 | + +Total = **7** (`EXPECTED_FINDING_COUNT`). + +aws-posture **complements** GuardDuty / Security Hub / Config (design §4 / Tier-2) — it is an +idle/anomalous-**spend** + idle-resource posture watch, not a threat detector, and never alarms on +missing data (a skipped/credential-less live call is noted, never counted — memory +`feedback_cloudwatch_alarms`). + +When you add/remove a detector or fixture, update both the fixture and `EXPECTED_FINDING_COUNT` +in the same commit (the canary edit is itself caught on the next run — design §6.4). diff --git a/checkers/fixtures/aws-posture/cost-anomalies.json b/checkers/fixtures/aws-posture/cost-anomalies.json new file mode 100644 index 0000000..4287230 --- /dev/null +++ b/checkers/fixtures/aws-posture/cost-anomalies.json @@ -0,0 +1,32 @@ +{ + "Anomalies": [ + { + "AnomalyId": "anomaly-0001", + "AnomalyStartDate": "2026-06-15", + "AnomalyEndDate": "2026-06-17", + "DimensionValue": "Amazon Elastic Compute Cloud - Compute", + "RootCauses": [ + { "Service": "Amazon Elastic Compute Cloud - Compute", "Region": "us-east-1" } + ], + "Impact": { + "MaxImpact": 142.55, + "TotalImpact": 268.40, + "TotalActualSpend": 410.10, + "TotalExpectedSpend": 141.70 + }, + "Feedback": "NO_FEEDBACK" + }, + { + "AnomalyId": "anomaly-0002-below-threshold", + "AnomalyStartDate": "2026-06-16", + "DimensionValue": "AWS Lambda", + "Impact": { + "MaxImpact": 1.10, + "TotalImpact": 2.05, + "TotalActualSpend": 9.00, + "TotalExpectedSpend": 6.95 + }, + "Feedback": "NO_FEEDBACK" + } + ] +} diff --git a/checkers/fixtures/aws-posture/describe-addresses.json b/checkers/fixtures/aws-posture/describe-addresses.json new file mode 100644 index 0000000..81df327 --- /dev/null +++ b/checkers/fixtures/aws-posture/describe-addresses.json @@ -0,0 +1,17 @@ +{ + "Addresses": [ + { + "PublicIp": "52.10.20.30", + "AllocationId": "eipalloc-idle-7001", + "Domain": "vpc", + "Tags": [ { "Key": "Name", "Value": "leftover-nat-eip" } ] + }, + { + "PublicIp": "52.40.50.60", + "AllocationId": "eipalloc-inuse-7002", + "Domain": "vpc", + "InstanceId": "i-0ff99ee88dd77cc66", + "AssociationId": "eipassoc-active-0001" + } + ] +} diff --git a/checkers/fixtures/aws-posture/describe-db-instances.json b/checkers/fixtures/aws-posture/describe-db-instances.json new file mode 100644 index 0000000..a7e4bcf --- /dev/null +++ b/checkers/fixtures/aws-posture/describe-db-instances.json @@ -0,0 +1,20 @@ +{ + "DBInstances": [ + { + "DBInstanceIdentifier": "idle-reporting-db", + "DBInstanceClass": "db.r5.large", + "Engine": "postgres", + "DBInstanceStatus": "available", + "MultiAZ": false, + "_FixtureMaxConnectionsLast14d": 0 + }, + { + "DBInstanceIdentifier": "prod-app-db", + "DBInstanceClass": "db.t3.medium", + "Engine": "postgres", + "DBInstanceStatus": "available", + "MultiAZ": true, + "_FixtureMaxConnectionsLast14d": 47 + } + ] +} diff --git a/checkers/fixtures/aws-posture/describe-instances.json b/checkers/fixtures/aws-posture/describe-instances.json new file mode 100644 index 0000000..dfea016 --- /dev/null +++ b/checkers/fixtures/aws-posture/describe-instances.json @@ -0,0 +1,27 @@ +{ + "Reservations": [ + { + "Instances": [ + { + "InstanceId": "i-0aa11bb22cc33dd44", + "InstanceType": "m5.large", + "State": { "Name": "stopped" }, + "StateTransitionReason": "User initiated (2026-02-01 09:14:00 GMT)", + "BlockDeviceMappings": [ + { "DeviceName": "/dev/xvda", "Ebs": { "VolumeId": "vol-stopped-root-001", "Status": "attached" } } + ], + "Tags": [ { "Key": "Name", "Value": "old-batch-runner" } ] + }, + { + "InstanceId": "i-0ff99ee88dd77cc66", + "InstanceType": "t3.micro", + "State": { "Name": "running" }, + "BlockDeviceMappings": [ + { "DeviceName": "/dev/xvda", "Ebs": { "VolumeId": "vol-running-root-002", "Status": "attached" } } + ], + "Tags": [ { "Key": "Name", "Value": "active-web" } ] + } + ] + } + ] +} diff --git a/checkers/fixtures/aws-posture/describe-load-balancers.json b/checkers/fixtures/aws-posture/describe-load-balancers.json new file mode 100644 index 0000000..ba4b1f9 --- /dev/null +++ b/checkers/fixtures/aws-posture/describe-load-balancers.json @@ -0,0 +1,18 @@ +{ + "LoadBalancers": [ + { + "LoadBalancerArn": "arn:aws:elasticloadbalancing:us-east-1:328440206208:loadbalancer/app/idle-alb/abc", + "LoadBalancerName": "idle-alb", + "Type": "application", + "State": { "Code": "active" }, + "_FixtureHealthyTargetCount": 0 + }, + { + "LoadBalancerArn": "arn:aws:elasticloadbalancing:us-east-1:328440206208:loadbalancer/app/active-alb/def", + "LoadBalancerName": "active-alb", + "Type": "application", + "State": { "Code": "active" }, + "_FixtureHealthyTargetCount": 3 + } + ] +} diff --git a/checkers/fixtures/aws-posture/describe-nat-gateways.json b/checkers/fixtures/aws-posture/describe-nat-gateways.json new file mode 100644 index 0000000..328add3 --- /dev/null +++ b/checkers/fixtures/aws-posture/describe-nat-gateways.json @@ -0,0 +1,19 @@ +{ + "NatGateways": [ + { + "NatGatewayId": "nat-idle-6001", + "State": "available", + "SubnetId": "subnet-abc123", + "VpcId": "vpc-def456", + "Tags": [ { "Key": "Name", "Value": "unused-private-subnet-nat" } ], + "_FixtureBytesOutLast14d": 0 + }, + { + "NatGatewayId": "nat-active-6002", + "State": "available", + "SubnetId": "subnet-xyz789", + "VpcId": "vpc-def456", + "_FixtureBytesOutLast14d": 9842113 + } + ] +} diff --git a/checkers/fixtures/aws-posture/describe-volumes.json b/checkers/fixtures/aws-posture/describe-volumes.json new file mode 100644 index 0000000..e340072 --- /dev/null +++ b/checkers/fixtures/aws-posture/describe-volumes.json @@ -0,0 +1,20 @@ +{ + "Volumes": [ + { + "VolumeId": "vol-unattached-9001", + "Size": 500, + "VolumeType": "gp3", + "State": "available", + "CreateTime": "2025-11-02T18:00:00.000Z", + "Attachments": [], + "Tags": [ { "Key": "Name", "Value": "orphaned-data-disk" } ] + }, + { + "VolumeId": "vol-running-root-002", + "Size": 8, + "VolumeType": "gp3", + "State": "in-use", + "Attachments": [ { "InstanceId": "i-0ff99ee88dd77cc66", "State": "attached" } ] + } + ] +} diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotenv.fixture b/checkers/fixtures/compliance-drift/BadName_repo/dotenv.fixture new file mode 100644 index 0000000..4d56164 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotenv.fixture @@ -0,0 +1 @@ +API_KEY=AKIAIOSFODNN7EXAMPLE diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..b1b7161 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +init diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/HEAD b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/config b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/config new file mode 100644 index 0000000..8bb2ccd --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/config @@ -0,0 +1,10 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/description b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/post-update.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/update.sample b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/index b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/index new file mode 100644 index 0000000..64af49e Binary files /dev/null and b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/index differ diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/info/exclude b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/logs/HEAD b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/logs/HEAD new file mode 100644 index 0000000..c0ca3c9 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 bc8f350556a9ba83a52c0896c69005ec5c872c71 t 1781805299 -0400 commit (initial): init diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/logs/refs/heads/main b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..c0ca3c9 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 bc8f350556a9ba83a52c0896c69005ec5c872c71 t 1781805299 -0400 commit (initial): init diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/4d/56164171e7ac47e50f3c6b06091d1f1e4e36a1 b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/4d/56164171e7ac47e50f3c6b06091d1f1e4e36a1 new file mode 100644 index 0000000..27a5379 Binary files /dev/null and b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/4d/56164171e7ac47e50f3c6b06091d1f1e4e36a1 differ diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/72/9065c84f82263e16b2ef5241f3d4f0fc0aca45 b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/72/9065c84f82263e16b2ef5241f3d4f0fc0aca45 new file mode 100644 index 0000000..184767e --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/72/9065c84f82263e16b2ef5241f3d4f0fc0aca45 @@ -0,0 +1 @@ +x+)JMU07b040031QK+c s,|)M6B|Abrvbz^Vq~ÓBnKPm WK \ No newline at end of file diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/bc/8f350556a9ba83a52c0896c69005ec5c872c71 b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/bc/8f350556a9ba83a52c0896c69005ec5c872c71 new file mode 100644 index 0000000..04249ea Binary files /dev/null and b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/bc/8f350556a9ba83a52c0896c69005ec5c872c71 differ diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/e4/f0ea548f86dabe65d0507443f306ac0fdeeaa5 b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/e4/f0ea548f86dabe65d0507443f306ac0fdeeaa5 new file mode 100644 index 0000000..6f0e32e Binary files /dev/null and b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/objects/e4/f0ea548f86dabe65d0507443f306ac0fdeeaa5 differ diff --git a/checkers/fixtures/compliance-drift/BadName_repo/dotgit/refs/heads/main b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/refs/heads/main new file mode 100644 index 0000000..4d44489 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +bc8f350556a9ba83a52c0896c69005ec5c872c71 diff --git a/checkers/fixtures/compliance-drift/BadName_repo/package.json b/checkers/fixtures/compliance-drift/BadName_repo/package.json new file mode 100644 index 0000000..e4f0ea5 --- /dev/null +++ b/checkers/fixtures/compliance-drift/BadName_repo/package.json @@ -0,0 +1 @@ +{"name":"x"} diff --git a/checkers/fixtures/compliance-drift/EXPECTED_DRIFT_COUNT b/checkers/fixtures/compliance-drift/EXPECTED_DRIFT_COUNT new file mode 100644 index 0000000..1e8b314 --- /dev/null +++ b/checkers/fixtures/compliance-drift/EXPECTED_DRIFT_COUNT @@ -0,0 +1 @@ +6 diff --git a/checkers/fixtures/compliance-drift/README.md b/checkers/fixtures/compliance-drift/README.md new file mode 100644 index 0000000..5c45d1d --- /dev/null +++ b/checkers/fixtures/compliance-drift/README.md @@ -0,0 +1,28 @@ +# compliance-drift canary fixtures + +Planted-drift corpus for `checkers/compliance-drift.sh --canary` (offline, no network/token). +The checker asserts the total drift count equals `EXPECTED_DRIFT_COUNT` (anti-complacency floor, +design §6.4). If a check regresses (stops firing), the count drops and the canary FAILS (exit 3). + +Fixtures (each a real git checkout so the tracked-`.env` / `ls-files` checks work): + +| Fixture | Planted drift | Count | +|---|---|---| +| `clean-repo` | none — kebab name, README, ci.yaml, dependabot.yml, `.env` is **gitignored** (must NOT fire) | 0 | +| `BadName_repo` | non-kebab name; no README; no ci.yaml; has `package.json` but no `dependabot.yml`; tracked `.env` with values | 5 | +| `docs-repo` | docs-only (CI skipped via DOCS_ONLY_REPOS), kebab name, no README | 1 | + +Total = **6** (`EXPECTED_DRIFT_COUNT`). The canary pins `DOCS_ONLY_REPOS=docs-repo` and +`COMPLIANCE_EXEMPT=""` internally so it is deterministic regardless of the operator's env. + +**Secret-fixture naming:** `BadName_repo`'s planted tracked-secret env file is committed as +`dotenv.fixture`, NOT `.env`. The repo's root `.gitignore` lists `.env`, so a literal `.env` +fixture would silently never be committed — on a fresh clone the `secrets-committed` drift would +vanish and the count would drop to 5 (this regression was caught by this very canary). The +`--canary` materialization renames `dotenv.fixture` → `.env` in its temp work area; the +`dotgit/` index already TRACKS `.env`, so `git ls-files` still reports it. This mirrors the +`.fixture`-suffix convention the `dependency-cve` fixtures use for their manifests. Keep any new +committed secret fixture under a non-gitignored name and rename it in the canary. + +When you add/remove a check or fixture, update both the fixture and `EXPECTED_DRIFT_COUNT` +in the same commit (the canary edit is itself caught on the next run — design §6.4). diff --git a/checkers/fixtures/compliance-drift/clean-repo/.github/dependabot.yml b/checkers/fixtures/compliance-drift/clean-repo/.github/dependabot.yml new file mode 100644 index 0000000..22817d2 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/.github/dependabot.yml @@ -0,0 +1 @@ +version: 2 diff --git a/checkers/fixtures/compliance-drift/clean-repo/.github/workflows/ci.yaml b/checkers/fixtures/compliance-drift/clean-repo/.github/workflows/ci.yaml new file mode 100644 index 0000000..5843981 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/.github/workflows/ci.yaml @@ -0,0 +1 @@ +name: CI diff --git a/checkers/fixtures/compliance-drift/clean-repo/.gitignore b/checkers/fixtures/compliance-drift/clean-repo/.gitignore new file mode 100644 index 0000000..713d500 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/.gitignore @@ -0,0 +1,2 @@ +node_modules/ +.env diff --git a/checkers/fixtures/compliance-drift/clean-repo/README.md b/checkers/fixtures/compliance-drift/clean-repo/README.md new file mode 100644 index 0000000..2f2f27c --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/README.md @@ -0,0 +1 @@ +# clean-repo diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/compliance-drift/clean-repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..b1b7161 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +init diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/HEAD b/checkers/fixtures/compliance-drift/clean-repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/config b/checkers/fixtures/compliance-drift/clean-repo/dotgit/config new file mode 100644 index 0000000..8bb2ccd --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/config @@ -0,0 +1,10 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/description b/checkers/fixtures/compliance-drift/clean-repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/update.sample b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/index b/checkers/fixtures/compliance-drift/clean-repo/dotgit/index new file mode 100644 index 0000000..93f0dab Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/index differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/info/exclude b/checkers/fixtures/compliance-drift/clean-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/logs/HEAD b/checkers/fixtures/compliance-drift/clean-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..b6e12ed --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 72baacfb9265a352ce186809392c0c849c6220e4 t 1781805299 -0400 commit (initial): init diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/compliance-drift/clean-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..b6e12ed --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 72baacfb9265a352ce186809392c0c849c6220e4 t 1781805299 -0400 commit (initial): init diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/22/817d2a9c7fc1f62d5670ca1e44948446543973 b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/22/817d2a9c7fc1f62d5670ca1e44948446543973 new file mode 100644 index 0000000..10bb808 Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/22/817d2a9c7fc1f62d5670ca1e44948446543973 differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/2f/2f27c44f1ffd173eb9771fab6b77d0ee075bb5 b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/2f/2f27c44f1ffd173eb9771fab6b77d0ee075bb5 new file mode 100644 index 0000000..003dbbf Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/2f/2f27c44f1ffd173eb9771fab6b77d0ee075bb5 differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/58/439813fe88d3d045a3093999f382522a7a7327 b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/58/439813fe88d3d045a3093999f382522a7a7327 new file mode 100644 index 0000000..1c07e1b Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/58/439813fe88d3d045a3093999f382522a7a7327 differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/5d/51e08f85f54ae3c0b94e94e669fb64868538cb b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/5d/51e08f85f54ae3c0b94e94e669fb64868538cb new file mode 100644 index 0000000..426c77b Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/5d/51e08f85f54ae3c0b94e94e669fb64868538cb differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/71/3d5006dabfe2792c6ee1dbb4fdbdff2c44e304 b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/71/3d5006dabfe2792c6ee1dbb4fdbdff2c44e304 new file mode 100644 index 0000000..c0f1465 Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/71/3d5006dabfe2792c6ee1dbb4fdbdff2c44e304 differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/72/baacfb9265a352ce186809392c0c849c6220e4 b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/72/baacfb9265a352ce186809392c0c849c6220e4 new file mode 100644 index 0000000..c7569df Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/72/baacfb9265a352ce186809392c0c849c6220e4 differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/ac/9e4bf172ad292a8d19f7acfb7676c3a0d3bfbd b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/ac/9e4bf172ad292a8d19f7acfb7676c3a0d3bfbd new file mode 100644 index 0000000..be64983 Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/ac/9e4bf172ad292a8d19f7acfb7676c3a0d3bfbd differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/b9/0fb0f8fa06bca16e51fa38bbd44fc6735d2b6d b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/b9/0fb0f8fa06bca16e51fa38bbd44fc6735d2b6d new file mode 100644 index 0000000..c3c50ad Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/b9/0fb0f8fa06bca16e51fa38bbd44fc6735d2b6d differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/c4/30364d61f3b0be2614d2c76f873edd7547a54a b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/c4/30364d61f3b0be2614d2c76f873edd7547a54a new file mode 100644 index 0000000..b0a7a24 Binary files /dev/null and b/checkers/fixtures/compliance-drift/clean-repo/dotgit/objects/c4/30364d61f3b0be2614d2c76f873edd7547a54a differ diff --git a/checkers/fixtures/compliance-drift/clean-repo/dotgit/refs/heads/main b/checkers/fixtures/compliance-drift/clean-repo/dotgit/refs/heads/main new file mode 100644 index 0000000..8084ef9 --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +72baacfb9265a352ce186809392c0c849c6220e4 diff --git a/checkers/fixtures/compliance-drift/clean-repo/package.json b/checkers/fixtures/compliance-drift/clean-repo/package.json new file mode 100644 index 0000000..b90fb0f --- /dev/null +++ b/checkers/fixtures/compliance-drift/clean-repo/package.json @@ -0,0 +1 @@ +{"name":"clean-repo"} diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/compliance-drift/docs-repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..b1b7161 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +init diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/HEAD b/checkers/fixtures/compliance-drift/docs-repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/config b/checkers/fixtures/compliance-drift/docs-repo/dotgit/config new file mode 100644 index 0000000..8bb2ccd --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/config @@ -0,0 +1,10 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/description b/checkers/fixtures/compliance-drift/docs-repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/update.sample b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/index b/checkers/fixtures/compliance-drift/docs-repo/dotgit/index new file mode 100644 index 0000000..9d83913 Binary files /dev/null and b/checkers/fixtures/compliance-drift/docs-repo/dotgit/index differ diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/info/exclude b/checkers/fixtures/compliance-drift/docs-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/logs/HEAD b/checkers/fixtures/compliance-drift/docs-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..a1496c0 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 79906bf4bbcd829d2a1f611b11b058dd90827217 t 1781805299 -0400 commit (initial): init diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/compliance-drift/docs-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..a1496c0 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 79906bf4bbcd829d2a1f611b11b058dd90827217 t 1781805299 -0400 commit (initial): init diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/48/cdce85287243a96a9e7d47855104acbcb79837 b/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/48/cdce85287243a96a9e7d47855104acbcb79837 new file mode 100644 index 0000000..5d54a9e Binary files /dev/null and b/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/48/cdce85287243a96a9e7d47855104acbcb79837 differ diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/60/03c9aac8de93dc4777594f2b0d46ee8345ccea b/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/60/03c9aac8de93dc4777594f2b0d46ee8345ccea new file mode 100644 index 0000000..c500c91 Binary files /dev/null and b/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/60/03c9aac8de93dc4777594f2b0d46ee8345ccea differ diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/79/906bf4bbcd829d2a1f611b11b058dd90827217 b/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/79/906bf4bbcd829d2a1f611b11b058dd90827217 new file mode 100644 index 0000000..19c91fb Binary files /dev/null and b/checkers/fixtures/compliance-drift/docs-repo/dotgit/objects/79/906bf4bbcd829d2a1f611b11b058dd90827217 differ diff --git a/checkers/fixtures/compliance-drift/docs-repo/dotgit/refs/heads/main b/checkers/fixtures/compliance-drift/docs-repo/dotgit/refs/heads/main new file mode 100644 index 0000000..ab3a75a --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +79906bf4bbcd829d2a1f611b11b058dd90827217 diff --git a/checkers/fixtures/compliance-drift/docs-repo/index.html b/checkers/fixtures/compliance-drift/docs-repo/index.html new file mode 100644 index 0000000..48cdce8 --- /dev/null +++ b/checkers/fixtures/compliance-drift/docs-repo/index.html @@ -0,0 +1 @@ +placeholder diff --git a/checkers/fixtures/confluence-doc/EXPECTED_GAP_COUNT b/checkers/fixtures/confluence-doc/EXPECTED_GAP_COUNT new file mode 100644 index 0000000..00750ed --- /dev/null +++ b/checkers/fixtures/confluence-doc/EXPECTED_GAP_COUNT @@ -0,0 +1 @@ +3 diff --git a/checkers/fixtures/confluence-doc/README.md b/checkers/fixtures/confluence-doc/README.md new file mode 100644 index 0000000..038fbe5 --- /dev/null +++ b/checkers/fixtures/confluence-doc/README.md @@ -0,0 +1,47 @@ +# confluence-doc canary fixtures + +Planted doc-gap corpus for `checkers/confluence-doc.sh --canary` (offline, no network/token). +The checker asserts the total doc-gap count equals `EXPECTED_GAP_COUNT` (anti-complacency +floor, design §6.4). If a gap check regresses (stops firing) or the fixture changes, the count +drifts and the canary FAILS (exit 3). + +`--canary` implies `--dry-run + --no-api`, so the LIVE Confluence API checks (page-existence + +staleness, which need the gated `confluence-bot` token, D6) are SKIPPED and noted — they are +never counted as a gap on missing data (memory `feedback_cloudwatch_alarms`). + +## Fixture inputs + +| File | Role | +|---|---| +| `repos.txt` | the repo set to diff against the page-ID map (one repo name per line) | +| `mock-page-map.json` | a MOCK IT page-ID map (same shape as `project_confluence_migration`) | +| `mock-aws-inventory.json` | a MOCK read-only AWS inventory (what the API/collector would return) | + +## The 3 planted gaps + +| Check | Subject | Why it's a gap | +|---|---|---| +| repo-documented | `orphan-tool-repo` | no page in the mock map (and not doc-exempt) | +| aws-documented | `afi-backup-monitor` (Lambda) | inventory resource with no page in the mock map | +| required-page | `IAM & Access Management` | a REQUIRED standing page omitted from the mock map | + +Non-gaps proving the checks are precise (must NOT inflate the count): +- `payments-dashboard`, `seahaven-slack-bot` repos → matched to their pages. +- `engineering-handbook` repo → `DOC_EXEMPT_REPOS` → skipped, not a gap. +- `payments-dashboard` Lambda → matched to the "Payments Dashboard" page. +- `Incident Response Runbooks`, `Backup & Disaster Recovery` required pages → present in the map. +- The LIVE API staleness/existence check → SKIPPED (no creds in canary), noted, not a gap. + +Total = **3** (`EXPECTED_GAP_COUNT`). + +When you add/remove a check, a fixture input, or a planted gap, update the fixture(s) and +`EXPECTED_GAP_COUNT` in the same commit (the canary edit is itself caught on the next run — +design §6.4). + +## Not exercised offline (PROVISIONING — gated) + +The LIVE Confluence reads (and the on-demand WRITE path via +`~/.claude/scripts/confluence_mermaid.py`, including the page-1540098 live dry-run that must list +all 16 weweave Mermaid macros) require the `confluence-bot` service account + token. That account +creation, its 90-day rotation, and the Mermaid live dry-run are provisioning steps documented in +the checker's PROVISIONING footer — they are NOT performed by the canary. diff --git a/checkers/fixtures/confluence-doc/mock-aws-inventory.json b/checkers/fixtures/confluence-doc/mock-aws-inventory.json new file mode 100644 index 0000000..017b13f --- /dev/null +++ b/checkers/fixtures/confluence-doc/mock-aws-inventory.json @@ -0,0 +1,7 @@ +{ + "_comment": "MOCK read-only AWS inventory for confluence-doc.sh --canary. Stands in for what a read-only AWS inventory collector would emit (stacks/Lambdas). Each .resources[] entry is matched (by name token) against the page-ID map. 'payments-dashboard' matches the 'Payments Dashboard' page (no gap); 'afi-backup-monitor' has no page (1 planted gap).", + "resources": [ + { "type": "Lambda", "name": "payments-dashboard", "stack": "payments-dashboard" }, + { "type": "Lambda", "name": "afi-backup-monitor", "stack": "afi-backup-monitor" } + ] +} diff --git a/checkers/fixtures/confluence-doc/mock-page-map.json b/checkers/fixtures/confluence-doc/mock-page-map.json new file mode 100644 index 0000000..e3b84aa --- /dev/null +++ b/checkers/fixtures/confluence-doc/mock-page-map.json @@ -0,0 +1,9 @@ +{ + "_comment": "MOCK IT page-ID map for confluence-doc.sh --canary. Shape matches the real project_confluence_migration export: {\"\": }. Deliberately OMITS 'IAM & Access Management' (a REQUIRED page) and any page for 'orphan-tool-repo' / the 'afi-backup-monitor' Lambda, so the canary plants exactly 3 gaps. No live IDs are hardcoded into the checker — they live here.", + "AWS Cloud Infrastructure": 917505, + "AWS Architecture Map": 1540098, + "Payments Dashboard": 524602, + "Seahaven Slack Bot": 819202, + "Incident Response Runbooks": 1179652, + "Backup & Disaster Recovery": 1867778 +} diff --git a/checkers/fixtures/confluence-doc/repos.txt b/checkers/fixtures/confluence-doc/repos.txt new file mode 100644 index 0000000..009cb45 --- /dev/null +++ b/checkers/fixtures/confluence-doc/repos.txt @@ -0,0 +1,4 @@ +payments-dashboard +seahaven-slack-bot +engineering-handbook +orphan-tool-repo diff --git a/checkers/fixtures/dependency-cve/EXPECTED_VULN_COUNT b/checkers/fixtures/dependency-cve/EXPECTED_VULN_COUNT new file mode 100644 index 0000000..0cfbf08 --- /dev/null +++ b/checkers/fixtures/dependency-cve/EXPECTED_VULN_COUNT @@ -0,0 +1 @@ +2 diff --git a/checkers/fixtures/dependency-cve/README.md b/checkers/fixtures/dependency-cve/README.md new file mode 100644 index 0000000..54d8822 --- /dev/null +++ b/checkers/fixtures/dependency-cve/README.md @@ -0,0 +1,42 @@ +# dependency-cve canary fixtures + +Planted-vulnerable-dependency corpus for `checkers/dependency-cve.sh --canary` (offline, +no network/token). The checker asserts the total vulnerable-dependency count equals +`EXPECTED_VULN_COUNT` (anti-complacency floor, design §6.4). If extraction or matching +regresses (a parser stops firing, or the advisory match breaks), the count drops and the +canary FAILS (exit 3). + +## Offline advisory source + +OSV needs the network, so the canary CANNOT call `api.osv.dev`. Instead, `--canary` +(and the `--advisories-file PATH` override) makes the checker consult the local +`osv-advisories.json` fixture INSTEAD of the network — keyed by `ECOSYSTEM|package|version`. +This keeps the canary fully offline and deterministic. The fixture mirrors real advisory +ids/summaries/fixed-versions so a finding looks like a live one, but nothing is fetched. + +## Fixture repos (each a real git checkout; `dotgit/` is renamed to `.git/` at run time) + +The git metadata is shipped as `dotgit/` (not `.git/`) so these commit into the orchestrator +repo WITHOUT becoming nested submodules — the SAME trick `compliance-drift` fixtures use. The +checker copies each fixture to a temp area and renames `dotgit` → `.git` before scanning. + +| Fixture | Ecosystem | Pinned deps | Vulnerable match | Count | +|---|---|---|---|---| +| `vuln-py-repo` | PyPI (`requirements.txt`) | `flask==2.0.1`, `jinja2==2.11.2`, `requests==2.31.0` | `jinja2==2.11.2` → `GHSA-g3rq-g295-4j3m` | 1 | +| `vuln-js-repo` | npm (`package-lock.json`) | `lodash 4.17.15`, `left-pad 1.3.0` | `lodash 4.17.15` → `GHSA-p6mc-m468-83gw` | 1 | +| `clean-repo` | PyPI (`requirements.txt`) | `requests==2.31.0`, `urllib3==2.2.1` | none (no advisory entry) | 0 | + +Total = **2** (`EXPECTED_VULN_COUNT`). Two ecosystems are exercised (PyPI + npm) so a +regression in either parser is caught. + +When you add/remove a parser, a fixture, or an advisory entry, update the fixture(s), +`osv-advisories.json`, and `EXPECTED_VULN_COUNT` in the same commit (the canary edit is +itself caught on the next run — design §6.4). + +**Manifest naming:** the dependency manifests are stored with a `.fixture` suffix +(`requirements.txt.fixture`, `package-lock.json.fixture`) so GitHub's dependency graph / +the `dependency-review` CI action does NOT parse the deliberately-vulnerable canary pins as +real project dependencies (which would fail the PR gate). The checker's `--canary` +materialization strips the `.fixture` suffix in its temp work area before scanning, so the +per-ecosystem parsers still dispatch on the real names. Keep this suffix on any new +manifest fixture. diff --git a/checkers/fixtures/dependency-cve/clean-repo/README.md b/checkers/fixtures/dependency-cve/clean-repo/README.md new file mode 100644 index 0000000..c6df7ee --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/README.md @@ -0,0 +1,2 @@ +# clean-repo +Fixture: only non-vulnerable pinned deps; must produce NO findings. diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/dependency-cve/clean-repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..ee8c1ee --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +fixture diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/HEAD b/checkers/fixtures/dependency-cve/clean-repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/config b/checkers/fixtures/dependency-cve/clean-repo/dotgit/config new file mode 100644 index 0000000..f888611 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/config @@ -0,0 +1,12 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t +[commit] + gpgsign = false diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/description b/checkers/fixtures/dependency-cve/clean-repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/update.sample b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/index b/checkers/fixtures/dependency-cve/clean-repo/dotgit/index new file mode 100644 index 0000000..4e2e957 Binary files /dev/null and b/checkers/fixtures/dependency-cve/clean-repo/dotgit/index differ diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/info/exclude b/checkers/fixtures/dependency-cve/clean-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/logs/HEAD b/checkers/fixtures/dependency-cve/clean-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..de9a2da --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 a480a93df80db47ae333cdbdeb6b7fa3338945fe t 1781808419 -0400 commit (initial): fixture diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/dependency-cve/clean-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..de9a2da --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 a480a93df80db47ae333cdbdeb6b7fa3338945fe t 1781808419 -0400 commit (initial): fixture diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/37/62189d06d73852a1d6c7360d5057f06d4a6757 b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/37/62189d06d73852a1d6c7360d5057f06d4a6757 new file mode 100644 index 0000000..7267f91 --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/37/62189d06d73852a1d6c7360d5057f06d4a6757 @@ -0,0 +1 @@ +x+)JMU0d040031QrutuMa8vU=#U(z!(UTZXYWRWRQ+#ݭL>7͍$ \ No newline at end of file diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/a4/80a93df80db47ae333cdbdeb6b7fa3338945fe b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/a4/80a93df80db47ae333cdbdeb6b7fa3338945fe new file mode 100644 index 0000000..63e995e Binary files /dev/null and b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/a4/80a93df80db47ae333cdbdeb6b7fa3338945fe differ diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/c6/df7ee447bf5baa58bb4959a752fd2072e81114 b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/c6/df7ee447bf5baa58bb4959a752fd2072e81114 new file mode 100644 index 0000000..71b75cc --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/c6/df7ee447bf5baa58bb4959a752fd2072e81114 @@ -0,0 +1 @@ +xA@0Pk5 C?MjllD%~}RT)^bp|#&e,+@xd. \ No newline at end of file diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/e6/0a66cde22db502e6f0cdf92e91cbd9b138be8b b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/e6/0a66cde22db502e6f0cdf92e91cbd9b138be8b new file mode 100644 index 0000000..9da65ab Binary files /dev/null and b/checkers/fixtures/dependency-cve/clean-repo/dotgit/objects/e6/0a66cde22db502e6f0cdf92e91cbd9b138be8b differ diff --git a/checkers/fixtures/dependency-cve/clean-repo/dotgit/refs/heads/main b/checkers/fixtures/dependency-cve/clean-repo/dotgit/refs/heads/main new file mode 100644 index 0000000..fe6338d --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +a480a93df80db47ae333cdbdeb6b7fa3338945fe diff --git a/checkers/fixtures/dependency-cve/clean-repo/requirements.txt.fixture b/checkers/fixtures/dependency-cve/clean-repo/requirements.txt.fixture new file mode 100644 index 0000000..e60a66c --- /dev/null +++ b/checkers/fixtures/dependency-cve/clean-repo/requirements.txt.fixture @@ -0,0 +1,3 @@ +# all current / non-vulnerable pins +requests==2.31.0 +urllib3==2.2.1 diff --git a/checkers/fixtures/dependency-cve/osv-advisories.json b/checkers/fixtures/dependency-cve/osv-advisories.json new file mode 100644 index 0000000..6c5b402 --- /dev/null +++ b/checkers/fixtures/dependency-cve/osv-advisories.json @@ -0,0 +1,23 @@ +{ + "_comment": "Offline advisory fixture for dependency-cve.sh --canary (and --advisories-file). This stands in for the live OSV querybatch API so the canary is fully offline + deterministic. Each entry is keyed by 'ECOSYSTEM|package|version' (ECOSYSTEM matches OSV ecosystem names: PyPI, npm, NuGet) and carries the fields the checker emits in a finding's proof. These mirror REAL advisories (GHSA/CVE ids + summaries + fixed versions) so the fixture is realistic, but the checker NEVER reaches the network in canary mode — it reads only this file.", + "advisories": { + "PyPI|jinja2|2.11.2": [ + { + "id": "GHSA-g3rq-g295-4j3m", + "summary": "Jinja2 ReDoS in the urlize filter via the urlize regex", + "severity": "high", + "cvss": 7.5, + "fixed_version": "2.11.3" + } + ], + "npm|lodash|4.17.15": [ + { + "id": "GHSA-p6mc-m468-83gw", + "summary": "Prototype pollution in lodash (zipObjectDeep / set / setWith)", + "severity": "high", + "cvss": 7.4, + "fixed_version": "4.17.19" + } + ] + } +} diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/README.md b/checkers/fixtures/dependency-cve/vuln-js-repo/README.md new file mode 100644 index 0000000..04a61f1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/README.md @@ -0,0 +1,2 @@ +# vuln-js-repo +Fixture: pins lodash 4.17.15 (planted, known-vulnerable per the offline advisory fixture). diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..ee8c1ee --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +fixture diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/HEAD b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/config b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/config new file mode 100644 index 0000000..f888611 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/config @@ -0,0 +1,12 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t +[commit] + gpgsign = false diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/description b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/update.sample b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/index b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/index new file mode 100644 index 0000000..148d1d3 Binary files /dev/null and b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/index differ diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/info/exclude b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/logs/HEAD b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..8c5cec0 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 a3670ed0a5d8a573dd0a05aef0062738cfc99512 t 1781808419 -0400 commit (initial): fixture diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..8c5cec0 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 a3670ed0a5d8a573dd0a05aef0062738cfc99512 t 1781808419 -0400 commit (initial): fixture diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/04/a61f1e5cc08e1462578b765336dcceb5d4927c b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/04/a61f1e5cc08e1462578b765336dcceb5d4927c new file mode 100644 index 0000000..b8d2d94 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/04/a61f1e5cc08e1462578b765336dcceb5d4927c @@ -0,0 +1,3 @@ +x%[ +1 PgQłp)S II;^Xb]Xܣ;ӽbtCcqQ/q?ILR +!v? 'T \ No newline at end of file diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/32/f1266a3a1e4455383258de2c85369df3f4bc66 b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/32/f1266a3a1e4455383258de2c85369df3f4bc66 new file mode 100644 index 0000000..e7718e5 Binary files /dev/null and b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/32/f1266a3a1e4455383258de2c85369df3f4bc66 differ diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/a3/670ed0a5d8a573dd0a05aef0062738cfc99512 b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/a3/670ed0a5d8a573dd0a05aef0062738cfc99512 new file mode 100644 index 0000000..7b3bf35 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/a3/670ed0a5d8a573dd0a05aef0062738cfc99512 @@ -0,0 +1,2 @@ +x}A +@ F@!#@)^N.D"x|xu[%HT¬pqUE}jnn@NS+xc̜=!-Gt$ \ No newline at end of file diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/ea/2be04d982807e7e7f93012953c4f98c7e1f5e0 b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/ea/2be04d982807e7e7f93012953c4f98c7e1f5e0 new file mode 100644 index 0000000..191aa13 Binary files /dev/null and b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/objects/ea/2be04d982807e7e7f93012953c4f98c7e1f5e0 differ diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/refs/heads/main b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/refs/heads/main new file mode 100644 index 0000000..215221f --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +a3670ed0a5d8a573dd0a05aef0062738cfc99512 diff --git a/checkers/fixtures/dependency-cve/vuln-js-repo/package-lock.json.fixture b/checkers/fixtures/dependency-cve/vuln-js-repo/package-lock.json.fixture new file mode 100644 index 0000000..32f1266 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-js-repo/package-lock.json.fixture @@ -0,0 +1,23 @@ +{ + "name": "vuln-js-repo", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "vuln-js-repo", + "version": "1.0.0", + "dependencies": { "lodash": "4.17.15", "left-pad": "1.3.0" } + }, + "node_modules/lodash": { + "version": "4.17.15", + "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz", + "integrity": "sha512-fake" + }, + "node_modules/left-pad": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "integrity": "sha512-fake" + } + } +} diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/README.md b/checkers/fixtures/dependency-cve/vuln-py-repo/README.md new file mode 100644 index 0000000..12b523e --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/README.md @@ -0,0 +1,2 @@ +# vuln-py-repo +Fixture: pins jinja2==2.11.2 (planted, known-vulnerable per the offline advisory fixture). diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..ee8c1ee --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +fixture diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/HEAD b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/config b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/config new file mode 100644 index 0000000..f888611 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/config @@ -0,0 +1,12 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t +[commit] + gpgsign = false diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/description b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/update.sample b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/index b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/index new file mode 100644 index 0000000..845996a Binary files /dev/null and b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/index differ diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/info/exclude b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/logs/HEAD b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..4b8eddf --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 a318bef74d77c826d0137c7db34aa5a539dbcee3 t 1781808419 -0400 commit (initial): fixture diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..4b8eddf --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 a318bef74d77c826d0137c7db34aa5a539dbcee3 t 1781808419 -0400 commit (initial): fixture diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/12/b523ebed36453519cb27baa9194baa3c65437a b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/12/b523ebed36453519cb27baa9194baa3c65437a new file mode 100644 index 0000000..45647a0 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/12/b523ebed36453519cb27baa9194baa3c65437a @@ -0,0 +1,4 @@ +x%A +0P=7 +&,\zN0u Ӵۋx2pV36<ꧯFwhs9q>qRNi%-'Rf? BHVf;?< +'A \ No newline at end of file diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/2a/9f78a311018981582d02f1a725c594adc09629 b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/2a/9f78a311018981582d02f1a725c594adc09629 new file mode 100644 index 0000000..95957ab Binary files /dev/null and b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/2a/9f78a311018981582d02f1a725c594adc09629 differ diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/8b/f6aeab1646a6033fe0bd18d99045fe3f0238f0 b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/8b/f6aeab1646a6033fe0bd18d99045fe3f0238f0 new file mode 100644 index 0000000..2c720a5 Binary files /dev/null and b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/8b/f6aeab1646a6033fe0bd18d99045fe3f0238f0 differ diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/a3/18bef74d77c826d0137c7db34aa5a539dbcee3 b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/a3/18bef74d77c826d0137c7db34aa5a539dbcee3 new file mode 100644 index 0000000..ba56be1 Binary files /dev/null and b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/objects/a3/18bef74d77c826d0137c7db34aa5a539dbcee3 differ diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/refs/heads/main b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/refs/heads/main new file mode 100644 index 0000000..3e4c315 --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +a318bef74d77c826d0137c7db34aa5a539dbcee3 diff --git a/checkers/fixtures/dependency-cve/vuln-py-repo/requirements.txt.fixture b/checkers/fixtures/dependency-cve/vuln-py-repo/requirements.txt.fixture new file mode 100644 index 0000000..8bf6aea --- /dev/null +++ b/checkers/fixtures/dependency-cve/vuln-py-repo/requirements.txt.fixture @@ -0,0 +1,4 @@ +# pinned deps for the python service +flask==2.0.1 +jinja2==2.11.2 +requests==2.31.0 diff --git a/checkers/fixtures/doc-drift/EXPECTED_DRIFT_COUNT b/checkers/fixtures/doc-drift/EXPECTED_DRIFT_COUNT new file mode 100644 index 0000000..b8626c4 --- /dev/null +++ b/checkers/fixtures/doc-drift/EXPECTED_DRIFT_COUNT @@ -0,0 +1 @@ +4 diff --git a/checkers/fixtures/doc-drift/README.md b/checkers/fixtures/doc-drift/README.md new file mode 100644 index 0000000..248364c --- /dev/null +++ b/checkers/fixtures/doc-drift/README.md @@ -0,0 +1,43 @@ +# doc-drift canary fixtures + +Planted-drift corpus for `checkers/doc-drift.sh --canary` (offline, no network/token). +The checker asserts the total drift count equals `EXPECTED_DRIFT_COUNT` (anti-complacency +floor, design §6.4). If a check regresses (stops firing), the count drops and the canary +FAILS (exit 3). + +doc-drift flags repos whose **architecture moved but the README did not** (design §4, +doc-drift row). It is deliberately deterministic and grounded — no fuzzy LLM judgment. The +LLM judge layer (Gemini large-context) is a later enhancement and is inert offline (see the +`maybe_judge` stub in the checker). + +Each fixture is a real git checkout (its `.git` is shipped as `dotgit/` so it commits into +THIS repo without becoming a nested submodule; the checker renames it back to `.git/` at run +time, the same trick `compliance-drift.sh` / `dependency-cve.sh` use). Real commit history is +required because the staleness check reads `git log` dates. + +## Detected drift (the deterministic checklist) + +| Check | Rule cited | What fires | +|---|---|---| +| `readme-omits-component` | global CLAUDE.md: "README must accurately describe architecture, services, data flow" | A README exists but omits mention of a major existing component present in the tree: a top-level service dir, a SAM/CDK stack (`template.yaml` / `app.py` / `cdk.json`), a Lambda handler dir, or an `openapi`/`docs` API spec. | +| `readme-stale-vs-code` | global CLAUDE.md: "update the README in the same commit" as functionality changes | The README's last-touched commit is far older than the newest code commit (≥ `DOC_DRIFT_STALE_DAYS` days) AND ≥ `DOC_DRIFT_STALE_COMMITS` substantial code commits landed after the README was last touched. | + +A repo with **no README at all** is SKIPPED by doc-drift, not flagged — `readme-present` is +`compliance-drift.sh`'s job, and double-flagging would be a false alarm +(memory `feedback_cloudwatch_alarms`). + +## Fixtures + +| Fixture | Planted drift | Count | +|---|---|---| +| `clean-repo` | none — README names every component (`api/`, the SAM stack, `handlers/`, `openapi/`) and the README was committed alongside the code | 0 | +| `drift-omits-repo` | README mentions only `notifier-service`; omits `payments-service/`, the SAM `template.yaml` stack, and the `handlers/charge` Lambda dir | 3 | +| `drift-stale-repo` | README names its one component (no omission) but was last touched 2026-01-05 while 5 substantial code commits landed in 2026-06 — stale | 1 | +| `no-readme-repo` | no README — doc-drift SKIPS it (must NOT fire; compliance-drift owns this) | 0 | + +Total = **4** (`EXPECTED_DRIFT_COUNT`). The canary pins the staleness thresholds it was +authored against (`DOC_DRIFT_STALE_DAYS`, `DOC_DRIFT_STALE_COMMITS`) internally so it is +deterministic regardless of the operator's env. + +When you add/remove a check or fixture, update both the fixture and `EXPECTED_DRIFT_COUNT` +in the same commit (the canary edit is itself caught on the next run — design §6.4). diff --git a/checkers/fixtures/doc-drift/clean-repo/README.md b/checkers/fixtures/doc-drift/clean-repo/README.md new file mode 100644 index 0000000..bfde979 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/README.md @@ -0,0 +1,10 @@ +# clean-repo + +Well-documented service. Architecture: + +- The **api/** service exposes the public HTTP surface. +- A SAM stack (see template.yaml) provisions the IngestFn Lambda. +- Lambda handler code lives under handlers/ingest. +- The HTTP contract is published in the openapi/ spec. + +Data flow: api -> IngestFn -> downstream. diff --git a/checkers/fixtures/doc-drift/clean-repo/api/main.go b/checkers/fixtures/doc-drift/clean-repo/api/main.go new file mode 100644 index 0000000..06ab7d0 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/api/main.go @@ -0,0 +1 @@ +package main diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/doc-drift/clean-repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..ffc6555 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +init: code + matching README diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/HEAD b/checkers/fixtures/doc-drift/clean-repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/config b/checkers/fixtures/doc-drift/clean-repo/dotgit/config new file mode 100644 index 0000000..8bb2ccd --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/config @@ -0,0 +1,10 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/description b/checkers/fixtures/doc-drift/clean-repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/update.sample b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/index b/checkers/fixtures/doc-drift/clean-repo/dotgit/index new file mode 100644 index 0000000..e228a0e Binary files /dev/null and b/checkers/fixtures/doc-drift/clean-repo/dotgit/index differ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/info/exclude b/checkers/fixtures/doc-drift/clean-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/logs/HEAD b/checkers/fixtures/doc-drift/clean-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..d44addb --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 e5c55ac820d3272863a874fc1e202908241c2acf t 1780329600 -0400 commit (initial): init: code + matching README diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/doc-drift/clean-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..d44addb --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 e5c55ac820d3272863a874fc1e202908241c2acf t 1780329600 -0400 commit (initial): init: code + matching README diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/06/ab7d0f9a35a7d1070711496d6ca1cb892a258f b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/06/ab7d0f9a35a7d1070711496d6ca1cb892a258f new file mode 100644 index 0000000..8182c9a Binary files /dev/null and b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/06/ab7d0f9a35a7d1070711496d6ca1cb892a258f differ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/0b/5c163a57c647bd824a907f7e0b9aa6335a7d77 b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/0b/5c163a57c647bd824a907f7e0b9aa6335a7d77 new file mode 100644 index 0000000..657ddba Binary files /dev/null and b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/0b/5c163a57c647bd824a907f7e0b9aa6335a7d77 differ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/1f/86368c694ecb3d9d64788a988ca4a5365e4df8 b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/1f/86368c694ecb3d9d64788a988ca4a5365e4df8 new file mode 100644 index 0000000..502fcf6 Binary files /dev/null and b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/1f/86368c694ecb3d9d64788a988ca4a5365e4df8 differ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/51/3273c393a63fbff76e46c8a8ed159bd9e83a1c b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/51/3273c393a63fbff76e46c8a8ed159bd9e83a1c new file mode 100644 index 0000000..dee7780 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/51/3273c393a63fbff76e46c8a8ed159bd9e83a1c @@ -0,0 +1 @@ +xe1 0F+nH,%jCr&.=gߜ Nirj*M;;כּ:9yH!$bU7wBV^$-F-b- \ No newline at end of file diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/88/72f0d44bfbbfa113423377b41597af8faacb8d b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/88/72f0d44bfbbfa113423377b41597af8faacb8d new file mode 100644 index 0000000..79dfdce Binary files /dev/null and b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/88/72f0d44bfbbfa113423377b41597af8faacb8d differ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/98/ec3d6272badf42441f11b2c53b42961a33f5f7 b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/98/ec3d6272badf42441f11b2c53b42961a33f5f7 new file mode 100644 index 0000000..afd3888 Binary files /dev/null and b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/98/ec3d6272badf42441f11b2c53b42961a33f5f7 differ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/a2/549621f3ce0547771b96e53f14e2fcd74a3e50 b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/a2/549621f3ce0547771b96e53f14e2fcd74a3e50 new file mode 100644 index 0000000..d0257c5 Binary files /dev/null and b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/a2/549621f3ce0547771b96e53f14e2fcd74a3e50 differ diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/b9/270df7070cc6a5e7dbdec610a7ce4f54c47b20 b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/b9/270df7070cc6a5e7dbdec610a7ce4f54c47b20 new file mode 100644 index 0000000..302def6 --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/b9/270df7070cc6a5e7dbdec610a7ce4f54c47b20 @@ -0,0 +1 @@ +x+)JMU06e040031QMKg`[]?tEvvAܜ;T \ No newline at end of file diff --git a/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/bf/de979a9dc263aafe98de15bc024e98440984d7 b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/bf/de979a9dc263aafe98de15bc024e98440984d7 new file mode 100644 index 0000000..ed685cd --- /dev/null +++ b/checkers/fixtures/doc-drift/clean-repo/dotgit/objects/bf/de979a9dc263aafe98de15bc024e98440984d7 @@ -0,0 +1 @@ +x=AK0=W\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/update.sample b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/index b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/index new file mode 100644 index 0000000..3a1ce27 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/index differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/info/exclude b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/logs/HEAD b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..b2878de --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 ff9ded83431a6059a99140b744c351e43bb04eed t 1781107200 -0400 commit (initial): init diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..b2878de --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 ff9ded83431a6059a99140b744c351e43bb04eed t 1781107200 -0400 commit (initial): init diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/19/366a9a93232cf60a444d9a1d4114bc55084d0f b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/19/366a9a93232cf60a444d9a1d4114bc55084d0f new file mode 100644 index 0000000..3a527c2 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/19/366a9a93232cf60a444d9a1d4114bc55084d0f differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/1f/86368c694ecb3d9d64788a988ca4a5365e4df8 b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/1f/86368c694ecb3d9d64788a988ca4a5365e4df8 new file mode 100644 index 0000000..502fcf6 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/1f/86368c694ecb3d9d64788a988ca4a5365e4df8 differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/36/31a0436785d485c1f063f82fe6755d6cf9ee89 b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/36/31a0436785d485c1f063f82fe6755d6cf9ee89 new file mode 100644 index 0000000..f469015 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/36/31a0436785d485c1f063f82fe6755d6cf9ee89 @@ -0,0 +1,2 @@ +xeM +1 @a=E]EzdH6dƝp?.Wʻ'y[R|IP>`έ8˴-b<;j@r'3u?$h#vl~(n e6q \ No newline at end of file diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/96/f70ef65cc77ae047293b093219c7f996e1d6fa b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/96/f70ef65cc77ae047293b093219c7f996e1d6fa new file mode 100644 index 0000000..5b2b01f Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/96/f70ef65cc77ae047293b093219c7f996e1d6fa differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/98/ec3d6272badf42441f11b2c53b42961a33f5f7 b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/98/ec3d6272badf42441f11b2c53b42961a33f5f7 new file mode 100644 index 0000000..afd3888 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/98/ec3d6272badf42441f11b2c53b42961a33f5f7 differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/162ac5a718f5b673c538badf3506c17d988fc8 b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/162ac5a718f5b673c538badf3506c17d988fc8 new file mode 100644 index 0000000..fb0a314 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/162ac5a718f5b673c538badf3506c17d988fc8 differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/24dce1a9a1c4b5bc277b5beeedec49d5604a75 b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/24dce1a9a1c4b5bc277b5beeedec49d5604a75 new file mode 100644 index 0000000..1f0b023 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/24dce1a9a1c4b5bc277b5beeedec49d5604a75 differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/81d157c401c61bebf0c4ef31dc7d3777edbcba b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/81d157c401c61bebf0c4ef31dc7d3777edbcba new file mode 100644 index 0000000..c040cc4 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/9a/81d157c401c61bebf0c4ef31dc7d3777edbcba differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/a1/7e6a98edf78d6c4f540aca77932f81e2b4ccdc b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/a1/7e6a98edf78d6c4f540aca77932f81e2b4ccdc new file mode 100644 index 0000000..6acf046 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/a1/7e6a98edf78d6c4f540aca77932f81e2b4ccdc differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/cc/b1aee04842afc8dcb1e6b7f292de8ddef50f73 b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/cc/b1aee04842afc8dcb1e6b7f292de8ddef50f73 new file mode 100644 index 0000000..16e0835 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/cc/b1aee04842afc8dcb1e6b7f292de8ddef50f73 differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/ff/9ded83431a6059a99140b744c351e43bb04eed b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/ff/9ded83431a6059a99140b744c351e43bb04eed new file mode 100644 index 0000000..c4cb0bf Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/objects/ff/9ded83431a6059a99140b744c351e43bb04eed differ diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/refs/heads/main b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/refs/heads/main new file mode 100644 index 0000000..555f7a3 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +ff9ded83431a6059a99140b744c351e43bb04eed diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/handlers/charge/app.py b/checkers/fixtures/doc-drift/drift-omits-repo/handlers/charge/app.py new file mode 100644 index 0000000..b473d36 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/handlers/charge/app.py @@ -0,0 +1,2 @@ +def handler(event, ctx): + return {} diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/notifier-service/main.py b/checkers/fixtures/doc-drift/drift-omits-repo/notifier-service/main.py new file mode 100644 index 0000000..af2e9f2 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/notifier-service/main.py @@ -0,0 +1,2 @@ +class Notifier: + pass diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/payments-service/main.py b/checkers/fixtures/doc-drift/drift-omits-repo/payments-service/main.py new file mode 100644 index 0000000..2ffacb2 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/payments-service/main.py @@ -0,0 +1,2 @@ +class Payments: + pass diff --git a/checkers/fixtures/doc-drift/drift-omits-repo/template.yaml b/checkers/fixtures/doc-drift/drift-omits-repo/template.yaml new file mode 100644 index 0000000..3f49180 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-omits-repo/template.yaml @@ -0,0 +1,11 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Resources: + ChargeFn: + Type: AWS::Serverless::Function + Properties: + Handler: app.handler + Runtime: python3.12 + InlineCode: | + def handler(event, context): + return {"statusCode": 200} diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/README.md b/checkers/fixtures/doc-drift/drift-stale-repo/README.md new file mode 100644 index 0000000..c7d8d3f --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/README.md @@ -0,0 +1,5 @@ +# drift-stale-repo + +- The **worker-service** processes the queue. + +Architecture is documented and complete as of this writing. diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/COMMIT_EDITMSG b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/COMMIT_EDITMSG new file mode 100644 index 0000000..67a10ad --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/COMMIT_EDITMSG @@ -0,0 +1 @@ +feat: add feature_5 to worker-service diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/HEAD b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/HEAD new file mode 100644 index 0000000..b870d82 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/HEAD @@ -0,0 +1 @@ +ref: refs/heads/main diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/config b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/config new file mode 100644 index 0000000..8bb2ccd --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/config @@ -0,0 +1,10 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[user] + email = t@t + name = t diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/description b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/description new file mode 100644 index 0000000..498b267 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/description @@ -0,0 +1 @@ +Unnamed repository; edit this file 'description' to name the repository. diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/applypatch-msg.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/applypatch-msg.sample new file mode 100755 index 0000000..a5d7b84 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/applypatch-msg.sample @@ -0,0 +1,15 @@ +#!/bin/sh +# +# An example hook script to check the commit log message taken by +# applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. The hook is +# allowed to edit the commit message file. +# +# To enable this hook, rename this file to "applypatch-msg". + +. git-sh-setup +commitmsg="$(git rev-parse --git-path hooks/commit-msg)" +test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"} +: diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/commit-msg.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/commit-msg.sample new file mode 100755 index 0000000..b58d118 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/commit-msg.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to check the commit log message. +# Called by "git commit" with one argument, the name of the file +# that has the commit message. The hook should exit with non-zero +# status after issuing an appropriate message if it wants to stop the +# commit. The hook is allowed to edit the commit message file. +# +# To enable this hook, rename this file to "commit-msg". + +# Uncomment the below to add a Signed-off-by line to the message. +# Doing this in a hook is a bad idea in general, but the prepare-commit-msg +# hook is more suited to it. +# +# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/update.sample b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/index b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/index new file mode 100644 index 0000000..a4e5a46 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/index differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/info/exclude b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/logs/HEAD b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..64f7bd2 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/logs/HEAD @@ -0,0 +1,6 @@ +0000000000000000000000000000000000000000 7687db2a5781d77b42ced78a6bca02c37a8dbbf0 t 1767632400 -0500 commit (initial): init: worker-service + README +7687db2a5781d77b42ced78a6bca02c37a8dbbf0 af8b041ef281bb9d89401efcdf549a9a452f0ecf t 1781193600 -0400 commit: feat: add feature_1 to worker-service +af8b041ef281bb9d89401efcdf549a9a452f0ecf 93a7db735d0cfe42f0a57d956915f5e5a7f87378 t 1781280000 -0400 commit: feat: add feature_2 to worker-service +93a7db735d0cfe42f0a57d956915f5e5a7f87378 9c2018ca90ae8305bec517e0b8b91b5d8a755404 t 1781366400 -0400 commit: feat: add feature_3 to worker-service +9c2018ca90ae8305bec517e0b8b91b5d8a755404 6b7c13685ae818268d30ed3cf27c86da2820f186 t 1781452800 -0400 commit: feat: add feature_4 to worker-service +6b7c13685ae818268d30ed3cf27c86da2820f186 0cd9efe3a4957b0f5336c33f4e40e9e4d779c243 t 1781539200 -0400 commit: feat: add feature_5 to worker-service diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..64f7bd2 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/logs/refs/heads/main @@ -0,0 +1,6 @@ +0000000000000000000000000000000000000000 7687db2a5781d77b42ced78a6bca02c37a8dbbf0 t 1767632400 -0500 commit (initial): init: worker-service + README +7687db2a5781d77b42ced78a6bca02c37a8dbbf0 af8b041ef281bb9d89401efcdf549a9a452f0ecf t 1781193600 -0400 commit: feat: add feature_1 to worker-service +af8b041ef281bb9d89401efcdf549a9a452f0ecf 93a7db735d0cfe42f0a57d956915f5e5a7f87378 t 1781280000 -0400 commit: feat: add feature_2 to worker-service +93a7db735d0cfe42f0a57d956915f5e5a7f87378 9c2018ca90ae8305bec517e0b8b91b5d8a755404 t 1781366400 -0400 commit: feat: add feature_3 to worker-service +9c2018ca90ae8305bec517e0b8b91b5d8a755404 6b7c13685ae818268d30ed3cf27c86da2820f186 t 1781452800 -0400 commit: feat: add feature_4 to worker-service +6b7c13685ae818268d30ed3cf27c86da2820f186 0cd9efe3a4957b0f5336c33f4e40e9e4d779c243 t 1781539200 -0400 commit: feat: add feature_5 to worker-service diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/06/2858f6d6f54e5a930a45178929532313b7a231 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/06/2858f6d6f54e5a930a45178929532313b7a231 new file mode 100644 index 0000000..60a4d11 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/06/2858f6d6f54e5a930a45178929532313b7a231 differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/0c/30159d993e4b7fc86add22ed6ce984618552ef b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/0c/30159d993e4b7fc86add22ed6ce984618552ef new file mode 100644 index 0000000..26b4e48 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/0c/30159d993e4b7fc86add22ed6ce984618552ef differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/0c/d9efe3a4957b0f5336c33f4e40e9e4d779c243 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/0c/d9efe3a4957b0f5336c33f4e40e9e4d779c243 new file mode 100644 index 0000000..5b45f57 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/0c/d9efe3a4957b0f5336c33f4e40e9e4d779c243 @@ -0,0 +1,3 @@ +x}K +@])O+"D&=$FƎ^\#+Bz9̦z]:('p] + Q&UPKkK])5< \ No newline at end of file diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/17/001f0544766545e2b63b2d3e7454ffa28cba39 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/17/001f0544766545e2b63b2d3e7454ffa28cba39 new file mode 100644 index 0000000..5d7e377 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/17/001f0544766545e2b63b2d3e7454ffa28cba39 differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/34/a52c1c070ac4df6225a1ef1a73643bcf92ef44 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/34/a52c1c070ac4df6225a1ef1a73643bcf92ef44 new file mode 100644 index 0000000..acdcb43 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/34/a52c1c070ac4df6225a1ef1a73643bcf92ef44 differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/4c/217577a9492a8030c5980e5d6796768781ed6d b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/4c/217577a9492a8030c5980e5d6796768781ed6d new file mode 100644 index 0000000..1032d34 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/4c/217577a9492a8030c5980e5d6796768781ed6d differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/59/c4d8defd13e7623f2af0073db64ce8ec4a1612 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/59/c4d8defd13e7623f2af0073db64ce8ec4a1612 new file mode 100644 index 0000000..abf5582 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/59/c4d8defd13e7623f2af0073db64ce8ec4a1612 differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/5e/a038ee524a2a7200c3e4eb22febc5aeceb24a5 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/5e/a038ee524a2a7200c3e4eb22febc5aeceb24a5 new file mode 100644 index 0000000..751f738 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/5e/a038ee524a2a7200c3e4eb22febc5aeceb24a5 differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/60/3e3162216f19595bd6df1db44faf0f3c168f8f b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/60/3e3162216f19595bd6df1db44faf0f3c168f8f new file mode 100644 index 0000000..ea99a67 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/60/3e3162216f19595bd6df1db44faf0f3c168f8f differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/64/74695394af5333896c7b296879398ef18776ca b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/64/74695394af5333896c7b296879398ef18776ca new file mode 100644 index 0000000..240db10 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/64/74695394af5333896c7b296879398ef18776ca @@ -0,0 +1 @@ +x+)JMU044e040031QHKM,)-J7+dضBؕ7y3:#Eֶˎ8,eaURXZS ieosw.f \ No newline at end of file diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/6b/7c13685ae818268d30ed3cf27c86da2820f186 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/6b/7c13685ae818268d30ed3cf27c86da2820f186 new file mode 100644 index 0000000..a80de5e --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/6b/7c13685ae818268d30ed3cf27c86da2820f186 @@ -0,0 +1,2 @@ +x}A +1 @Q=E. Ħ"M$m3(#5E/z=;$ĕ3(3ӦL:,"K9餽2"spa7FiZPM&jc6RKE33!}i!fIao oA{o= |2.6s< \ No newline at end of file diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/76/87db2a5781d77b42ced78a6bca02c37a8dbbf0 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/76/87db2a5781d77b42ced78a6bca02c37a8dbbf0 new file mode 100644 index 0000000..3af3019 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/76/87db2a5781d77b42ced78a6bca02c37a8dbbf0 @@ -0,0 +1,2 @@ +x}A +0Fa90Lt A@p/y.F@IR졘DrH\zڭ62!^:.Ɔ?ĕؖ޵ѺګdК.t,f \ No newline at end of file diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/76/fcd8c1bd06f4648062d7865d83e0ec5279d8f8 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/76/fcd8c1bd06f4648062d7865d83e0ec5279d8f8 new file mode 100644 index 0000000..8b238cc Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/76/fcd8c1bd06f4648062d7865d83e0ec5279d8f8 differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/8d/2121584af4558168be908795ae74dfbda6169e b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/8d/2121584af4558168be908795ae74dfbda6169e new file mode 100644 index 0000000..0fa22fd Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/8d/2121584af4558168be908795ae74dfbda6169e differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/8f/a23b3da38f76c4418ca50353902b048836568f b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/8f/a23b3da38f76c4418ca50353902b048836568f new file mode 100644 index 0000000..466ddd2 Binary files /dev/null and b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/8f/a23b3da38f76c4418ca50353902b048836568f differ diff --git a/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/93/a7db735d0cfe42f0a57d956915f5e5a7f87378 b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/93/a7db735d0cfe42f0a57d956915f5e5a7f87378 new file mode 100644 index 0000000..fb04569 --- /dev/null +++ b/checkers/fixtures/doc-drift/drift-stale-repo/dotgit/objects/93/a7db735d0cfe42f0a57d956915f5e5a7f87378 @@ -0,0 +1,2 @@ +x}Q +1 D)rVDMhfcj!|1źXR䊥5lї\).*$/Signed-off-by: \1/p') +# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" + +# This example catches duplicate Signed-off-by lines. + +test "" = "$(grep '^Signed-off-by: ' "$1" | + sort | uniq -c | sed -e '/^[ ]*1[ ]/d')" || { + echo >&2 Duplicate Signed-off-by lines. + exit 1 +} diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/fsmonitor-watchman.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/fsmonitor-watchman.sample new file mode 100755 index 0000000..23e856f --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/fsmonitor-watchman.sample @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +use strict; +use warnings; +use IPC::Open2; + +# An example hook script to integrate Watchman +# (https://facebook.github.io/watchman/) with git to speed up detecting +# new and modified files. +# +# The hook is passed a version (currently 2) and last update token +# formatted as a string and outputs to stdout a new update token and +# all files that have been modified since the update token. Paths must +# be relative to the root of the working tree and separated by a single NUL. +# +# To enable this hook, rename this file to "query-watchman" and set +# 'git config core.fsmonitor .git/hooks/query-watchman' +# +my ($version, $last_update_token) = @ARGV; + +# Uncomment for debugging +# print STDERR "$0 $version $last_update_token\n"; + +# Check the hook interface version +if ($version ne 2) { + die "Unsupported query-fsmonitor hook version '$version'.\n" . + "Falling back to scanning...\n"; +} + +my $git_work_tree = get_working_dir(); + +my $retry = 1; + +my $json_pkg; +eval { + require JSON::XS; + $json_pkg = "JSON::XS"; + 1; +} or do { + require JSON::PP; + $json_pkg = "JSON::PP"; +}; + +launch_watchman(); + +sub launch_watchman { + my $o = watchman_query(); + if (is_work_tree_watched($o)) { + output_result($o->{clock}, @{$o->{files}}); + } +} + +sub output_result { + my ($clockid, @files) = @_; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # binmode $fh, ":utf8"; + # print $fh "$clockid\n@files\n"; + # close $fh; + + binmode STDOUT, ":utf8"; + print $clockid; + print "\0"; + local $, = "\0"; + print @files; +} + +sub watchman_clock { + my $response = qx/watchman clock "$git_work_tree"/; + die "Failed to get clock id on '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + + return $json_pkg->new->utf8->decode($response); +} + +sub watchman_query { + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') + or die "open2() failed: $!\n" . + "Falling back to scanning...\n"; + + # In the query expression below we're asking for names of files that + # changed since $last_update_token but not from the .git folder. + # + # To accomplish this, we're using the "since" generator to use the + # recency index to select candidate nodes and "fields" to limit the + # output to file names only. Then we're using the "expression" term to + # further constrain the results. + my $last_update_line = ""; + if (substr($last_update_token, 0, 1) eq "c") { + $last_update_token = "\"$last_update_token\""; + $last_update_line = qq[\n"since": $last_update_token,]; + } + my $query = <<" END"; + ["query", "$git_work_tree", {$last_update_line + "fields": ["name"], + "expression": ["not", ["dirname", ".git"]] + }] + END + + # Uncomment for debugging the watchman query + # open (my $fh, ">", ".git/watchman-query.json"); + # print $fh $query; + # close $fh; + + print CHLD_IN $query; + close CHLD_IN; + my $response = do {local $/; }; + + # Uncomment for debugging the watch response + # open ($fh, ">", ".git/watchman-response.json"); + # print $fh $response; + # close $fh; + + die "Watchman: command returned no output.\n" . + "Falling back to scanning...\n" if $response eq ""; + die "Watchman: command returned invalid output: $response\n" . + "Falling back to scanning...\n" unless $response =~ /^\{/; + + return $json_pkg->new->utf8->decode($response); +} + +sub is_work_tree_watched { + my ($output) = @_; + my $error = $output->{error}; + if ($retry > 0 and $error and $error =~ m/unable to resolve root .* directory (.*) is not watched/) { + $retry--; + my $response = qx/watchman watch "$git_work_tree"/; + die "Failed to make watchman watch '$git_work_tree'.\n" . + "Falling back to scanning...\n" if $? != 0; + $output = $json_pkg->new->utf8->decode($response); + $error = $output->{error}; + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + # Uncomment for debugging watchman output + # open (my $fh, ">", ".git/watchman-output.out"); + # close $fh; + + # Watchman will always return all files on the first query so + # return the fast "everything is dirty" flag to git and do the + # Watchman query just to get it over with now so we won't pay + # the cost in git to look up each individual file. + my $o = watchman_clock(); + $error = $output->{error}; + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + output_result($o->{clock}, ("/")); + $last_update_token = $o->{clock}; + + eval { launch_watchman() }; + return 0; + } + + die "Watchman: $error.\n" . + "Falling back to scanning...\n" if $error; + + return 1; +} + +sub get_working_dir { + my $working_dir; + if ($^O =~ 'msys' || $^O =~ 'cygwin') { + $working_dir = Win32::GetCwd(); + $working_dir =~ tr/\\/\//; + } else { + require Cwd; + $working_dir = Cwd::cwd(); + } + + return $working_dir; +} diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/post-update.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/post-update.sample new file mode 100755 index 0000000..ec17ec1 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/post-update.sample @@ -0,0 +1,8 @@ +#!/bin/sh +# +# An example hook script to prepare a packed repository for use over +# dumb transports. +# +# To enable this hook, rename this file to "post-update". + +exec git update-server-info diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-applypatch.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-applypatch.sample new file mode 100755 index 0000000..4142082 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-applypatch.sample @@ -0,0 +1,14 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed +# by applypatch from an e-mail message. +# +# The hook should exit with non-zero status after issuing an +# appropriate message if it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-applypatch". + +. git-sh-setup +precommit="$(git rev-parse --git-path hooks/pre-commit)" +test -x "$precommit" && exec "$precommit" ${1+"$@"} +: diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-commit.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-commit.sample new file mode 100755 index 0000000..29ed5ee --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-commit.sample @@ -0,0 +1,49 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git commit" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message if +# it wants to stop the commit. +# +# To enable this hook, rename this file to "pre-commit". + +if git rev-parse --verify HEAD >/dev/null 2>&1 +then + against=HEAD +else + # Initial commit: diff against an empty tree object + against=$(git hash-object -t tree /dev/null) +fi + +# If you want to allow non-ASCII filenames set this variable to true. +allownonascii=$(git config --type=bool hooks.allownonascii) + +# Redirect output to stderr. +exec 1>&2 + +# Cross platform projects tend to avoid non-ASCII filenames; prevent +# them from being added to the repository. We exploit the fact that the +# printable range starts at the space character and ends with tilde. +if [ "$allownonascii" != "true" ] && + # Note that the use of brackets around a tr range is ok here, (it's + # even required, for portability to Solaris 10's /usr/bin/tr), since + # the square bracket bytes happen to fall in the designated range. + test $(git diff-index --cached --name-only --diff-filter=A -z $against | + LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0 +then + cat <<\EOF +Error: Attempt to add a non-ASCII file name. + +This can cause problems if you want to work with people on other platforms. + +To be portable it is advisable to rename the file. + +If you know what you are doing you can disable this check using: + + git config hooks.allownonascii true +EOF + exit 1 +fi + +# If there are whitespace errors, print the offending file names and fail. +exec git diff-index --check --cached $against -- diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-merge-commit.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-merge-commit.sample new file mode 100755 index 0000000..399eab1 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-merge-commit.sample @@ -0,0 +1,13 @@ +#!/bin/sh +# +# An example hook script to verify what is about to be committed. +# Called by "git merge" with no arguments. The hook should +# exit with non-zero status after issuing an appropriate message to +# stderr if it wants to stop the merge commit. +# +# To enable this hook, rename this file to "pre-merge-commit". + +. git-sh-setup +test -x "$GIT_DIR/hooks/pre-commit" && + exec "$GIT_DIR/hooks/pre-commit" +: diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-push.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-push.sample new file mode 100755 index 0000000..4ce688d --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-push.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# An example hook script to verify what is about to be pushed. Called by "git +# push" after it has checked the remote status, but before anything has been +# pushed. If this script exits with a non-zero status nothing will be pushed. +# +# This hook is called with the following parameters: +# +# $1 -- Name of the remote to which the push is being done +# $2 -- URL to which the push is being done +# +# If pushing without using a named remote those arguments will be equal. +# +# Information about the commits which are being pushed is supplied as lines to +# the standard input in the form: +# +# +# +# This sample shows how to prevent push of commits where the log message starts +# with "WIP" (work in progress). + +remote="$1" +url="$2" + +zero=$(git hash-object --stdin &2 "Found WIP commit in $local_ref, not pushing" + exit 1 + fi + fi +done + +exit 0 diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-rebase.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-rebase.sample new file mode 100755 index 0000000..6cbef5c --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-rebase.sample @@ -0,0 +1,169 @@ +#!/bin/sh +# +# Copyright (c) 2006, 2008 Junio C Hamano +# +# The "pre-rebase" hook is run just before "git rebase" starts doing +# its job, and can prevent the command from running by exiting with +# non-zero status. +# +# The hook is called with the following parameters: +# +# $1 -- the upstream the series was forked from. +# $2 -- the branch being rebased (or empty when rebasing the current branch). +# +# This sample shows how to prevent topic branches that are already +# merged to 'next' branch from getting rebased, because allowing it +# would result in rebasing already published history. + +publish=next +basebranch="$1" +if test "$#" = 2 +then + topic="refs/heads/$2" +else + topic=`git symbolic-ref HEAD` || + exit 0 ;# we do not interrupt rebasing detached HEAD +fi + +case "$topic" in +refs/heads/??/*) + ;; +*) + exit 0 ;# we do not interrupt others. + ;; +esac + +# Now we are dealing with a topic branch being rebased +# on top of master. Is it OK to rebase it? + +# Does the topic really exist? +git show-ref -q "$topic" || { + echo >&2 "No such branch $topic" + exit 1 +} + +# Is topic fully merged to master? +not_in_master=`git rev-list --pretty=oneline ^master "$topic"` +if test -z "$not_in_master" +then + echo >&2 "$topic is fully merged to master; better remove it." + exit 1 ;# we could allow it, but there is no point. +fi + +# Is topic ever merged to next? If so you should not be rebasing it. +only_next_1=`git rev-list ^master "^$topic" ${publish} | sort` +only_next_2=`git rev-list ^master ${publish} | sort` +if test "$only_next_1" = "$only_next_2" +then + not_in_topic=`git rev-list "^$topic" master` + if test -z "$not_in_topic" + then + echo >&2 "$topic is already up to date with master" + exit 1 ;# we could allow it, but there is no point. + else + exit 0 + fi +else + not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"` + /usr/bin/perl -e ' + my $topic = $ARGV[0]; + my $msg = "* $topic has commits already merged to public branch:\n"; + my (%not_in_next) = map { + /^([0-9a-f]+) /; + ($1 => 1); + } split(/\n/, $ARGV[1]); + for my $elem (map { + /^([0-9a-f]+) (.*)$/; + [$1 => $2]; + } split(/\n/, $ARGV[2])) { + if (!exists $not_in_next{$elem->[0]}) { + if ($msg) { + print STDERR $msg; + undef $msg; + } + print STDERR " $elem->[1]\n"; + } + } + ' "$topic" "$not_in_next" "$not_in_master" + exit 1 +fi + +<<\DOC_END + +This sample hook safeguards topic branches that have been +published from being rewound. + +The workflow assumed here is: + + * Once a topic branch forks from "master", "master" is never + merged into it again (either directly or indirectly). + + * Once a topic branch is fully cooked and merged into "master", + it is deleted. If you need to build on top of it to correct + earlier mistakes, a new topic branch is created by forking at + the tip of the "master". This is not strictly necessary, but + it makes it easier to keep your history simple. + + * Whenever you need to test or publish your changes to topic + branches, merge them into "next" branch. + +The script, being an example, hardcodes the publish branch name +to be "next", but it is trivial to make it configurable via +$GIT_DIR/config mechanism. + +With this workflow, you would want to know: + +(1) ... if a topic branch has ever been merged to "next". Young + topic branches can have stupid mistakes you would rather + clean up before publishing, and things that have not been + merged into other branches can be easily rebased without + affecting other people. But once it is published, you would + not want to rewind it. + +(2) ... if a topic branch has been fully merged to "master". + Then you can delete it. More importantly, you should not + build on top of it -- other people may already want to + change things related to the topic as patches against your + "master", so if you need further changes, it is better to + fork the topic (perhaps with the same name) afresh from the + tip of "master". + +Let's look at this example: + + o---o---o---o---o---o---o---o---o---o "next" + / / / / + / a---a---b A / / + / / / / + / / c---c---c---c B / + / / / \ / + / / / b---b C \ / + / / / / \ / + ---o---o---o---o---o---o---o---o---o---o---o "master" + + +A, B and C are topic branches. + + * A has one fix since it was merged up to "next". + + * B has finished. It has been fully merged up to "master" and "next", + and is ready to be deleted. + + * C has not merged to "next" at all. + +We would want to allow C to be rebased, refuse A, and encourage +B to be deleted. + +To compute (1): + + git rev-list ^master ^topic next + git rev-list ^master next + + if these match, topic has not merged in next at all. + +To compute (2): + + git rev-list master..topic + + if this is empty, it is fully merged to "master". + +DOC_END diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-receive.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-receive.sample new file mode 100755 index 0000000..a1fd29e --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/pre-receive.sample @@ -0,0 +1,24 @@ +#!/bin/sh +# +# An example hook script to make use of push options. +# The example simply echoes all push options that start with 'echoback=' +# and rejects all pushes when the "reject" push option is used. +# +# To enable this hook, rename this file to "pre-receive". + +if test -n "$GIT_PUSH_OPTION_COUNT" +then + i=0 + while test "$i" -lt "$GIT_PUSH_OPTION_COUNT" + do + eval "value=\$GIT_PUSH_OPTION_$i" + case "$value" in + echoback=*) + echo "echo from the pre-receive-hook: ${value#*=}" >&2 + ;; + reject) + exit 1 + esac + i=$((i + 1)) + done +fi diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/prepare-commit-msg.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/prepare-commit-msg.sample new file mode 100755 index 0000000..10fa14c --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/prepare-commit-msg.sample @@ -0,0 +1,42 @@ +#!/bin/sh +# +# An example hook script to prepare the commit log message. +# Called by "git commit" with the name of the file that has the +# commit message, followed by the description of the commit +# message's source. The hook's purpose is to edit the commit +# message file. If the hook fails with a non-zero status, +# the commit is aborted. +# +# To enable this hook, rename this file to "prepare-commit-msg". + +# This hook includes three examples. The first one removes the +# "# Please enter the commit message..." help message. +# +# The second includes the output of "git diff --name-status -r" +# into the message, just before the "git status" output. It is +# commented because it doesn't cope with --amend or with squashed +# commits. +# +# The third example adds a Signed-off-by line to the message, that can +# still be edited. This is rarely a good idea. + +COMMIT_MSG_FILE=$1 +COMMIT_SOURCE=$2 +SHA1=$3 + +/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE" + +# case "$COMMIT_SOURCE,$SHA1" in +# ,|template,) +# /usr/bin/perl -i.bak -pe ' +# print "\n" . `git diff --cached --name-status -r` +# if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;; +# *) ;; +# esac + +# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p') +# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE" +# if test -z "$COMMIT_SOURCE" +# then +# /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE" +# fi diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/push-to-checkout.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/push-to-checkout.sample new file mode 100755 index 0000000..af5a0c0 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/push-to-checkout.sample @@ -0,0 +1,78 @@ +#!/bin/sh + +# An example hook script to update a checked-out tree on a git push. +# +# This hook is invoked by git-receive-pack(1) when it reacts to git +# push and updates reference(s) in its repository, and when the push +# tries to update the branch that is currently checked out and the +# receive.denyCurrentBranch configuration variable is set to +# updateInstead. +# +# By default, such a push is refused if the working tree and the index +# of the remote repository has any difference from the currently +# checked out commit; when both the working tree and the index match +# the current commit, they are updated to match the newly pushed tip +# of the branch. This hook is to be used to override the default +# behaviour; however the code below reimplements the default behaviour +# as a starting point for convenient modification. +# +# The hook receives the commit with which the tip of the current +# branch is going to be updated: +commit=$1 + +# It can exit with a non-zero status to refuse the push (when it does +# so, it must not modify the index or the working tree). +die () { + echo >&2 "$*" + exit 1 +} + +# Or it can make any necessary changes to the working tree and to the +# index to bring them to the desired state when the tip of the current +# branch is updated to the new commit, and exit with a zero status. +# +# For example, the hook can simply run git read-tree -u -m HEAD "$1" +# in order to emulate git fetch that is run in the reverse direction +# with git push, as the two-tree form of git read-tree -u -m is +# essentially the same as git switch or git checkout that switches +# branches while keeping the local changes in the working tree that do +# not interfere with the difference between the branches. + +# The below is a more-or-less exact translation to shell of the C code +# for the default behaviour for git's push-to-checkout hook defined in +# the push_to_deploy() function in builtin/receive-pack.c. +# +# Note that the hook will be executed from the repository directory, +# not from the working tree, so if you want to perform operations on +# the working tree, you will have to adapt your code accordingly, e.g. +# by adding "cd .." or using relative paths. + +if ! git update-index -q --ignore-submodules --refresh +then + die "Up-to-date check failed" +fi + +if ! git diff-files --quiet --ignore-submodules -- +then + die "Working directory has unstaged changes" +fi + +# This is a rough translation of: +# +# head_has_history() ? "HEAD" : EMPTY_TREE_SHA1_HEX +if git cat-file -e HEAD 2>/dev/null +then + head=HEAD +else + head=$(git hash-object -t tree --stdin &2 + exit 1 +} + +unset GIT_DIR GIT_WORK_TREE +cd "$worktree" && + +if grep -q "^diff --git " "$1" +then + validate_patch "$1" +else + validate_cover_letter "$1" +fi && + +if test "$GIT_SENDEMAIL_FILE_COUNTER" = "$GIT_SENDEMAIL_FILE_TOTAL" +then + git config --unset-all sendemail.validateWorktree && + trap 'git worktree remove -ff "$worktree"' EXIT && + validate_series +fi diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/update.sample b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/update.sample new file mode 100755 index 0000000..c4d426b --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/hooks/update.sample @@ -0,0 +1,128 @@ +#!/bin/sh +# +# An example hook script to block unannotated tags from entering. +# Called by "git receive-pack" with arguments: refname sha1-old sha1-new +# +# To enable this hook, rename this file to "update". +# +# Config +# ------ +# hooks.allowunannotated +# This boolean sets whether unannotated tags will be allowed into the +# repository. By default they won't be. +# hooks.allowdeletetag +# This boolean sets whether deleting tags will be allowed in the +# repository. By default they won't be. +# hooks.allowmodifytag +# This boolean sets whether a tag may be modified after creation. By default +# it won't be. +# hooks.allowdeletebranch +# This boolean sets whether deleting branches will be allowed in the +# repository. By default they won't be. +# hooks.denycreatebranch +# This boolean sets whether remotely creating branches will be denied +# in the repository. By default this is allowed. +# + +# --- Command line +refname="$1" +oldrev="$2" +newrev="$3" + +# --- Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# --- Config +allowunannotated=$(git config --type=bool hooks.allowunannotated) +allowdeletebranch=$(git config --type=bool hooks.allowdeletebranch) +denycreatebranch=$(git config --type=bool hooks.denycreatebranch) +allowdeletetag=$(git config --type=bool hooks.allowdeletetag) +allowmodifytag=$(git config --type=bool hooks.allowmodifytag) + +# check for no description +projectdesc=$(sed -e '1q' "$GIT_DIR/description") +case "$projectdesc" in +"Unnamed repository"* | "") + echo "*** Project description file hasn't been set" >&2 + exit 1 + ;; +esac + +# --- Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin &2 + echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2 + exit 1 + fi + ;; + refs/tags/*,delete) + # delete tag + if [ "$allowdeletetag" != "true" ]; then + echo "*** Deleting a tag is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/tags/*,tag) + # annotated tag + if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1 + then + echo "*** Tag '$refname' already exists." >&2 + echo "*** Modifying a tag is not allowed in this repository." >&2 + exit 1 + fi + ;; + refs/heads/*,commit) + # branch + if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then + echo "*** Creating a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/heads/*,delete) + # delete branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + refs/remotes/*,commit) + # tracking branch + ;; + refs/remotes/*,delete) + # delete tracking branch + if [ "$allowdeletebranch" != "true" ]; then + echo "*** Deleting a tracking branch is not allowed in this repository" >&2 + exit 1 + fi + ;; + *) + # Anything else (is there anything else?) + echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2 + exit 1 + ;; +esac + +# --- Finished +exit 0 diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/index b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/index new file mode 100644 index 0000000..8a6bc03 Binary files /dev/null and b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/index differ diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/info/exclude b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/info/exclude new file mode 100644 index 0000000..a5196d1 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/info/exclude @@ -0,0 +1,6 @@ +# git ls-files --others --exclude-from=.git/info/exclude +# Lines that start with '#' are comments. +# For a project mostly in C, the following would be a good set of +# exclude patterns (uncomment them if you want to use them): +# *.[oa] +# *~ diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/logs/HEAD b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/logs/HEAD new file mode 100644 index 0000000..4a14af5 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/logs/HEAD @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 aa56c53150461eebd587634d76f26cf626a18e3b t 1781107200 -0400 commit (initial): init: code, no README diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/logs/refs/heads/main b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/logs/refs/heads/main new file mode 100644 index 0000000..4a14af5 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/logs/refs/heads/main @@ -0,0 +1 @@ +0000000000000000000000000000000000000000 aa56c53150461eebd587634d76f26cf626a18e3b t 1781107200 -0400 commit (initial): init: code, no README diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/2a/508708278fea9839105388e392dda2a0b42527 b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/2a/508708278fea9839105388e392dda2a0b42527 new file mode 100644 index 0000000..1ad3453 Binary files /dev/null and b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/2a/508708278fea9839105388e392dda2a0b42527 differ diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/6e/b474c4350e80479203ab76b02a02822e6c53cb b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/6e/b474c4350e80479203ab76b02a02822e6c53cb new file mode 100644 index 0000000..33dadde Binary files /dev/null and b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/6e/b474c4350e80479203ab76b02a02822e6c53cb differ diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/90/7ed9bc9b45714bd6d0be7d42463409082cf085 b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/90/7ed9bc9b45714bd6d0be7d42463409082cf085 new file mode 100644 index 0000000..873f7ff Binary files /dev/null and b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/90/7ed9bc9b45714bd6d0be7d42463409082cf085 differ diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/aa/56c53150461eebd587634d76f26cf626a18e3b b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/aa/56c53150461eebd587634d76f26cf626a18e3b new file mode 100644 index 0000000..a3f10e6 Binary files /dev/null and b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/objects/aa/56c53150461eebd587634d76f26cf626a18e3b differ diff --git a/checkers/fixtures/doc-drift/no-readme-repo/dotgit/refs/heads/main b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/refs/heads/main new file mode 100644 index 0000000..b5d670a --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/dotgit/refs/heads/main @@ -0,0 +1 @@ +aa56c53150461eebd587634d76f26cf626a18e3b diff --git a/checkers/fixtures/doc-drift/no-readme-repo/some-service/main.py b/checkers/fixtures/doc-drift/no-readme-repo/some-service/main.py new file mode 100644 index 0000000..9985397 --- /dev/null +++ b/checkers/fixtures/doc-drift/no-readme-repo/some-service/main.py @@ -0,0 +1,2 @@ +class S: + pass diff --git a/checkers/fixtures/plan-groomer/EXPECTED_PLAN_ITEMS b/checkers/fixtures/plan-groomer/EXPECTED_PLAN_ITEMS new file mode 100644 index 0000000..7ed6ff8 --- /dev/null +++ b/checkers/fixtures/plan-groomer/EXPECTED_PLAN_ITEMS @@ -0,0 +1 @@ +5 diff --git a/checkers/fixtures/plan-groomer/README.md b/checkers/fixtures/plan-groomer/README.md new file mode 100644 index 0000000..646b045 --- /dev/null +++ b/checkers/fixtures/plan-groomer/README.md @@ -0,0 +1,39 @@ +# plan-groomer canary fixtures + +Sample sibling-checker reports for `checkers/plan-groomer.sh --canary` (offline, no network/ +token). The planner asserts the groomed-plan **item count** equals `EXPECTED_PLAN_ITEMS` +(anti-complacency floor, design §6.4). If aggregation or dedup regresses, the count drifts +and the canary FAILS (exit 3). + +## How the canary works + +`--canary` points `$REPORT_ROOT_BASE` at `sample-reports/` and writes the groomed plan into a +mode-700 temp dir (so the canary writes nothing under `$HOME`). It reads each source checker's +**latest** `/.json`, normalizes every `.findings[]` into a plan item +`{repo, severity, source, title, action}`, **dedupes** on `repo|source|title`, prioritizes by +severity, and writes the plan into the mode-600 report. + +These are plain report JSON files (no `dotgit/` trick needed — plan-groomer reads sibling +reports, it does not scan git checkouts). + +## Fixture report set + +| Source checker | Date dir | Findings | Contributes to plan | +|---|---|---|---| +| `compliance-drift` | `2026-06-10` (OLD) | 1 | **0** — sentinel: older date MUST be skipped (latest-date selection) | +| `compliance-drift` | `2026-06-17` (latest) | 3 | **2** — two of the three are an exact duplicate (`payments-dashboard` / README) that must dedup to one | +| `dependency-cve` | `2026-06-17` | 2 | **2** — `jinja2` (high) + `lodash` (critical) | +| `doc-drift` | `2026-06-17` | 1 | **1** — stale README arch section | +| `confluence-doc` | (none) | — | **0** — no report present; noted in `missing_sources`, NEVER invented as work | + +Total groomed plan items = **5** (`EXPECTED_PLAN_ITEMS`). + +This exercises four invariants in one run: +1. **latest-date selection** — the `2026-06-10` sentinel must not leak into the plan. +2. **dedup** — the duplicate README finding collapses to one item. +3. **multi-source aggregation** — three different checkers feed one prioritized plan. +4. **no-data discipline** — a missing source (`confluence-doc`) is noted, never fabricated. + +When you add/remove a source checker, a fixture report, or a finding, update the fixture(s) +and `EXPECTED_PLAN_ITEMS` in the same commit (the canary edit is itself caught on the next run +— design §6.4). diff --git a/checkers/fixtures/plan-groomer/sample-reports/compliance-drift/2026-06-10/compliance-drift.json b/checkers/fixtures/plan-groomer/sample-reports/compliance-drift/2026-06-10/compliance-drift.json new file mode 100644 index 0000000..d9be0df --- /dev/null +++ b/checkers/fixtures/plan-groomer/sample-reports/compliance-drift/2026-06-10/compliance-drift.json @@ -0,0 +1,22 @@ +{ + "checker": "compliance-drift", + "generated": "2026-06-10T03:00:00Z", + "org": "Sea-Haven-Industries", + "api_checks_ran": false, + "repos_scanned": 1, + "drift_count": 1, + "repos_with_drift": 1, + "findings": [ + { + "repo": "STALE-repo-should-be-ignored", + "id": "STALE-repo-should-be-ignored-naming-repo", + "title": "This finding is from an OLDER date and MUST NOT appear in the groomed plan", + "severity": "high", + "category": "other", + "check": "naming-repo", + "status": "confirmed", + "proof": {"outcome": "older-date sentinel: latest-date selection must skip this"} + } + ], + "skipped_checks": [] +} diff --git a/checkers/fixtures/plan-groomer/sample-reports/compliance-drift/2026-06-17/compliance-drift.json b/checkers/fixtures/plan-groomer/sample-reports/compliance-drift/2026-06-17/compliance-drift.json new file mode 100644 index 0000000..e797ee0 --- /dev/null +++ b/checkers/fixtures/plan-groomer/sample-reports/compliance-drift/2026-06-17/compliance-drift.json @@ -0,0 +1,42 @@ +{ + "checker": "compliance-drift", + "generated": "2026-06-17T03:00:00Z", + "org": "Sea-Haven-Industries", + "api_checks_ran": false, + "repos_scanned": 2, + "drift_count": 3, + "repos_with_drift": 2, + "findings": [ + { + "repo": "payments-dashboard", + "id": "payments-dashboard-readme-missing", + "title": "No README.md at repo root", + "severity": "high", + "category": "other", + "check": "readme-present", + "status": "confirmed", + "proof": {"outcome": "global CLAUDE.md / github-standards.md: every repo must have a README"} + }, + { + "repo": "payments-dashboard", + "id": "payments-dashboard-readme-missing-dup", + "title": "No README.md at repo root", + "severity": "high", + "category": "other", + "check": "readme-present", + "status": "confirmed", + "proof": {"outcome": "DUPLICATE of the row above (same repo+source+title) — must dedup to one plan item"} + }, + { + "repo": "slack-bot", + "id": "slack-bot-merge-automerge-off", + "title": "allow_auto_merge disabled", + "severity": "low", + "category": "other", + "check": "merge-settings", + "status": "confirmed", + "proof": {"outcome": "github-standards.md: enable auto-merge (allow_auto_merge)"} + } + ], + "skipped_checks": [] +} diff --git a/checkers/fixtures/plan-groomer/sample-reports/dependency-cve/2026-06-17/dependency-cve.json b/checkers/fixtures/plan-groomer/sample-reports/dependency-cve/2026-06-17/dependency-cve.json new file mode 100644 index 0000000..1c55f40 --- /dev/null +++ b/checkers/fixtures/plan-groomer/sample-reports/dependency-cve/2026-06-17/dependency-cve.json @@ -0,0 +1,32 @@ +{ + "checker": "dependency-cve", + "generated": "2026-06-17T03:05:00Z", + "org": "Sea-Haven-Industries", + "advisory_mode": "offline", + "repos_scanned": 2, + "vuln_count": 2, + "repos_with_vulns": 2, + "findings": [ + { + "repo": "payments-dashboard", + "id": "payments-dashboard-vuln-jinja2-2-11-2-GHSA-g3rq-g295-4j3m", + "title": "jinja2 2.11.2 is vulnerable (GHSA-g3rq-g295-4j3m)", + "severity": "high", + "category": "other", + "check": "vulnerable-dependency", + "status": "confirmed", + "proof": {"package": "jinja2", "version": "2.11.2", "advisory_id": "GHSA-g3rq-g295-4j3m", "summary": "Jinja2 ReDoS in the urlize filter", "fixed_version": "2.11.3"} + }, + { + "repo": "slack-bot", + "id": "slack-bot-vuln-lodash-4-17-15-GHSA-p6mc-m468-83gw", + "title": "lodash 4.17.15 is vulnerable (GHSA-p6mc-m468-83gw)", + "severity": "critical", + "category": "other", + "check": "vulnerable-dependency", + "status": "confirmed", + "proof": {"package": "lodash", "version": "4.17.15", "advisory_id": "GHSA-p6mc-m468-83gw", "summary": "Prototype pollution in lodash", "fixed_version": "4.17.19"} + } + ], + "skipped_checks": [] +} diff --git a/checkers/fixtures/plan-groomer/sample-reports/doc-drift/2026-06-17/doc-drift.json b/checkers/fixtures/plan-groomer/sample-reports/doc-drift/2026-06-17/doc-drift.json new file mode 100644 index 0000000..ac35d52 --- /dev/null +++ b/checkers/fixtures/plan-groomer/sample-reports/doc-drift/2026-06-17/doc-drift.json @@ -0,0 +1,21 @@ +{ + "checker": "doc-drift", + "generated": "2026-06-17T03:10:00Z", + "org": "Sea-Haven-Industries", + "repos_scanned": 1, + "drift_count": 1, + "repos_with_drift": 1, + "findings": [ + { + "repo": "payments-dashboard", + "id": "payments-dashboard-readme-stale-arch", + "title": "README architecture section predates the new Lambda; docs lag code", + "severity": "medium", + "category": "other", + "check": "readme-stale", + "status": "confirmed", + "proof": {"outcome": "git log shows handler change after the README's last edit"} + } + ], + "skipped_checks": [] +} diff --git a/checkers/plan-groomer.sh b/checkers/plan-groomer.sh new file mode 100755 index 0000000..996afdf --- /dev/null +++ b/checkers/plan-groomer.sh @@ -0,0 +1,316 @@ +#!/usr/bin/env bash +# plan-groomer.sh — Plane-1 planner for the R720 agent-team (REPORT-ONLY). +# +# Design refs: docs/r720-agent-team-design.md §4 (planner roster: plan-groomer — +# "Drafts a groomed weekly plan INTO the mode-600 report for now (D3); auto-write to +# Notion/Jira is a later toggle once trusted") and §7 Phase 4 ("planner + confluence-doc"). +# This mirrors compliance-drift.sh / dependency-cve.sh conventions VERBATIM so the +# coordinator (§5) can drive it identically — BUT its output discipline is different: +# it is REPORT-ONLY, not ALARM-only. +# +# WHAT IT DOES (read-only): +# Aggregates the actionable items the OTHER Plane-1 checkers already produced into a +# single prioritized "groomed weekly plan". It does NOT re-scan repos or hit any network: +# it reads the LATEST per-checker JSON reports under $REPORT_ROOT_BASE///. +# Sources consumed (each optional — a missing checker is noted, never invented as work): +# compliance-drift//compliance-drift.json (.findings[]) +# dependency-cve//dependency-cve.json (.findings[]) +# doc-drift//doc-drift.json (.findings[], if Phase-3 doc-drift exists) +# confluence-doc//confluence-doc.json (.findings[], the Phase-4 sibling) +# Each finding is normalized to a plan item {repo, severity, source, title, action}, +# DEDUPED (same repo+source+title collapses), grouped by severity then repo, and written +# into a prioritized plan in the mode-600 report (JSON + human text). +# +# REPORTING (D3 — REPORT-ONLY, the key difference from the ALARM-only checkers): +# - Writes a per-run JSON + text report under $REPORT_ROOT//, mode 600 (umask 077). +# - Slack: posts NOTHING by default. A groomed plan is a digest, not an alarm — auto-write +# to Notion/Jira (or a Slack digest) is a later toggle once signal quality is trusted (D3). +# There is intentionally NO post_slack_alarm() call in the default path; --notify is a +# future seam left inert here. A clean week (zero items) still writes an (empty) plan. +# - Reuses the substrate's redact() for the in-report digest string (defense-in-depth). +# +# SUBSTRATE REUSE (lib/sweep_substrate.sh, sourced — bash dynamic scoping): +# redact -> mask any secret-shaped value that leaked into an upstream report title. +# (discover_repos/mirror_repo/post_slack_alarm are intentionally NOT used: plan-groomer +# neither clones nor alarms — it only reads sibling reports and writes one mode-600 plan.) +# +# CANARY / DRY-RUN (offline, no network, no token): +# --canary points $REPORT_ROOT_BASE at a fixture set of sample checker reports +# (checkers/fixtures/plan-groomer/sample-reports///.json) and asserts +# the groomed-plan ITEM COUNT equals EXPECTED_PLAN_ITEMS (anti-complacency floor, design §6.4). +# If aggregation/dedup regresses, the count drifts and the canary FAILS (exit 3). --canary +# implies --dry-run. Fully offline + deterministic. --dry-run also suppresses the (inert) +# --notify seam. +# +# SCOPE / SAFETY: +# Read-only. No network, no token, no clones, no agent_team/ writes, no systemd wiring — +# that is provisioning (gated). See "PROVISIONING (NOT DONE HERE)" at the bottom. +# +# Exit: 0 = ran (always, report-only); 2 = setup/usage error; 3 = canary assertion FAILED. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[plan-groomer] $*" >&2; } +die() { echo "[plan-groomer] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate --------------------------------------------------------- +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSTRATE="$HERE/../lib/sweep_substrate.sh" +[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE" +# shellcheck source=../lib/sweep_substrate.sh +. "$SUBSTRATE" + +# --- Config + defaults (env, all optional) ------------------------------------ +GH_ORG="${GH_ORG:-Sea-Haven-Industries}" +# Base under which each checker writes its own // report tree (the parent of +# the per-checker REPORT_ROOTs the other checkers default to: $HOME/sweep-reports). +REPORT_ROOT_BASE="${REPORT_ROOT_BASE:-$HOME/sweep-reports}" +# plan-groomer's OWN report tree (separate from the checkers it reads). +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/plan-groomer}" +# Which sibling checkers to aggregate (space-separated; missing ones are noted, never invented). +SOURCE_CHECKERS="${SOURCE_CHECKERS:-compliance-drift dependency-cve doc-drift confluence-doc}" + +DRY_RUN=0 # --dry-run: suppress the (inert) --notify seam (report still written). +CANARY=0 # --canary: read the fixture report set + assert the known plan-item count. +NOTIFY=0 # --notify: INERT future seam (post the digest somewhere). Off by default (D3). +TARGETS_OVERRIDE="" # --targets "a b": restrict the groomed plan to these repo names only. + +usage() { + cat >&2 </dev/null || die "jq is required" + +# --- Report dir (mode 600 reports; matches sweep conventions) ----------------- +umask 077 +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + +# Canary redirects the SOURCE base at the fixture report set; output stays in a temp area so +# the canary writes nothing under $HOME. +if [ "$CANARY" -eq 1 ]; then + FIXTURE_ROOT="$HERE/fixtures/plan-groomer" + [ -d "$FIXTURE_ROOT/sample-reports" ] || die "canary fixture missing: $FIXTURE_ROOT/sample-reports" + REPORT_ROOT_BASE="$FIXTURE_ROOT/sample-reports" + CANARY_OUT="$(mktemp -d "${TMPDIR:-/tmp}/plan-groomer-canary.XXXXXX")" + trap 'rm -rf "$CANARY_OUT"' EXIT + REPORT_ROOT="$CANARY_OUT/plan-groomer" + # Pin the source list the fixtures were authored against (deterministic regardless of env). + SOURCE_CHECKERS="compliance-drift dependency-cve doc-drift confluence-doc" +fi + +REPORT_DIR="$REPORT_ROOT/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +# shellcheck disable=SC2034 # named for parity with the ALARM-only checkers' substrate contract +SWEEP_LOG="$REPORT_DIR/plan-groomer.log" +REPORT_JSON="$REPORT_DIR/plan-groomer.json" +REPORT_TXT="$REPORT_DIR/plan-groomer.txt" + +log "=== plan-groomer $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN notify=$NOTIFY) ===" + +# ------------------------------------------------------------------------------ +# Severity ordering: the jq sort below maps severity->rank; no shell helper needed. +# ------------------------------------------------------------------------------ +in_csv() { # needle space-list -> 0 if present + local n="$1" list="$2" t; for t in $list; do [ "$t" = "$n" ] && return 0; done; return 1 +} + +# --- Resolve the LATEST date dir for one checker under $REPORT_ROOT_BASE ------- +latest_report_json() { # checker_name -> path to its latest /.json, or "" if none + local checker="$1" + local base="$REPORT_ROOT_BASE/$checker" d name latest="" + [ -d "$base" ] || { echo ""; return 0; } + # Date dirs are YYYY-MM-DD; lexical sort == chronological. Glob the dirs, keep only + # YYYY-MM-DD names, sort newest-first, pick the newest that actually has the JSON. + for d in $(for p in "$base"/*/; do + [ -d "$p" ] || continue + name="$(basename "$p")" + [[ "$name" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] && echo "$name" + done | sort -r); do + if [ -f "$base/$d/$checker.json" ]; then latest="$base/$d/$checker.json"; break; fi + done + echo "$latest" +} + +# ============================================================================== +# AGGREGATE: pull findings from each sibling checker's latest report into plan items. +# ============================================================================== +declare -a PLAN_ITEMS=() # one normalized JSON plan-item per upstream finding +declare -a MISSING_SOURCES=() # checkers with no report found — noted, NEVER invented as work +note_missing() { MISSING_SOURCES+=( "$1" ); } + +for checker in $SOURCE_CHECKERS; do + rj="$(latest_report_json "$checker")" + if [ -z "$rj" ]; then + note_missing "$checker(no-report-found)" + log " [$checker] no report under $REPORT_ROOT_BASE/$checker — skipped (not invented as work)" + continue + fi + if ! jq -e '.findings | type=="array"' "$rj" >/dev/null 2>&1; then + note_missing "$checker(report-unparseable)" + log " [$checker] report $rj has no .findings[] array — skipped" + continue + fi + cnt="$(jq '.findings | length' "$rj")" + log " [$checker] $rj -> $cnt finding(s)" + # Normalize each finding to a plan item. Title/severity/repo come straight off the finding + # schema the checkers emit (see finding.schema.json spirit). The "action" is a stable, + # source-derived hint (no fabrication — it just labels what kind of remediation this is). + while IFS= read -r item; do + [ -n "$item" ] && PLAN_ITEMS+=( "$item" ) + done < <(jq -c --arg src "$checker" ' + .findings[] + | { repo: (.repo // "unknown"), + severity: (.severity // "medium"), + source: $src, + title: (.title // .id // "untitled"), + action: ( if $src=="dependency-cve" then "upgrade vulnerable dependency" + elif $src=="compliance-drift" then "fix convention drift" + elif $src=="doc-drift" then "reconcile docs with code" + elif $src=="confluence-doc" then "close documentation gap" + else "review finding" end ) } + ' "$rj") +done + +# --- Optional repo-scope restriction (--targets) ------------------------------ +if [ -n "$TARGETS_OVERRIDE" ] && [ "${#PLAN_ITEMS[@]}" -gt 0 ]; then + declare -a FILTERED=() + for item in "${PLAN_ITEMS[@]}"; do + r="$(echo "$item" | jq -r '.repo')" + in_csv "$r" "$TARGETS_OVERRIDE" && FILTERED+=( "$item" ) + done + PLAN_ITEMS=( "${FILTERED[@]+"${FILTERED[@]}"}" ) + log "targets filter '$TARGETS_OVERRIDE' -> ${#PLAN_ITEMS[@]} item(s)" +fi + +# ============================================================================== +# DEDUP + PRIORITIZE: collapse identical (repo|source|title), then sort by severity desc, +# then repo, then source. Add a stable rank int so downstream consumers can re-sort. +# ============================================================================== +if [ "${#PLAN_ITEMS[@]}" -gt 0 ]; then + RAW_JSON="$(printf '%s\n' "${PLAN_ITEMS[@]}" | jq -cs .)" +else + RAW_JSON="[]" +fi + +GROOMED_JSON="$(echo "$RAW_JSON" | jq -c ' + # dedup on repo|source|title + ( reduce .[] as $x ({}; .[($x.repo+"|"+$x.source+"|"+$x.title)] //= $x) ) | [ .[] ] + | map(. + { rank: ( {critical:4, high:3, medium:2, low:1}[.severity] // 0 ) }) + | sort_by([ (-.rank), .repo, .source, .title ]) +')" + +N_ITEMS="$(echo "$GROOMED_JSON" | jq 'length')" +N_CRIT_HIGH="$(echo "$GROOMED_JSON" | jq '[.[]|select(.severity=="critical" or .severity=="high")]|length')" +N_REPOS="$(echo "$GROOMED_JSON" | jq '[.[].repo]|unique|length')" + +if [ "${#MISSING_SOURCES[@]}" -gt 0 ]; then + MISSING_JSON="$(printf '%s\n' "${MISSING_SOURCES[@]}" | jq -R . | jq -cs .)" +else + MISSING_JSON="[]" +fi + +# ============================================================================== +# ASSEMBLE REPORT (JSON + text), mode 600 +# ============================================================================== +jq -n \ + --arg planner "plan-groomer" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \ + --arg sources "$SOURCE_CHECKERS" \ + --argjson items "$GROOMED_JSON" --argjson missing "$MISSING_JSON" \ + '{planner:$planner, generated:$ts, org:$org, mode:"report-only", + sources_considered:($sources|split(" ")), + plan_item_count:($items|length), + crit_high_count:([$items[]|select(.severity=="critical" or .severity=="high")]|length), + repos_in_plan:([$items[].repo]|unique|length), + plan:$items, missing_sources:$missing}' > "$REPORT_JSON" + +{ + echo "plan-groomer — groomed weekly plan — $UTC_STAMP" + echo "org=$GH_ORG sources=[$SOURCE_CHECKERS] mode=report-only (D3: no auto-write)" + echo "plan items: $N_ITEMS ($N_CRIT_HIGH crit/high) across $N_REPOS repo(s)" + echo + if [ "$N_ITEMS" -gt 0 ]; then + echo "PRIORITIZED PLAN (severity desc, then repo):" + echo "$GROOMED_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo) — \(.title)\n action: \(.action) (source: \(.source))"' + else + echo "No actionable items aggregated this run (clean week, or no upstream reports)." + fi + if [ "$(echo "$MISSING_JSON" | jq 'length')" -gt 0 ]; then + echo; echo "sources with no report (NOT invented as work):" + echo "$MISSING_JSON" | jq -r '.[] | " - \(.)"' + fi +} > "$REPORT_TXT" +chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true + +# Defense-in-depth: the digest line that a future --notify seam would push is redacted now. +DIGEST="$(echo "$GROOMED_JSON" | jq -r ' + group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' \ + | sed 's/^/• /' | redact)" + +log "report: $REPORT_JSON ($N_ITEMS plan item(s), $N_REPOS repo(s))" + +# ============================================================================== +# CANARY ASSERTION (anti-complacency floor, design §6.4) +# ============================================================================== +if [ "$CANARY" -eq 1 ]; then + EXPECT_FILE="$HERE/fixtures/plan-groomer/EXPECTED_PLAN_ITEMS" + [ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE" + EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")" + log "canary assertion: expected plan items=$EXPECTED, got=$N_ITEMS" + if [ "$N_ITEMS" -ne "$EXPECTED" ]; then + echo "[plan-groomer] CANARY FAIL: groomed-plan item count mismatch (expected $EXPECTED, got $N_ITEMS)" >&2 + echo " -> aggregation or dedup regressed, or the fixture changed. See $REPORT_TXT." >&2 + exit 3 + fi + log "canary PASS: groomed plan has all $EXPECTED expected item(s) (dedup intact)." +fi + +# ============================================================================== +# REPORT-ONLY ROUTING (D3): the plan lives in the mode-600 report. Post NOTHING. +# ============================================================================== +if [ "$NOTIFY" -eq 1 ] && [ "$DRY_RUN" -eq 0 ]; then + # INERT future seam: when D3's "once trusted" toggle flips, this is where the digest would + # be pushed to Slack/Notion/Jira. It is intentionally a no-op in this phase — plan-groomer + # is REPORT-ONLY and must not auto-write. The composed digest is available in $DIGEST. + log "--notify requested but inert in this phase (D3: report-only; auto-write is a later toggle). No push." +fi +: "${DIGEST:?}" >/dev/null 2>&1 || true # DIGEST is composed for the future seam; keep it referenced. +log "REPORT-ONLY: groomed plan written to the mode-600 report; nothing posted (D3)." +exit 0 + +# ============================================================================== +# PROVISIONING (NOT DONE HERE — gated, later phases): +# - REPORT-ONLY by design (D3). The auto-write path (push the groomed plan to Notion/Jira, +# or a weekly Slack digest) is a LATER TOGGLE, flipped only once signal quality is trusted. +# The --notify seam above is intentionally inert; wiring a real destination is provisioning. +# - No systemd unit / timer is installed by this script. Wiring it into the weekly schedule +# (alongside the other Plane-1 checkers under the coordinator) is provisioning and is gated. +# - The coordinator (design §5, checker_coordinator.sh) registers + drives this planner; that +# registry edit is done centrally, NOT in this script. +# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations for the +# build session, tracked outside this script. +# ============================================================================== diff --git a/finding.schema.json b/finding.schema.json new file mode 100644 index 0000000..52be56e --- /dev/null +++ b/finding.schema.json @@ -0,0 +1,69 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Sea Haven security-review finding", + "description": "Structured finding contract for /sh-security-review. Same shape for interactive (Path A) and automated (Path B) runs, and the input review.sh reads to make the block decision.", + "type": "object", + "required": ["findings", "summary"], + "properties": { + "findings": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "title", "severity", "cwe", "file", "category", "data_flow", "proof", "status"], + "properties": { + "id": { "type": "string", "description": "stable slug, e.g. sqli-payment-handler-get-payment" }, + "title": { "type": "string" }, + "severity": { + "type": "string", + "enum": ["critical", "high", "medium", "low", "info", "unverified"], + "description": "unverified = a claim with no accepted proof; auto-downgraded from its claimed severity" + }, + "claimed_severity": { + "type": "string", + "enum": ["critical", "high", "medium", "low", "info"], + "description": "the detector's original severity before the verifier ruled" + }, + "cwe": { "type": "string", "pattern": "^CWE-[0-9]+$" }, + "file": { "type": "string" }, + "line": { "type": ["integer", "null"] }, + "category": { + "type": "string", + "enum": ["injection", "authz", "secrets-crypto", "iac-iam", "web-client", "logic", "other"] + }, + "data_flow": { + "type": "string", + "description": "numbered plain-English trace from untrusted source to dangerous sink" + }, + "proof": { + "type": "object", + "required": ["input", "outcome"], + "properties": { + "input": { "type": "string", "description": "concrete malicious input / trigger" }, + "outcome": { "type": "string", "description": "the specific bad result it produces" }, + "test": { "type": ["string", "null"], "description": "optional failing-test sketch" } + } + }, + "status": { + "type": "string", + "enum": ["confirmed", "unverified", "suppressed"], + "description": "confirmed = verifier accepted proof; unverified = no accepted proof; suppressed = dismissed with justification" + }, + "suppression_justification": { + "type": ["string", "null"], + "description": "REQUIRED when status=suppressed; logged and surfaced in the report" + }, + "recommendation": { "type": "string" } + } + } + }, + "summary": { + "type": "object", + "required": ["confirmed_critical", "confirmed_high", "block"], + "properties": { + "confirmed_critical": { "type": "integer" }, + "confirmed_high": { "type": "integer" }, + "block": { "type": "boolean", "description": "true if any confirmed critical/high is unsuppressed (the gate condition)" } + } + } + } +} diff --git a/hooks/pre-commit b/hooks/pre-commit new file mode 100755 index 0000000..268973d --- /dev/null +++ b/hooks/pre-commit @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Sea Haven security-review pre-commit hook: FAST deterministic scanners only (sub-30s). +# The full agentic review is the on-demand /sh-security-review slash command — run that before pushing. +# Honors the same skip/suppress controls as pre-push so a suppressed FP doesn't block the commit. +# --no-verify skips this local fast feedback; the pre-push hook + nightly VM sweep are the backstop. +set -uo pipefail +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || exit 0 +[ -f "$REPO_ROOT/.security-review-skip" ] && exit 0 +REVIEW_SH="${SH_REVIEW_SH:-$HOME/Documents/repositories/orchestrator/security-review/review.sh}" +if [ ! -f "$REVIEW_SH" ]; then + echo "security-review: review.sh not found at $REVIEW_SH (set SH_REVIEW_SH to override) — skipping" >&2 + exit 0 +fi +# Nothing staged -> nothing to do. +git diff --cached --name-only --diff-filter=ACM | grep -q . || exit 0 +SUP=() +# Suppressions: prefer a MACHINE-LEVEL file kept out of repo history +# (//suppressions.json), else fall back to a repo-local +# .security-review/suppressions.json. Keyed by repo basename — adequate for the +# current single-namespace layout under ~/Documents/repositories. +MACHINE_SUP="${SH_SECURITY_SUPPRESSIONS_DIR:-$HOME/.config/sea-haven/security-review}/$(basename "$REPO_ROOT")/suppressions.json" +if [ -f "$MACHINE_SUP" ]; then + SUP=(--suppressions "$MACHINE_SUP") +elif [ -f "$REPO_ROOT/.security-review/suppressions.json" ]; then + SUP=(--suppressions "$REPO_ROOT/.security-review/suppressions.json") +fi +# ${SUP[@]+"${SUP[@]}"} = bash-3.2-safe expansion of a possibly-empty array under set -u. +exec bash "$REVIEW_SH" --scanners-only ${SUP[@]+"${SUP[@]}"} "$REPO_ROOT" diff --git a/hooks/pre-push b/hooks/pre-push new file mode 100755 index 0000000..7ca47df --- /dev/null +++ b/hooks/pre-push @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Sea Haven global pre-push security gate — fast deterministic scanners (review.sh --scanners-only). +# Installed via install-hooks.sh --global: lays this down at ~/.config/git/hooks/pre-push and sets +# git config --global core.hooksPath ~/.config/git/hooks +# Skip a repo: add a .security-review-skip file at its root. Bypass once: git push --no-verify. +# Deep agentic pass = on-demand /sh-security-review; nightly VM sweep = the backstop. +set -uo pipefail +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || exit 0 +[ -f "$REPO_ROOT/.security-review-skip" ] && exit 0 +REVIEW_SH="${SH_REVIEW_SH:-$HOME/Documents/repositories/orchestrator/security-review/review.sh}" +if [ -f "$REVIEW_SH" ]; then + SUP=() + # Suppressions: prefer a MACHINE-LEVEL file kept out of repo history + # (//suppressions.json), else fall back to a repo-local + # .security-review/suppressions.json. Keyed by repo basename — adequate for the + # current single-namespace layout under ~/Documents/repositories. + MACHINE_SUP="${SH_SECURITY_SUPPRESSIONS_DIR:-$HOME/.config/sea-haven/security-review}/$(basename "$REPO_ROOT")/suppressions.json" + if [ -f "$MACHINE_SUP" ]; then + SUP=(--suppressions "$MACHINE_SUP") + elif [ -f "$REPO_ROOT/.security-review/suppressions.json" ]; then + SUP=(--suppressions "$REPO_ROOT/.security-review/suppressions.json") + fi + echo "security-review: scanning $REPO_ROOT (scanners-only) before push..." >&2 + # ${SUP[@]+"${SUP[@]}"} = bash-3.2-safe expansion of a possibly-empty array under set -u. + if ! bash "$REVIEW_SH" --scanners-only ${SUP[@]+"${SUP[@]}"} "$REPO_ROOT"; then + echo "security-review: BLOCKED (confirmed crit/high). Fix it, suppress with justification, or 'git push --no-verify' to override." >&2 + exit 1 + fi +else + echo "security-review: review.sh not found at $REVIEW_SH (set SH_REVIEW_SH) — skipping gate" >&2 +fi +# Don't silently disable a repo-local pre-push hook: chain to it if present. +LOCAL_HOOK="$REPO_ROOT/.git/hooks/pre-push" +[ -x "$LOCAL_HOOK" ] && exec "$LOCAL_HOOK" "$@" +exit 0 diff --git a/iam/CROSS-REVIEW-PACKET.md b/iam/CROSS-REVIEW-PACKET.md new file mode 100644 index 0000000..f30ef3e --- /dev/null +++ b/iam/CROSS-REVIEW-PACKET.md @@ -0,0 +1,182 @@ +# Cross-review packet — R720 aws-posture IAM (step-ca → Roles Anywhere → read-only AWS role) + +> **GPT-4.1 cross-review 2026-06-18: APPROVE, no BLOCKs; FIXes applied** +> (`aws:SourceAccount` added to the trust policy; `ec2:DescribeImages` removed from the +> permission policy). NIT answers recorded in `aws-posture-readonly-policy.rationale.md`: +> snapshots = account-owned idle-spend signal (kept); `s3:ListAllMyBuckets` = names-only +> inventory, no object data (kept); no `logs:*` needed; `aws:RequestedRegion` deliberately +> SKIPPED (global-endpoint `ce:*`/`s3:ListAllMyBuckets` could be DENYed by a blanket region pin). + +**Audience:** the mandatory GPT-4.1 IAM cross-review + Adam. +**Status:** these are AUTHORED FILES, nothing is applied to AWS. The review has now PASSED +(APPROVE, no BLOCKs), which unblocks **building** aws-posture (done in this Phase-3 change set, +PROVISIONING-GATED — the checker makes no AWS call until step-ca + Roles Anywhere are stood up). +Approving this packet unblocks provisioning; it does not itself change AWS. + +**Account:** 328440206208 · **Region:** us-east-1 · **Box:** the always-on R720 secrev VM +(single-user, unattended, currently holds a long-lived read-only GitHub PAT). + +## Why this exists + +aws-posture (design D5 / §4) is a weekly idle/anomalous-spend watch (the design flags ≈$330/mo +of waste). It must read Cost Explorer + utilization metrics + an idle-resource inventory from +the account. The decision (D5): **the box stays read-only, and it authenticates to AWS via IAM +Roles Anywhere using short-lived leaf certs issued by a new internal step-ca — NO long-lived +AWS access key on the box.** The short-lived self-expiring leaf is strictly stronger than the +box's existing long-lived PAT. + +## Files in this packet + +| File | What it is | +|---|---| +| `aws-posture-readonly-policy.json` | The least-privilege **permission policy** (valid IAM JSON, applyable as-is). | +| `aws-posture-readonly-policy.rationale.md` | Statement-by-statement least-privilege rationale (IAM JSON can't hold comments). | +| `aws-posture-trust-policy.json` | The role's **trust policy** — who may assume it (Roles Anywhere + pinned cert CN/issuer + pinned trust-anchor ARN). | +| `roles-anywhere-config.json` | The Roles Anywhere **trust anchor + profile** config (pins the step-ca root; 1h session cap). | +| `step-ca-config-sketch.md` | The internal **CA** config + the systemd-timer **auto-renewal** approach. | +| `CROSS-REVIEW-PACKET.md` | This document. | + +## Trust model, end to end + +``` +step-ca ROOT cert (CN="Sea Haven Internal CA - R720 Roles Anywhere") + │ pinned as the Roles Anywhere trust anchor (roles-anywhere-config.json) + ▼ +step-ca issues a SHORT-LIVED leaf (CN="r720-aws-posture", ~24h, auto-renewed hourly by systemd timer) + │ stored mode-600 on the box; private key never leaves the box; no AWS key on disk + ▼ +box calls AWS via aws_signing_helper credential-process, signing with the leaf + ▼ +IAM Roles Anywhere trust anchor (r720-aws-posture-step-ca) + │ validates: leaf chains to the pinned root? yes -> emit session tags + │ aws:PrincipalTag/x509Subject/CN = "r720-aws-posture" + │ aws:PrincipalTag/x509Issuer/CN = "Sea Haven Internal CA - R720 Roles Anywhere" + ▼ +Roles Anywhere profile (r720-aws-posture-readonly, durationSeconds=3600) + │ binds ONLY the one role + ▼ +sts:AssumeRole on role/r720-aws-posture-readonly + │ trust policy (aws-posture-trust-policy.json) requires ALL of: + │ (1) Principal = rolesanywhere.amazonaws.com (came via Roles Anywhere) + │ (2) aws:SourceArn = THIS trust anchor (not some other anchor) + │ (2b) aws:SourceAccount = 328440206208 (confused-deputy guard, added in cross-review) + │ (3) x509Subject/CN = "r720-aws-posture" AND x509Issuer/CN = the internal CA + ▼ +1-hour STS session, permissions = aws-posture-readonly-policy.json (read-only cost + idle inventory) + ▼ +read-only AWS APIs: ce:Get*, cloudwatch:GetMetric*/DescribeAlarms, ec2/elb/rds:Describe*, + lambda:List/GetFunctionConfiguration, s3:ListAllMyBuckets/GetBucketLocation +``` + +Three independent conditions must ALL hold to assume the role: via Roles Anywhere, from THIS +anchor (in THIS account, via the `aws:SourceAccount` guard added in cross-review), presenting a +leaf with the pinned subject CN + issuer CN. Any one missing → AssumeRole denied. + +## Least-privilege rationale (summary; full table in the rationale .md) + +- **Read-only only.** No write/modify/delete verb in any service. No `iam:*` mutation, no + privilege-escalation path, no `sts` onward-chaining. +- **`Resource: "*"` only where AWS gives no choice.** Cost Explorer, the CloudWatch metric-data + calls, and the EC2/ELB/RDS `Describe*` list operations do not support resource-level ARNs; + least-privilege there is the **action allow-list**, not the resource. There are no wildcard + *actions* (`ce:*`, `ec2:*`) anywhere — every action is an explicit read verb. +- **No data-plane reads.** Deliberately excludes `s3:GetObject`, `secretsmanager:GetSecretValue`, + `ssm:GetParameter*`, `kms:Decrypt`, `logs:GetLogEvents`. The role enumerates and prices the + account; it cannot read application data, secrets, or logs. +- **Tighter than the AWS-managed `ReadOnlyAccess`/`ViewOnlyAccess`** (those include object reads, + table reads, etc.) — this is the small cost+inventory subset aws-posture actually queries. + +## Blast radius if the leaf (or its private key) is compromised + +- **Ceiling = read-only enumeration + pricing of account 328440206208 for ≤1 hour per session** + (STS `durationSeconds=3600`), and only while a valid unexpired leaf exists (~24h leaf life). +- **Cannot:** read S3 object data, read secrets/SSM params, read CloudWatch *logs*, modify or + delete any resource, touch IAM, assume any other role, or act in any other account/region + scope beyond what read-only describe calls expose. +- **Containment levers, fastest first:** + 1. **Disable the Roles Anywhere profile or trust anchor** (`enabled:false`) → immediately stops + all new credential vending, regardless of leaf validity. (seconds) + 2. **Detach/empty the role's permission policy** → any still-live session loses all access at + the next AWS authz check. (seconds) + 3. **Revoke at the CA / rotate the leaf** → step-ca stops renewing; the leaf self-expires within + its ≤24h window even with no action. +- The self-expiring leaf means even a "do nothing" outcome bounds exposure to the leaf lifetime — + unlike the box's current long-lived PAT, which would persist until manually rotated. + +## Rollback (EXERCISED, not just written — design §7 "exercised, not merely written") + +Teardown order is the reverse of provisioning; each step is independently sufficient to cut +access. Tested via a simulated teardown/re-provision against throwaway names (see "Exercise" +below) before this packet is accepted. + +```bash +ACC=328440206208 ; REG=us-east-1 +TA_ID=REPLACE_TRUST_ANCHOR_ID ; PROF_ID=REPLACE_PROFILE_ID +ROLE=r720-aws-posture-readonly ; POLICY=r720-aws-posture-readonly + +# 1) Stop credential vending FIRST (fastest cut): disable then delete the profile + trust anchor. +aws rolesanywhere disable-profile --profile-id "$PROF_ID" --region "$REG" +aws rolesanywhere disable-trust-anchor --trust-anchor-id "$TA_ID" --region "$REG" +aws rolesanywhere delete-profile --profile-id "$PROF_ID" --region "$REG" +aws rolesanywhere delete-trust-anchor --trust-anchor-id "$TA_ID" --region "$REG" + +# 2) Remove the role + its permission policy. +POLICY_ARN="arn:aws:iam::$ACC:policy/$POLICY" +aws iam detach-role-policy --role-name "$ROLE" --policy-arn "$POLICY_ARN" +aws iam delete-role --role-name "$ROLE" +aws iam delete-policy --policy-arn "$POLICY_ARN" + +# 3) Remove the internal CA + the leaf on the box (no AWS state involved). +sudo systemctl disable --now aws-posture-cert-renew.timer step-ca.service +sudo rm -f /etc/aws-posture/leaf.crt /etc/aws-posture/leaf.key +sudo rm -rf /etc/step-ca # destroys root/intermediate/keys -> no further leaves issuable +# (optional) remove the [profile r720-aws-posture] block from ~/.aws/config +``` + +**Re-provision** = re-run `step ca init` (step-ca-config-sketch.md) → re-create role + policy + +trust anchor + profile → drop in the new trust-anchor/profile ARNs. A revert point (VM snapshot +per `feedback_ec2_replacement_snapshot`) is taken before provisioning so the whole change is one +snapshot-restore away from gone. + +### Exercise log (to be completed before acceptance) + +> Run the provision → assume-once (confirm read-only works, confirm a write is denied) → run the +> rollback above against throwaway-suffixed names → confirm AssumeRole now fails and the CA is +> gone. Paste the transcript here. Until this is filled in, the rollback is "written, not +> exercised" and the phase is NOT accepted (design §7). + +## Specific things for the cross-reviewer to scrutinize (with resolutions) + +1. **Trust-policy condition completeness.** Are `aws:PrincipalTag/x509Subject/CN` + + `aws:PrincipalTag/x509Issuer/CN` + `ArnEquals aws:SourceArn` (the trust anchor) sufficient + to prevent any other cert (or another trust anchor in the account) from assuming the role? + Is there a confused-deputy gap I should also pin (e.g. should I add `aws:SourceAccount`)? + → **RESOLVED (FIX applied):** added `aws:SourceAccount = 328440206208` to the StringEquals + condition. The trust now pins anchor (SourceArn) **and** account (SourceAccount) plus the + cert CN/issuer — closing the confused-deputy gap the reviewer raised. +2. **`Resource: "*"` statements.** Confirm each is an API that genuinely has no resource-level + support, and that no statement could be tightened with a condition (e.g. `aws:RequestedRegion` + = us-east-1) without breaking the checker. + → **RESOLVED (NIT, SKIPPED with rationale):** every `Resource:"*"` statement is an + API family without resource-level ARNs (ce, the cloudwatch metric-data calls, ec2/elb/rds + `Describe*`). `aws:RequestedRegion` is **NOT** applied — `ce:*` and `s3:ListAllMyBuckets` are + global-endpoint services that a blanket region condition could DENY. Full reasoning in + `aws-posture-readonly-policy.rationale.md` ("Cross-review NIT answers"). +3. **Action allow-list.** Anything in here that is NOT needed for idle/anomalous-spend (i.e. + over-grant), or any read verb that leaks data we don't want (the intent is pricing + inventory, + no object/secret/log data). + → **RESOLVED (FIX applied):** removed `ec2:DescribeImages` (AMIs are not an idle-spend signal). + `ec2:DescribeSnapshots` kept (orphan-snapshot waste, account-owned metadata only); + `s3:ListAllMyBuckets` kept (bucket *names* only, no `s3:GetObject`); no `logs:*` granted. +4. **Session duration vs leaf lifetime.** 1h STS session + ~24h leaf — acceptable blast window? + → Accepted as-is (no change requested). +5. **Rollback ordering.** Is disabling the profile/anchor first the correct fastest-cut order, + and does step 2/3 leave any orphaned grant? + → Accepted as-is (no change requested). + +## Process note + +Per global instructions this IAM change ALSO requires the GPT-4.1 cross-family review run via +`python3 ~/Documents/repositories/orchestrator/run.py ""` (it is an IAM +role/policy + trust-anchor change). This packet is the input to that review; aws-posture is not +built until the review is recorded. diff --git a/iam/README.md b/iam/README.md new file mode 100644 index 0000000..82add38 --- /dev/null +++ b/iam/README.md @@ -0,0 +1,30 @@ +# security-review/iam/ — Phase-3 IAM artifacts (authored for cross-review, NOT applied) + +These are the IAM / Roles Anywhere / step-ca artifacts for the R720 agent-team **aws-posture** +checker (design `docs/r720-agent-team-design.md` D5 / §4 / §6.3 / §7 Phase 3). + +**Nothing here is applied to AWS.** They are FILES for the mandatory GPT-4.1 IAM cross-review. + +**Cross-review status (2026-06-18): APPROVE, no BLOCKs.** FIXes applied — `aws:SourceAccount` +added to the trust policy; `ec2:DescribeImages` removed from the permission policy (see +`CROSS-REVIEW-PACKET.md` header + `aws-posture-readonly-policy.rationale.md`). The review passing +**unblocked building** `../checkers/aws-posture.sh` (built in this Phase-3 change set). That +checker stays **PROVISIONING-GATED**: it makes NO AWS call until step-ca + the Roles Anywhere +trust anchor + this role are stood up. Provisioning happens only after the review is recorded +(design §7, B3) — and a VM snapshot is taken first per `feedback_ec2_replacement_snapshot`. + +Decision (D5): the box stays **read-only** and authenticates to AWS via **Roles Anywhere** +short-lived leaf certs issued by a new internal **step-ca** — **no long-lived AWS key on the +box**. + +| File | Purpose | +|---|---| +| `CROSS-REVIEW-PACKET.md` | **Start here.** End-to-end trust model, least-privilege rationale, blast radius, exercised rollback, and the specific items for the reviewer. | +| `aws-posture-readonly-policy.json` | Least-privilege read-only permission policy (valid, applyable IAM JSON). | +| `aws-posture-readonly-policy.rationale.md` | Statement-by-statement rationale (IAM JSON can't carry comments). | +| `aws-posture-trust-policy.json` | Role trust policy — pins Roles Anywhere + the leaf subject/issuer CN + trust-anchor ARN. | +| `roles-anywhere-config.json` | Trust-anchor (pins step-ca root) + profile (1h session) config. | +| `step-ca-config-sketch.md` | Internal CA config + systemd-timer auto-renewal of the short-lived leaf. | + +Per global instructions this IAM change also requires the GPT-4.1 cross-family review via +`orchestrator/run.py`; this directory is that review's input. diff --git a/iam/aws-posture-readonly-policy.json b/iam/aws-posture-readonly-policy.json new file mode 100644 index 0000000..482fec6 --- /dev/null +++ b/iam/aws-posture-readonly-policy.json @@ -0,0 +1,71 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CostAndUsageReadOnly", + "Effect": "Allow", + "Action": [ + "ce:GetCostAndUsage", + "ce:GetCostAndUsageWithResources", + "ce:GetCostForecast", + "ce:GetDimensionValues", + "ce:GetTags", + "ce:GetReservationUtilization", + "ce:GetSavingsPlansUtilization", + "ce:GetAnomalies", + "ce:GetAnomalyMonitors", + "ce:GetAnomalySubscriptions" + ], + "Resource": "*" + }, + { + "Sid": "CloudWatchMetricsReadOnly", + "Effect": "Allow", + "Action": [ + "cloudwatch:GetMetricData", + "cloudwatch:GetMetricStatistics", + "cloudwatch:ListMetrics", + "cloudwatch:DescribeAlarms", + "cloudwatch:DescribeAlarmsForMetric" + ], + "Resource": "*" + }, + { + "Sid": "Ec2DescribeReadOnly", + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeInstanceStatus", + "ec2:DescribeVolumes", + "ec2:DescribeAddresses", + "ec2:DescribeNatGateways", + "ec2:DescribeSnapshots", + "ec2:DescribeRegions" + ], + "Resource": "*" + }, + { + "Sid": "ElbAndRdsDescribeReadOnly", + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:DescribeLoadBalancers", + "elasticloadbalancing:DescribeTargetGroups", + "elasticloadbalancing:DescribeTargetHealth", + "rds:DescribeDBInstances", + "rds:DescribeDBClusters" + ], + "Resource": "*" + }, + { + "Sid": "LambdaAndStorageInventoryReadOnly", + "Effect": "Allow", + "Action": [ + "lambda:ListFunctions", + "lambda:GetFunctionConfiguration", + "s3:ListAllMyBuckets", + "s3:GetBucketLocation" + ], + "Resource": "*" + } + ] +} diff --git a/iam/aws-posture-readonly-policy.rationale.md b/iam/aws-posture-readonly-policy.rationale.md new file mode 100644 index 0000000..94c4b09 --- /dev/null +++ b/iam/aws-posture-readonly-policy.rationale.md @@ -0,0 +1,77 @@ +# aws-posture-readonly-policy.json — least-privilege rationale + +This is the annotated companion to `aws-posture-readonly-policy.json`. The policy JSON itself +is kept strictly valid (no inline `Comment` keys — IAM rejects those), so all rationale lives +here. This policy is the permission set for the **aws-posture** checker (design D5 / §4): +idle / anomalous-spend watch on the Sea Haven AWS account (328440206208, us-east-1). + +**Cross-review status (2026-06-18):** GPT-4.1 IAM cross-review returned **APPROVE, no BLOCKs**. +FIXes applied to the policy as a result: +- **Removed `ec2:DescribeImages`** (data minimization — AMIs are not part of the idle-spend + signal; orphan EBS snapshots already cover the storage-waste case via `ec2:DescribeSnapshots`). +- The trust policy (`aws-posture-trust-policy.json`) gained **`aws:SourceAccount` = + `328440206208`** as an extra confused-deputy guard alongside the existing `aws:SourceArn` + trust-anchor pin (see that file). + +**aws-posture itself is built in Phase-3 (this change set) but stays PROVISIONING-GATED** — the +checker never calls AWS until step-ca + Roles Anywhere (this packet) are stood up. This file + the +policy are the IAM cross-review inputs (design §7, B3). + +## Design principle + +The box stays **read-only**. There is **no write action, no `iam:*` mutating action, no +`Resource` wildcard where AWS supports resource-level scoping**. Idle-spend posture is an +account-wide, list-oriented read: most of the actions below are AWS APIs that *do not support +resource-level ARNs at all* (Cost Explorer, the CloudWatch metric-data calls, and the EC2/ELB/ +RDS `Describe*` list operations). For those, least-privilege is enforced by the **action +allow-list** (only the specific read verbs), not by narrowing `Resource`. + +## Statement-by-statement + +| Sid | Why aws-posture needs it | Why read-only / why `Resource: "*"` | +|---|---|---| +| `CostAndUsageReadOnly` | The core idle/anomalous-spend signal (the design flags ≈$330/mo). `GetCostAndUsage`, forecasts, dimensions, and the native CE anomaly detectors. | Cost Explorer is an account-scoped service; its API has no resource-level ARNs, so `Resource:*` is the only valid form. Only `Get*` verbs — no `ce:Update*/Create*/Delete*`, no budget mutation. | +| `CloudWatchMetricsReadOnly` | Correlate spend with utilization (an instance billing but at ~0% CPU is idle). `GetMetricData`/`GetMetricStatistics`/`ListMetrics`; `DescribeAlarms*` to see whether an idle resource is already alarmed. | These metric-read APIs do not support resource-level permissions. **No `PutMetricData`, no alarm create/modify/delete.** | +| `Ec2DescribeReadOnly` | The classic idle-spend inventory: stopped instances still paying for EBS, unattached volumes, unassociated Elastic IPs, idle NAT gateways, orphan snapshots. (`ec2:DescribeImages` was **removed** in cross-review — AMIs are not an idle-spend signal aws-posture acts on.) | `Describe*` is read-only; these list calls don't take resource ARNs. **No `Run*/Start*/Stop*/Terminate*/Modify*/Create*/Delete*`.** | +| `ElbAndRdsDescribeReadOnly` | Idle load balancers (no healthy targets) and idle/oversized RDS are frequent waste. `Describe*` only. | List APIs without resource-level ARNs. **No `rds:Modify*/Delete*/Reboot*`, no ELB mutation.** | +| `LambdaAndStorageInventoryReadOnly` | Inventory functions + buckets to correlate against CloudWatch idle metrics. | **Deliberately excludes `s3:GetObject`** — the role never reads object *data*, only `ListAllMyBuckets` + `GetBucketLocation` (existence/region). **No `lambda:InvokeFunction`, no Lambda mutation.** This is the tightest the inventory can be while still seeing what exists. | + +## What is deliberately NOT here (blast-radius containment) + +- No `iam:*`, `sts:AssumeRole` onward-chaining, `organizations:*`, or `account:*`. +- No `s3:GetObject` / `s3:GetObjectVersion` (no data-plane read of any bucket). +- No `secretsmanager:GetSecretValue` / `ssm:GetParameter*` (no secret read). +- No `kms:Decrypt`, no `logs:GetLogEvents` (no log/data exfil path). +- No write/modify/delete verb in any service. + +A leaked session from this role can **enumerate and price the account, and nothing more** — it +cannot read application data, secrets, or change a single resource. + +## Cross-review NIT answers (2026-06-18) + +- **`ec2:DescribeSnapshots` kept (NIT: is it needed?)** — yes. Orphan EBS snapshots are a common + idle-spend line item (snapshots of long-deleted volumes keep billing); the checker lists them + to flag that waste. It returns only account-owned metadata (we query with `OwnerIds=["self"]`), + no snapshot data. `ec2:DescribeImages` (AMIs) was the over-grant and was **removed**. +- **`s3:ListAllMyBuckets` kept (NIT: data exposure?)** — it returns only bucket *names* you own, + no object data and no bucket contents; `s3:GetBucketLocation` returns only the region. Both are + account-owned inventory queries needed to correlate idle buckets/regions against cost. **No + `s3:GetObject`** anywhere, so there is no data-plane read path. +- **No `logs:*` (NIT: do we need CloudWatch Logs?)** — no. aws-posture reasons over *metrics* + (`cloudwatch:GetMetric*`) and the cost/inventory describe calls; it never needs log *events*. + Omitting `logs:GetLogEvents`/`logs:FilterLogEvents` keeps the role off the log-exfil path. +- **`aws:RequestedRegion` condition (NIT: optional region pin) — SKIPPED, deliberately.** The + reviewer flagged this as optional. It is **NOT applied** because Cost Explorer (`ce:*`) and + `s3:ListAllMyBuckets` are **global-endpoint services** that resolve to us-east-1 with request + contexts where `aws:RequestedRegion` does not reliably equal `us-east-1` — a blanket region + condition risks **DENYing the core cost signal**. Scoping it to a separate statement covering + only the regional `Describe*` calls (ec2/rds/elb/cloudwatch) would add a fourth+ statement for + marginal benefit (the action allow-list already bounds blast radius, and the box only ever runs + in us-east-1). Per the task's guidance, we prefer SKIP over a region pin that could break the + global-service statements. + +## Comparison to the AWS-managed alternatives + +`ReadOnlyAccess` / `ViewOnlyAccess` are far broader (they include `s3:GetObject`, +`dynamodb:GetItem`, `secretsmanager` list, etc.). This custom policy is intentionally a small +fraction of those — only the cost + idle-inventory surface the checker actually queries. diff --git a/iam/aws-posture-trust-policy.json b/iam/aws-posture-trust-policy.json new file mode 100644 index 0000000..ffb812f --- /dev/null +++ b/iam/aws-posture-trust-policy.json @@ -0,0 +1,27 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "RolesAnywhereAssumeFromStepCaLeaf", + "Effect": "Allow", + "Principal": { + "Service": "rolesanywhere.amazonaws.com" + }, + "Action": [ + "sts:AssumeRole", + "sts:TagSession", + "sts:SetSourceIdentity" + ], + "Condition": { + "StringEquals": { + "aws:PrincipalTag/x509Subject/CN": "r720-aws-posture", + "aws:PrincipalTag/x509Issuer/CN": "Sea Haven Internal CA - R720 Roles Anywhere", + "aws:SourceAccount": "328440206208" + }, + "ArnEquals": { + "aws:SourceArn": "arn:aws:rolesanywhere:us-east-1:328440206208:trust-anchor/REPLACE_WITH_TRUST_ANCHOR_ID" + } + } + } + ] +} diff --git a/iam/roles-anywhere-config.json b/iam/roles-anywhere-config.json new file mode 100644 index 0000000..35ba002 --- /dev/null +++ b/iam/roles-anywhere-config.json @@ -0,0 +1,38 @@ +{ + "_comment": "Reviewer-facing config describing the IAM Roles Anywhere trust-anchor + profile to be created. NOT applied — provisioning is gated behind the GPT-4.1 cross-review + Adam. Values prefixed REPLACE_WITH_* are filled in at provisioning time. Region us-east-1, account 328440206208.", + + "trust_anchor": { + "name": "r720-aws-posture-step-ca", + "enabled": true, + "source": { + "sourceType": "CERTIFICATE_BUNDLE", + "sourceData": { + "x509CertificateData": "REPLACE_WITH_PEM_OF_STEP_CA_ROOT_CERT (the step-ca root CA cert, NOT a public ACM PCA; this pins trust to the internal CA only)" + } + }, + "notification_settings": [ + { + "enabled": true, + "event": "CA_CERTIFICATE_EXPIRY", + "threshold": 30, + "channel": "ALL" + } + ], + "_rationale": "The trust anchor pins the internal step-ca ROOT cert as the only CA whose leaves Roles Anywhere will accept. Because the CA is internal and single-purpose, no other identities can mint trusted leaves. CA-expiry notifications are on so the anchor cannot silently go stale." + }, + + "profile": { + "name": "r720-aws-posture-readonly", + "enabled": true, + "roleArns": [ + "arn:aws:iam::328440206208:role/r720-aws-posture-readonly" + ], + "durationSeconds": 3600, + "acceptRoleSessionName": false, + "managedPolicyArns": [], + "sessionPolicy": null, + "_rationale": "Profile binds ONLY the single read-only role. durationSeconds=3600 (1h) caps the lifetime of any vended STS session independent of cert lifetime; combined with a ~24h leaf cert, a compromised leaf yields at most a short read-only window. No extra managed policies; no session-policy widening." + }, + + "_binding_note": "The role's trust policy (aws-posture-trust-policy.json) additionally pins aws:PrincipalTag/x509Subject/CN = 'r720-aws-posture' AND aws:PrincipalTag/x509Issuer/CN, and ArnEquals on aws:SourceArn = this trust anchor. So three independent conditions must all hold for AssumeRole to succeed: (1) the call comes via Roles Anywhere, (2) from THIS trust anchor, (3) presenting a leaf whose subject CN and issuer CN match. Roles Anywhere maps x509 subject/issuer fields into aws:PrincipalTag/x509Subject/* and aws:PrincipalTag/x509Issuer/* session tags, which is what the trust policy keys on." +} diff --git a/iam/step-ca-config-sketch.md b/iam/step-ca-config-sketch.md new file mode 100644 index 0000000..346083d --- /dev/null +++ b/iam/step-ca-config-sketch.md @@ -0,0 +1,145 @@ +# step-ca config sketch — internal CA for aws-posture Roles Anywhere leaf certs + +Design ref: `docs/r720-agent-team-design.md` §6.3 (step-ca + Roles Anywhere, D5) and §7 Phase 3. + +**Not provisioned here.** This is the config + renewal approach for the GPT-4.1 cross-review. +step-ca is the small internal CA on the R720 box (Smallstep `step-ca`) whose **root** cert is +pinned as the Roles Anywhere trust anchor, and which issues a **short-lived leaf** that the box +presents to Roles Anywhere to obtain short-lived read-only STS credentials. **No long-lived AWS +key ever lands on the box** — the leaf self-expires and is auto-renewed by a systemd timer. + +## Trust chain (one CA, one purpose) + +``` +step-ca ROOT (offline-ish, long-lived) + └── step-ca intermediate (the online signer) + └── leaf CN=r720-aws-posture (short-lived, ~24h, auto-renewed) + └── presented to AWS IAM Roles Anywhere trust anchor + └── AssumeRole -> r720-aws-posture-readonly (1h STS session) +``` + +The trust anchor pins the **root** cert (`roles-anywhere-config.json` → `sourceData +.x509CertificateData`). The role trust policy (`aws-posture-trust-policy.json`) additionally +pins the leaf **subject CN** (`r720-aws-posture`) and **issuer CN**, so only this CA's leaf with +this exact CN can assume the role. + +## `ca.json` (sketch — the single-purpose provisioner) + +```jsonc +{ + "root": "/etc/step-ca/certs/root_ca.crt", + "crt": "/etc/step-ca/certs/intermediate_ca.crt", + "key": "/etc/step-ca/secrets/intermediate_ca_key", + "address": "127.0.0.1:8443", // localhost-only; the box is the sole client + "dnsNames": ["localhost", "r720.lan"], + "authority": { + "claims": { + "minTLSCertDuration": "5m", + "maxTLSCertDuration": "24h", // hard cap: leaves are short-lived + "defaultTLSCertDuration": "24h", + "disableRenewal": false + }, + "provisioners": [ + { + "type": "JWK", + "name": "aws-posture", + "key": { "use": "sig", "kty": "EC", "crv": "P-256", "alg": "ES256", "kid": "REPLACE", "x": "REPLACE", "y": "REPLACE" }, + "encryptedKey": "REPLACE_WITH_ENCRYPTED_PROVISIONER_KEY", + "claims": { + "maxTLSCertDuration": "24h", + "defaultTLSCertDuration": "24h" + }, + "options": { + "x509": { + // The provisioner only ever issues this one CN; templating keeps the + // subject/issuer fields the Roles Anywhere trust policy pins. + "templateData": { "CommonName": "r720-aws-posture" } + } + } + } + ] + } +} +``` + +Root CA subject CN: **`Sea Haven Internal CA - R720 Roles Anywhere`** (matches the +`x509Issuer/CN` condition in `aws-posture-trust-policy.json`). + +## Initial bootstrap (one-time, at provisioning) + +```bash +step ca init \ + --name "Sea Haven Internal CA - R720 Roles Anywhere" \ + --dns localhost --dns r720.lan --address 127.0.0.1:8443 \ + --provisioner aws-posture --deployment-type standalone + +# Issue the first leaf the box will present to Roles Anywhere: +step ca certificate "r720-aws-posture" \ + /etc/aws-posture/leaf.crt /etc/aws-posture/leaf.key \ + --not-after 24h --provisioner aws-posture +``` + +`leaf.key` is mode 600, owned by the unattended service user; it never leaves the box. + +## Auto-renewal — systemd timer (the leaf self-expires; the timer keeps it fresh) + +`step-ca` ships `step ca renew`, which no-ops until the cert is within its renewal window. + +`/etc/systemd/system/aws-posture-cert-renew.service`: +```ini +[Unit] +Description=Renew r720-aws-posture Roles Anywhere leaf certificate +After=network-online.target step-ca.service + +[Service] +Type=oneshot +User=aws-posture +# --expires-in: renew only when <8h of life remains; idempotent, safe to run hourly. +ExecStart=/usr/bin/step ca renew --force --expires-in 8h \ + /etc/aws-posture/leaf.crt /etc/aws-posture/leaf.key +# step-ca renew rewrites the cert in place; aws_signing_helper reads it fresh each call, +# so no service reload is needed. +``` + +`/etc/systemd/system/aws-posture-cert-renew.timer`: +```ini +[Unit] +Description=Hourly renewal check for the aws-posture leaf cert + +[Timer] +OnCalendar=hourly +RandomizedDelaySec=300 +Persistent=true # catch up a renewal missed while the box was off + +[Install] +WantedBy=timers.target +``` + +Hourly check + 8h renewal window + 24h cert = the leaf is always fresh and a missed window has +hours of slack. The timer mirrors the existing secrev launchd/systemd discipline. + +## How aws-posture USES the leaf (no AWS key on disk) + +aws-posture invokes AWS's `aws_signing_helper credential-process`, which signs the Roles +Anywhere request with the **leaf** and returns short-lived STS creds on stdout: + +```ini +# ~/.aws/config (on the box) +[profile r720-aws-posture] +credential_process = /usr/local/bin/aws_signing_helper credential-process \ + --certificate /etc/aws-posture/leaf.crt \ + --private-key /etc/aws-posture/leaf.key \ + --trust-anchor-arn arn:aws:rolesanywhere:us-east-1:328440206208:trust-anchor/REPLACE \ + --profile-arn arn:aws:rolesanywhere:us-east-1:328440206208:profile/REPLACE \ + --role-arn arn:aws:iam::328440206208:role/r720-aws-posture-readonly +``` + +The credentials live only in process memory for the 1h session duration; nothing long-lived is +written. This is **strictly stronger than the box's existing long-lived GitHub PAT** (design +§6.3): the AWS identity self-expires and rotates without operator action. + +## Capacity note (design §6.5) + +step-ca on a 4GB / 2 vCPU / 40GB box is negligible (a localhost signer + a tiny DB). Re-check +disk headroom after Phase 1 per §6.5; snapshot the Hyper-V VM before standing this up per +`feedback_ec2_replacement_snapshot`. diff --git a/install-hooks.sh b/install-hooks.sh new file mode 100755 index 0000000..6a10a78 --- /dev/null +++ b/install-hooks.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# install-hooks.sh — install the Sea Haven security-review git hooks + skill assets. +# +# Two modes: +# install-hooks.sh --global Lay the hooks down once for EVERY repo on this machine: +# writes ~/.config/git/hooks/{pre-commit,pre-push}, sets +# git config --global core.hooksPath, and links the skill +# prompt + finding.schema.json into ~/.claude (Path A). +# install-hooks.sh /path/to/repo Per-repo install: copy the hooks into /.git/hooks +# (use when a repo sets its own local core.hooksPath, e.g. +# husky, which would otherwise shadow the global hook). +# install-hooks.sh --help +# +# The hooks run review.sh --scanners-only (fast, deterministic). The full agentic review is the +# on-demand /sh-security-review skill; the nightly VM sweep is the backstop. Idempotent + re-runnable. +set -euo pipefail + +SRC="$(cd "$(dirname "$0")" && pwd)" +GLOBAL_HOOKS="$HOME/.config/git/hooks" +CLAUDE_DIR="$HOME/.claude" + +usage() { grep '^#' "$0" | sed 's/^# \{0,1\}//'; } + +install_one() { # install_one + local src="$1" dest="$2" + cp "$src" "$dest" + chmod +x "$dest" +} + +link_asset() { # link_asset (symlink so the repo stays source of truth) + local src="$1" dest="$2" + mkdir -p "$(dirname "$dest")" + ln -sfn "$src" "$dest" + echo " linked $dest -> $src" +} + +case "${1:-}" in + -h|--help|"") usage; exit 0;; + + --global) + echo "== Installing Sea Haven security-review hooks globally ==" + mkdir -p "$GLOBAL_HOOKS" + + # Warn (don't clobber silently) if a different global hooksPath is already set. + CURRENT="$(git config --global --get core.hooksPath || true)" + if [ -n "$CURRENT" ] && [ "$CURRENT" != "$GLOBAL_HOOKS" ]; then + echo " WARNING: git config --global core.hooksPath is already '$CURRENT'." >&2 + echo " Overwriting it with '$GLOBAL_HOOKS'. Re-point manually if that was intentional." >&2 + fi + + install_one "$SRC/hooks/pre-commit" "$GLOBAL_HOOKS/pre-commit" + install_one "$SRC/hooks/pre-push" "$GLOBAL_HOOKS/pre-push" + git config --global core.hooksPath "$GLOBAL_HOOKS" + echo " installed pre-commit + pre-push -> $GLOBAL_HOOKS" + echo " set git config --global core.hooksPath = $GLOBAL_HOOKS" + + # Path A assets: link the on-demand skill prompt + finding schema into ~/.claude. + link_asset "$SRC/skill/sh-security-review.md" "$CLAUDE_DIR/commands/sh-security-review.md" + link_asset "$SRC/finding.schema.json" "$CLAUDE_DIR/security-review/finding.schema.json" + + echo + echo "Done. Every repo on this machine is now gated by review.sh --scanners-only before push." + echo "Caveats: a repo that sets its OWN local core.hooksPath (e.g. husky) overrides this global hook" + echo " — run 'install-hooks.sh ' to gate it per-repo. Skip a repo with a" + echo " .security-review-skip file at its root; bypass once with 'git push --no-verify'." + ;; + + --*) + echo "unknown option: $1" >&2; usage; exit 2;; + + *) + REPO="$1" + [ -d "$REPO/.git" ] || { echo "not a git repo: $REPO" >&2; exit 1; } + echo "== Installing security-review hooks into $REPO/.git/hooks ==" + for h in pre-commit pre-push; do + DEST="$REPO/.git/hooks/$h" + [ -f "$DEST" ] && echo " warning: existing $h hook at $DEST will be overwritten" >&2 + install_one "$SRC/hooks/$h" "$DEST" + echo " installed $h -> $DEST" + done + echo "note: hooks run deterministic scanners only; full review is /sh-security-review (on demand)." + ;; +esac diff --git a/lib/sweep_substrate.sh b/lib/sweep_substrate.sh new file mode 100644 index 0000000..f8883d8 --- /dev/null +++ b/lib/sweep_substrate.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +# sweep_substrate.sh - shared, sourceable substrate for Sea Haven R720 sweeps. +# +# This module factors the reusable concerns out of nightly_sweep.sh (the LIVE sh-secrev +# Path B nightly sweep) so both secrev and the R720 agent-team (a separate track) can +# reuse one implementation. See docs/r720-agent-team-design.md section 7 Phase 0. +# +# Design contract (IMPORTANT - read before editing): +# These functions are extracted VERBATIM from nightly_sweep.sh. They preserve secrev +# behavior exactly. Bash uses dynamic scoping, so a function sourced here closes over +# the CALLER'S variables by name. Each function below documents which caller globals +# it reads or mutates. Callers MUST provide those globals (the names are part of the +# contract); this keeps the extraction zero-behavior-change versus the old inline copy. +# +# bash, stdlib/coreutils only (jq, curl, git, sed, date). No new dependencies. +# +# Usage: +# source "/lib/sweep_substrate.sh" +# ... then call the functions exactly as the inline versions were called. +# +# Functions (each small + individually testable): +# --- budget ledger --- +# add_spend AMOUNT accumulate agentic spend into TOTAL_SPEND (jq exact-add) +# over_budget true if TOTAL_SPEND >= TOTAL_BUDGET_USD (and ceiling > 0) +# --- Slack ALARM-only reporting (with secret redaction) --- +# redact stdin->stdout: mask AWS/GitHub/Slack/high-entropy secrets +# post_slack_alarm TEXT POST the alarm to SLACK_WEBHOOK_URL, or log-only if unset +# --- discovery (org enumeration via REST + GH_TOKEN, no gh CLI) --- +# discover_repos emit "nameclone_urldefault_branch" per non-archived repo +# --- mirror (clean shallow clone; token never persisted to .git/config) --- +# mirror_repo NAME URL BRANCH mirror one repo into MIRROR_DIR/NAME (0 ok / 1 fail) +# --- round-robin rotation (persistent cycle pointer) --- +# to_epoch DATE UTC date string -> epoch seconds (GNU or BSD date) +# --- canary / testbed gate --- +# canary_confirmed_count JSON count confirmed crit+high findings in a review.sh result JSON + +# --- budget ledger ------------------------------------------------------------ +# Reads/mutates caller globals: TOTAL_SPEND. Reads: TOTAL_BUDGET_USD. +add_spend() { TOTAL_SPEND="$(jq -n --argjson a "$TOTAL_SPEND" --argjson b "${1:-0}" '$a + $b')"; } +over_budget() { jq -n --argjson s "$TOTAL_SPEND" --argjson c "$TOTAL_BUDGET_USD" -e '$c > 0 and $s >= $c' >/dev/null; } + +# --- Secret redaction for the Slack string (defense-in-depth; reports stay on the VM) -- +redact() { + sed -E \ + -e 's/AKIA[0-9A-Z]{16}/AKIA****REDACTED****/g' \ + -e 's/gh[pousr]_[A-Za-z0-9]{20,}/gh*_****REDACTED****/g' \ + -e 's/(xox[baprs]-)[A-Za-z0-9-]{10,}/\1****REDACTED****/g' \ + -e 's/[A-Za-z0-9/+]{40,}/****REDACTED-HIENTROPY****/g' +} + +# --- Slack ALARM-only delivery ------------------------------------------------- +# Posts the (already-redacted, already-composed) alarm text. If SLACK_WEBHOOK_URL is +# unset or curl is missing, logs only; the alarm text remains in the sweep log. +# Reads caller globals: SLACK_WEBHOOK_URL, REPORT_DIR, SWEEP_LOG. Uses log() from caller. +post_slack_alarm() { # alarm_text + local slack_text="$1" + if [ -n "${SLACK_WEBHOOK_URL:-}" ] && command -v curl >/dev/null; then + local payload; payload="$(jq -n --arg t "$slack_text" '{text:$t}')" + if curl -fsS -X POST -H 'Content-Type: application/json' --data "$payload" "$SLACK_WEBHOOK_URL" >/dev/null 2>>"$REPORT_DIR/slack.log"; then + log "Slack alarm posted." + else + log "Slack POST FAILED — see $REPORT_DIR/slack.log. Alarm text is in $SWEEP_LOG." + fi + else + log "SLACK_WEBHOOK_URL unset (or curl missing) — alarm logged to $SWEEP_LOG only." + fi +} + +# --- Discovery: enumerate non-archived org repos via the REST API ------------- +# Emits "nameclone_urldefault_branch" per repo. Returns non-zero on failure. +# Reads caller globals: GH_TOKEN, GH_ORG, REPORT_DIR. +discover_repos() { + [ -n "${GH_TOKEN:-}" ] || { log "GH_TOKEN unset — cannot enumerate org"; return 1; } + local page=1 got body + while :; do + body="$(curl -fsS \ + -H "Authorization: Bearer $GH_TOKEN" \ + -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "https://api.github.com/orgs/$GH_ORG/repos?per_page=100&type=all&page=$page" 2>>"$REPORT_DIR/discover.log")" || return 1 + echo "$body" | jq -e 'type=="array"' >/dev/null 2>&1 || return 1 + got="$(echo "$body" | jq -r '[.[] | select(.archived==false)] | .[] | [.name, .clone_url, .default_branch] | @tsv')" + [ -n "$got" ] && echo "$got" + [ "$(echo "$body" | jq 'length')" -lt 100 ] && break + page=$((page+1)) + done + return 0 +} + +# --- Mirror one repo as a shallow clean clone ------------------------------------------- +# The token is NEVER persisted to .git/config: the fetch path passes the auth URL inline +# (transient, command-args only), and the clone path scrubs origin immediately after. So a +# failed fetch cannot leave GH_TOKEN at rest on disk. (Residual: the token is briefly visible +# in process args to a local `ps`; acceptable on this single-user unattended box.) +# Reads caller globals: MIRROR_DIR, GH_TOKEN. +mirror_repo() { # name clone_url default_branch -> 0 ok / 1 fail + local name="$1" url="$2" branch="$3" + local dir="$MIRROR_DIR/$name" + local auth_url="https://x-access-token:${GH_TOKEN}@${url#https://}" + if [ -d "$dir/.git" ]; then + git -C "$dir" fetch --depth=1 "$auth_url" "$branch" >/dev/null 2>&1 || return 1 + git -C "$dir" reset --hard FETCH_HEAD >/dev/null 2>&1 || return 1 + git -C "$dir" clean -fdq >/dev/null 2>&1 || true + else + git clone --depth=1 --branch "$branch" "$auth_url" "$dir" >/dev/null 2>&1 || return 1 + git -C "$dir" remote set-url origin "$url" >/dev/null 2>&1 || true # clone wrote auth URL → scrub it + fi + return 0 +} + +# --- Rotation helper: portable UTC date-string -> epoch ------------------------ +# Used by the round-robin rotation cycle-age accounting. GNU date (Linux/VM) and BSD +# date (macOS) both handled; unparseable -> 0. +to_epoch() { date -u -d "$1" +%s 2>/dev/null || date -u -j -f '%Y-%m-%d' "$1" +%s 2>/dev/null || echo 0; } + +# --- Canary / testbed gate helper --------------------------------------------- +# Count confirmed crit+high findings in a review.sh result JSON (the anti-complacency +# recall measure). Prints 0 if the file is missing/unreadable. +canary_confirmed_count() { # result_json + local result_json="$1" + if [ -n "$result_json" ] && [ -f "$result_json" ]; then + jq -r '[.findings[]? | select(.status=="confirmed" and (.severity|IN("critical","high")))] | length' "$result_json" 2>/dev/null || echo 0 + else + echo 0 + fi +} diff --git a/nightly_sweep.sh b/nightly_sweep.sh new file mode 100755 index 0000000..526efec --- /dev/null +++ b/nightly_sweep.sh @@ -0,0 +1,362 @@ +#!/usr/bin/env bash +# nightly_sweep.sh — Sea Haven Path B nightly security sweep (R720 / sh-secrev VM). +# +# TWO-TIER, CLEAN-CLONE AUTO-DISCOVERY (no per-repo wiring): +# Discovery: enumerate ALL Sea-Haven-Industries org repos via the GitHub REST API +# (curl + a read-only fine-grained PAT in GH_TOKEN — no gh CLI dependency), then +# mirror each into ~/repo-mirrors as a shallow clean clone (git clone --depth=1, +# default branch from the API). Scanning server-side clones (not developer working +# trees) structurally avoids surfacing local gitignored .env secrets. +# TIER 1 (every repo, every night, $0 Claude): review.sh --scanners-only over every +# mirror — complete deterministic baseline coverage. +# TIER 2 (bounded agentic): the expensive run_headless.py detector+verifier pass runs +# over a deterministic ROUND-ROBIN rotation that fits TOTAL_BUDGET_USD, with a +# persistent cycle pointer so every repo gets a deep pass within MAX_CYCLE_NIGHTS. +# This bounds the draw on the SHARED Max subscription limits (see memory +# reference-claude-subscription-billing): a clean night never scans all repos +# agentically. +# +# Anti-complacency: the canary testbed is ALWAYS scanned agentically first (block + +# recall floor). Reporting is Slack ALARM-ONLY (a clean night posts NOTHING — see +# memory feedback_cloudwatch_alarms). Secret-shaped values are redacted from the Slack +# string; on-disk reports are written mode 600. +# +# Skip a repo: a .security-review-skip file committed at its root, OR an entry in the +# central skip list ($CENTRAL_SKIP_FILE). Repos skipped via their OWN committed marker +# are LOGGED in the report (auditable — a sensitive repo cannot silently self-exclude). +# +# Contract notes: +# - run_headless.py REQUIRES CLAUDE_CODE_OAUTH_TOKEN and pops ANTHROPIC_API_KEY. +# Source ~/secrev.env before invoking (the systemd unit does this via EnvironmentFile). +# - review.sh re-derives the block decision (exit 1 = BLOCK). This script makes NO +# block decision itself; it only reports. +# +# Config (env, all optional except auth): +# GH_TOKEN read-only fine-grained PAT (Contents: read) — REQUIRED for discovery +# GH_ORG org to enumerate (default: Sea-Haven-Industries) +# MIRROR_DIR clean-clone mirror root (default: ~/repo-mirrors) +# CENTRAL_SKIP_FILE one repo name per line, # comments (default: ~/.secrev-skip.txt) +# TARGETS space-separated paths to scan INSTEAD of discovery (manual override) +# TESTBED canary corpus dir (default: ~/security-review-testbed) +# CANARY_FLOOR min confirmed crit+high the canary MUST surface (default: 10) +# TOTAL_BUDGET_USD hard agentic spend ceiling across the night (default: 120 — +# full deep-pass coverage of every repo per night; first-run +# data 2026-06-17 showed $20 covered only canary + 5 repos) +# PER_TARGET_BUDGET_USD passed to run_headless --total-budget-usd (default: 12) +# MAX_CYCLE_NIGHTS alarm if the agentic rotation hasn't covered every repo in this many nights (default: 4) +# MAX_AGENTIC_PER_NIGHT cap on repos given the deep agentic pass per night, for wall-clock bounding +# (default: 0 = unlimited, bounded only by TOTAL_BUDGET_USD) +# REPORT_ROOT base dir for logs+JSON (default: ~/sweep-reports) +# SLACK_WEBHOOK_URL incoming-webhook URL; if unset, alarms are logged only +# ENABLE_XMODEL_HOOK 1 to run the cross-family critical tiebreak (default: 0) +# ORCHESTRATOR_DIR orchestrator repo root (default: ~/orchestrator) +# VENV_PY python in the SDK venv (default: ~/orchestrator/.venv/bin/python) +# +# Exit: 0 = sweep completed (whether or not it alarmed); 2 = setup/usage error. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH" + +log() { echo "[nightly_sweep] $*" >&2; } +die() { echo "[nightly_sweep] FATAL: $*" >&2; exit 2; } + +# --- Shared substrate (discovery / mirror / budget / rotation / Slack / canary) - +# Factored out so secrev and the R720 agent-team reuse one implementation, WITHOUT +# changing any secrev behavior. The functions close over this script's globals by name +# (bash dynamic scoping); see lib/sweep_substrate.sh for the read/mutate contract. +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib/sweep_substrate.sh +. "$HERE/lib/sweep_substrate.sh" + +# --- Config + defaults -------------------------------------------------------- +ORCHESTRATOR_DIR="${ORCHESTRATOR_DIR:-$HOME/orchestrator}" +VENV_PY="${VENV_PY:-$ORCHESTRATOR_DIR/.venv/bin/python}" +RUN_HEADLESS="$HERE/run_headless.py" +REVIEW_SH="$HERE/review.sh" +GH_ORG="${GH_ORG:-Sea-Haven-Industries}" +MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}" +CENTRAL_SKIP_FILE="${CENTRAL_SKIP_FILE:-$HOME/.secrev-skip.txt}" +TESTBED="${TESTBED:-$HOME/security-review-testbed}" +CANARY_FLOOR="${CANARY_FLOOR:-14}" +TOTAL_BUDGET_USD="${TOTAL_BUDGET_USD:-120}" +PER_TARGET_BUDGET_USD="${PER_TARGET_BUDGET_USD:-12}" +MAX_CYCLE_NIGHTS="${MAX_CYCLE_NIGHTS:-6}" +MAX_AGENTIC_PER_NIGHT="${MAX_AGENTIC_PER_NIGHT:-0}" +REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports}" +ENABLE_XMODEL_HOOK="${ENABLE_XMODEL_HOOK:-0}" + +command -v jq >/dev/null || die "jq is required" +command -v curl >/dev/null || die "curl is required for org discovery" +command -v git >/dev/null || die "git is required" +[ -x "$VENV_PY" ] || die "venv python not found/executable: $VENV_PY" +[ -f "$RUN_HEADLESS" ] || die "run_headless.py not found: $RUN_HEADLESS" +[ -x "$REVIEW_SH" ] || die "review.sh not found/executable: $REVIEW_SH" +[ -n "${CLAUDE_CODE_OAUTH_TOKEN:-}" ] || die "CLAUDE_CODE_OAUTH_TOKEN not set (source ~/secrev.env)" + +UTC_DATE="$(date -u +%Y-%m-%d)" +UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +REPORT_DIR="$REPORT_ROOT/$UTC_DATE" +mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true +ROTATION_STATE="$REPORT_ROOT/.rotation-state.json" +SWEEP_LOG="$REPORT_DIR/sweep.log" +exec > >(tee -a "$SWEEP_LOG") 2>&1 +umask 077 # on-disk reports/logs are not world-readable + +log "=== nightly sweep $UTC_STAMP (two-tier auto-discovery) ===" +log "org=$GH_ORG mirror=$MIRROR_DIR report=$REPORT_DIR total-budget=\$$TOTAL_BUDGET_USD canary-floor=$CANARY_FLOOR" + +# --- Aggregate state ---------------------------------------------------------- +TOTAL_SPEND="0"; BUDGET_HIT=0 +declare -a ALARM_LINES=(); declare -a XMODEL_LINES=(); declare -a MARKER_SKIPS=() +# add_spend / over_budget (budget ledger), redact (Slack secret redaction), +# discover_repos (org enumeration), mirror_repo (clean shallow clone): provided by +# lib/sweep_substrate.sh, sourced above. They close over the globals defined here +# (TOTAL_SPEND, TOTAL_BUDGET_USD, GH_TOKEN, GH_ORG, MIRROR_DIR, REPORT_DIR). + +# --- Skip resolution: "" = scan, else reason ("marker"|"central") -------------- +declare -a CENTRAL_SKIP=() +if [ -f "$CENTRAL_SKIP_FILE" ]; then + while IFS= read -r line; do line="${line%%#*}"; line="$(echo "$line" | xargs || true)" + [ -n "$line" ] && CENTRAL_SKIP+=( "$line" ); done < "$CENTRAL_SKIP_FILE" +fi +skip_reason() { # name dir + local name="$1" dir="$2" + [ -f "$dir/.security-review-skip" ] && { echo "marker"; return; } + for s in ${CENTRAL_SKIP[@]+"${CENTRAL_SKIP[@]}"}; do [ "$s" = "$name" ] && { echo "central"; return; }; done + echo "" +} + +# --- xmodel cross-family critical tiebreak (GUARDED, never fails the sweep) ---- +xmodel_check_criticals() { + local label="$1" result_json="$2" + [ "$ENABLE_XMODEL_HOOK" = "1" ] || return 0 + [ -n "${OPENAI_API_KEY:-}" ] || { log " xmodel hook: OPENAI_API_KEY unset — skipping"; return 0; } + "$VENV_PY" -c 'import langchain_openai' >/dev/null 2>&1 || { log " xmodel hook: deps missing — skipping"; return 0; } + local crits n; crits="$(jq -c '[.findings[]? | select(.status=="confirmed" and .severity=="critical")]' "$result_json" 2>/dev/null || echo '[]')" + n="$(echo "$crits" | jq 'length')"; [ "${n:-0}" -gt 0 ] || return 0 + log " xmodel hook: re-checking $n confirmed critical(s) for $label" + local i=0 + while [ "$i" -lt "$n" ]; do + local summary; summary="$(echo "$crits" | jq -r --argjson i "$i" '.[$i] | "\(.cwe // "n/a") \(.file):\(.line // 0) — \(.title // .id) :: \(.data_flow // "")"')" + local verdict + if verdict="$(cd "$ORCHESTRATOR_DIR" && "$VENV_PY" run.py "Independently assess whether this is a real exploitable vulnerability (yes/no) and why: $summary" 2>>"$REPORT_DIR/xmodel.log")"; then + if echo "$verdict" | grep -qiE '(^|[^a-z])no([^a-z]|$)|not (a |an )?(real |exploitable )?vuln'; then + XMODEL_LINES+=( "DISAGREEMENT on $label critical: $summary (cross_reviewer says NOT a vuln)" ) + fi + else log " xmodel hook: run.py failed for a critical (logged) — continuing"; fi + i=$((i+1)) + done +} + +# --- TIER 1: deterministic scanners over a target dir ------------------------- +# Sets T1_BLOCK/T1_CRIT/T1_HIGH. review.sh exit 0 pass / 1 BLOCK / 2 setup. +T1_BLOCK=0; T1_CRIT=0; T1_HIGH=0 +scan_scanners() { # target slug + local target="$1" slug="$2" + local result_json="$REPORT_DIR/${slug}.scanners.json" + T1_BLOCK=0; T1_CRIT=0; T1_HIGH=0 + local sup=() + [ -f "$target/.security-review/suppressions.json" ] && sup=(--suppressions "$target/.security-review/suppressions.json") + set +e + "$REVIEW_SH" --scanners-only ${sup[@]+"${sup[@]}"} --json-out "$result_json" "$target" >"$REPORT_DIR/${slug}.scanners.log" 2>&1 + local rc=$? + set -e + [ "$rc" -eq 2 ] && { ALARM_LINES+=( "*$slug*: review.sh scanner setup error. See \`$REPORT_DIR/${slug}.scanners.log\`." ); return; } + T1_CRIT="$(jq -r '(.summary.confirmed_critical // 0)' "$result_json" 2>/dev/null || echo 0)" + T1_HIGH="$(jq -r '(.summary.confirmed_high // 0)' "$result_json" 2>/dev/null || echo 0)" + [ "$rc" -eq 1 ] && T1_BLOCK=1 + return 0 # MUST return 0: results go via globals; a falsey last cmd would trip set -e in the caller +} + +# --- TIER 2: agentic run_headless + full review.sh over a target dir ---------- +# Sets LAST_BLOCK/LAST_CRIT/LAST_HIGH/LAST_REASON/LAST_RESULT_JSON/LAST_ERRORS. +LAST_BLOCK=0; LAST_CRIT=0; LAST_HIGH=0; LAST_REASON=""; LAST_RESULT_JSON=""; LAST_ERRORS=0 +scan_agentic() { # target slug + local target="$1" slug="$2" + LAST_BLOCK=0; LAST_CRIT=0; LAST_HIGH=0; LAST_REASON=""; LAST_RESULT_JSON=""; LAST_ERRORS=0 + [ -d "$target" ] || { ALARM_LINES+=( "Target *$slug* ($target) missing — could not scan." ); LAST_ERRORS=1; return; } + local agent_json="$REPORT_DIR/${slug}.agent.json" result_json="$REPORT_DIR/${slug}.result.json" runner_log="$REPORT_DIR/${slug}.runner.log" + LAST_RESULT_JSON="$result_json" + log " [$slug] run_headless.py (per-target budget \$$PER_TARGET_BUDGET_USD)" + if ! "$VENV_PY" "$RUN_HEADLESS" "$target" --out "$agent_json" --total-budget-usd "$PER_TARGET_BUDGET_USD" >>"$runner_log" 2>&1; then + ALARM_LINES+=( "*$slug*: run_headless.py failed (setup error). See \`$runner_log\`." ); LAST_ERRORS=1; return + fi + [ -f "$agent_json" ] || { ALARM_LINES+=( "*$slug*: run_headless produced no JSON." ); LAST_ERRORS=1; return; } + local spend errs; spend="$(jq -r '(._meta.spend_usd // 0)' "$agent_json")"; errs="$(jq -r '(._meta.errors // []) | length' "$agent_json")" + add_spend "$spend"; LAST_ERRORS="$errs" + log " [$slug] spend \$$spend, runner errors $errs, total \$$TOTAL_SPEND" + [ "${errs:-0}" -gt 0 ] && ALARM_LINES+=( "*$slug*: run_headless reported $errs error(s): $(jq -r '(._meta.errors // []) | join("; ")' "$agent_json")" ) + local sup=() + [ -f "$target/.security-review/suppressions.json" ] && sup=(--suppressions "$target/.security-review/suppressions.json") + set +e + "$REVIEW_SH" --agent-findings "$agent_json" ${sup[@]+"${sup[@]}"} --json-out "$result_json" "$target" >"$REPORT_DIR/${slug}.review.log" 2>&1 + local rc=$? + set -e + [ "$rc" -eq 2 ] && { ALARM_LINES+=( "*$slug*: review.sh setup error. See \`$REPORT_DIR/${slug}.review.log\`." ); LAST_ERRORS=$((LAST_ERRORS+1)); return; } + LAST_CRIT="$(jq -r '(.summary.confirmed_critical // 0)' "$result_json" 2>/dev/null || echo 0)" + LAST_HIGH="$(jq -r '(.summary.confirmed_high // 0)' "$result_json" 2>/dev/null || echo 0)" + if [ "$rc" -eq 1 ]; then LAST_BLOCK=1; LAST_REASON="confirmed crit=$LAST_CRIT high=$LAST_HIGH"; log " [$slug] BLOCK ($LAST_REASON)" + else log " [$slug] PASS (crit=$LAST_CRIT high=$LAST_HIGH)"; fi +} + +# ============================== 1) CANARY ==================================== +CANARY_OK=1 +if [ -d "$TESTBED" ]; then + log "--- canary (anti-complacency): $TESTBED ---" + scan_agentic "$TESTBED" "canary" + CANARY_CONFIRMED="$(canary_confirmed_count "$LAST_RESULT_JSON")" + log "canary: block=$LAST_BLOCK confirmed(crit+high)=$CANARY_CONFIRMED (floor=$CANARY_FLOOR)" + if [ "$LAST_BLOCK" -ne 1 ]; then + CANARY_OK=0; ALARM_LINES+=( "*COMPLACENCY ALARM*: canary testbed did NOT block. Result: \`$LAST_RESULT_JSON\`" ) + elif [ "${CANARY_CONFIRMED:-0}" -lt "$CANARY_FLOOR" ]; then + CANARY_OK=0; ALARM_LINES+=( "*COMPLACENCY ALARM*: canary recall $CANARY_CONFIRMED < floor $CANARY_FLOOR. Result: \`$LAST_RESULT_JSON\`" ) + fi + xmodel_check_criticals "canary" "$LAST_RESULT_JSON" +else + CANARY_OK=0; ALARM_LINES+=( "*COMPLACENCY ALARM*: canary testbed missing at $TESTBED." ) +fi + +# ============================== 2) DISCOVER + MIRROR ========================= +declare -a REPO_NAMES=() # scan order (discovery order) +declare -A REPO_DIR=() +if [ -n "${TARGETS:-}" ]; then + # Manual override: scan explicit paths, no discovery/cloning. + # shellcheck disable=SC2206 + arr=( $TARGETS ) + for p in "${arr[@]}"; do + p="${p/#\~/$HOME}"; nm="$(basename "$p")" + REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p" + done + log "manual TARGETS override: ${REPO_NAMES[*]}" +else + mkdir -p "$MIRROR_DIR" + DISCOVERED="$REPORT_DIR/discovered.tsv" + if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then + NREPO="$(wc -l < "$DISCOVERED" | tr -d ' ')" + log "discovered $NREPO non-archived repo(s) in $GH_ORG" + while IFS=$'\t' read -r name url branch; do + [ -n "$name" ] || continue + if mirror_repo "$name" "$url" "$branch"; then + REPO_NAMES+=( "$name" ); REPO_DIR["$name"]="$MIRROR_DIR/$name" + else + log " mirror FAILED: $name"; ALARM_LINES+=( "*$name*: clone/pull failed — not scanned this night. See \`$REPORT_DIR/discover.log\`." ) + fi + done < "$DISCOVERED" + log "mirrored ${#REPO_NAMES[@]} repo(s) into $MIRROR_DIR" + else + ALARM_LINES+=( "*DISCOVERY ALARM*: org enumeration failed (GH_TOKEN missing/invalid or API error). Falling back to existing mirrors; coverage may be stale. See \`$REPORT_DIR/discover.log\`." ) + log "discovery failed — falling back to existing mirrors in $MIRROR_DIR" + if [ -d "$MIRROR_DIR" ]; then + for d in "$MIRROR_DIR"/*/; do [ -d "$d/.git" ] || continue; nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}"; done + fi + fi +fi + +# Resolve skips up front (so both tiers honor them and marker-skips are auditable). +declare -a SCANNABLE=() +for nm in ${REPO_NAMES[@]+"${REPO_NAMES[@]}"}; do + reason="$(skip_reason "$nm" "${REPO_DIR[$nm]}")" + if [ "$reason" = "marker" ]; then MARKER_SKIPS+=( "$nm" ); log " skip $nm (repo-committed .security-review-skip)" + elif [ "$reason" = "central" ]; then log " skip $nm (central skip list)" + else SCANNABLE+=( "$nm" ); fi +done +if [ "${#MARKER_SKIPS[@]}" -gt 0 ]; then + ALARM_LINES+=( "*self-excluded repos* (committed .security-review-skip, FYI/audit): ${MARKER_SKIPS[*]}" ) +fi +log "scannable repos: ${#SCANNABLE[@]} (skipped: $(( ${#REPO_NAMES[@]} - ${#SCANNABLE[@]} )))" + +# ============================== 3) TIER 1: scanners over ALL ================== +declare -a BLOCKED_T1=() +for nm in ${SCANNABLE[@]+"${SCANNABLE[@]}"}; do + scan_scanners "${REPO_DIR[$nm]}" "scan-$nm" + if [ "$T1_BLOCK" -eq 1 ]; then + BLOCKED_T1+=( "$nm" ) + ALARM_LINES+=( "*$nm* TIER1/scanners BLOCK: crit=$T1_CRIT high=$T1_HIGH. Result: \`$REPORT_DIR/scan-$nm.scanners.json\`" ) + fi +done +log "tier1 complete: ${#SCANNABLE[@]} scanned, ${#BLOCKED_T1[@]} blocked" + +# ============================== 4) TIER 2: agentic rotation =================== +# Persistent cycle state: {cycle_start, scanned:[names]}. Reset the cycle once every +# scannable repo has had a deep pass; alarm if a cycle runs longer than MAX_CYCLE_NIGHTS. +[ -f "$ROTATION_STATE" ] || echo "{\"cycle_start\":\"$UTC_DATE\",\"scanned\":[]}" > "$ROTATION_STATE" +SCANNED_JSON="$(jq -c '.scanned // []' "$ROTATION_STATE" 2>/dev/null || echo '[]')" +CYCLE_START="$(jq -r '.cycle_start // empty' "$ROTATION_STATE" 2>/dev/null || echo "$UTC_DATE")" +[ -n "$CYCLE_START" ] || CYCLE_START="$UTC_DATE" +if [ "${#SCANNABLE[@]}" -gt 0 ]; then + SCANNABLE_JSON="$(printf '%s\n' "${SCANNABLE[@]}" | jq -R . | jq -cs .)" +else + SCANNABLE_JSON="[]" +fi +# If every scannable repo is already in scanned[], the cycle is complete -> start fresh. +if jq -e -n --argjson sc "$SCANNED_JSON" --argjson all "$SCANNABLE_JSON" '($all - $sc) | length == 0' >/dev/null 2>&1 \ + && [ "$(echo "$SCANNABLE_JSON" | jq 'length')" -gt 0 ]; then + log "agentic rotation: cycle complete ($CYCLE_START) — starting a new cycle" + SCANNED_JSON="[]"; CYCLE_START="$UTC_DATE" +fi +# This night's agentic candidates = scannable repos not yet scanned this cycle, discovery order. +PENDING_JSON="$(jq -c -n --argjson all "$SCANNABLE_JSON" --argjson sc "$SCANNED_JSON" '$all - $sc')" +declare -a BLOCKED_T2=(); AGENTIC_DONE=0 +if over_budget; then + BUDGET_HIT=1; ALARM_LINES+=( "*BUDGET ALARM*: ceiling \$$TOTAL_BUDGET_USD hit after canary (\$$TOTAL_SPEND). No agentic rotation this night." ) +else + while read -r nm; do + [ -n "$nm" ] || continue + if over_budget; then BUDGET_HIT=1; log "budget ceiling hit (\$$TOTAL_SPEND) — pausing rotation"; break; fi + if [ "$MAX_AGENTIC_PER_NIGHT" -gt 0 ] && [ "$AGENTIC_DONE" -ge "$MAX_AGENTIC_PER_NIGHT" ]; then + log "per-night agentic cap ($MAX_AGENTIC_PER_NIGHT) reached — pausing rotation"; break; fi + log "--- agentic: $nm ---" + scan_agentic "${REPO_DIR[$nm]}" "scan-$nm" + SCANNED_JSON="$(echo "$SCANNED_JSON" | jq -c --arg n "$nm" '. + [$n] | unique')" + AGENTIC_DONE=$((AGENTIC_DONE+1)) + if [ "$LAST_BLOCK" -eq 1 ]; then + BLOCKED_T2+=( "$nm" ) + ALARM_LINES+=( "*$nm* TIER2/agentic BLOCK: $LAST_REASON. Result: \`$LAST_RESULT_JSON\`" ) + xmodel_check_criticals "$nm" "$LAST_RESULT_JSON" + fi + done < <(echo "$PENDING_JSON" | jq -r '.[]') +fi +# Persist rotation state. +jq -n --arg cs "$CYCLE_START" --argjson sc "$SCANNED_JSON" '{cycle_start:$cs, scanned:$sc}' > "$ROTATION_STATE" +# Coverage accounting + lag alarm. +REMAINING="$(jq -n --argjson all "$SCANNABLE_JSON" --argjson sc "$SCANNED_JSON" '($all - $sc) | length')" +CYCLE_AGE=$(( ( $(to_epoch "$UTC_DATE") - $(to_epoch "$CYCLE_START") ) / 86400 )) +log "agentic rotation: scanned $AGENTIC_DONE this night, $REMAINING still pending in cycle (started $CYCLE_START, age ${CYCLE_AGE}d)" +if [ "$REMAINING" -gt 0 ] && [ "$CYCLE_AGE" -ge "$MAX_CYCLE_NIGHTS" ]; then + ALARM_LINES+=( "*COVERAGE ALARM*: agentic rotation behind — $REMAINING repo(s) not deep-scanned in ${CYCLE_AGE}d (cycle since $CYCLE_START, max $MAX_CYCLE_NIGHTS). Raise budget or check for failures." ) +fi + +# Fold xmodel disagreements into the alarm set. +for x in ${XMODEL_LINES[@]+"${XMODEL_LINES[@]}"}; do ALARM_LINES+=( "$x" ); done + +# ============================== 5) ALARM-ONLY REPORT ========================= +ALARM=0 +[ "${#BLOCKED_T1[@]}" -gt 0 ] && ALARM=1 +[ "${#BLOCKED_T2[@]}" -gt 0 ] && ALARM=1 +[ "$CANARY_OK" -ne 1 ] && ALARM=1 +[ "$BUDGET_HIT" -eq 1 ] && ALARM=1 +[ "${#ALARM_LINES[@]}" -gt 0 ] && ALARM=1 + +SUMMARY_LINE="sweep $UTC_STAMP: scannable=${#SCANNABLE[@]} tier1_blocked=${#BLOCKED_T1[@]} tier2_scanned=$AGENTIC_DONE tier2_blocked=${#BLOCKED_T2[@]} canary_ok=$CANARY_OK spend=\$$TOTAL_SPEND/\$$TOTAL_BUDGET_USD alarm=$ALARM report=$REPORT_DIR" +echo "$SUMMARY_LINE" + +if [ "$ALARM" -ne 1 ]; then + log "clean night — no alarm conditions. Posting NOTHING to Slack (ALARM-only policy)." + exit 0 +fi + +ALARM_BODY="$(printf '%s\n' ${ALARM_LINES[@]+"${ALARM_LINES[@]}"} | sed 's/^/• /')" +SLACK_TEXT=":rotating_light: *Sea Haven nightly security sweep — ALARM* ($UTC_STAMP) +$ALARM_BODY + +Coverage: tier1 scanners ${#SCANNABLE[@]} repos · tier2 agentic $AGENTIC_DONE this night ($REMAINING pending) · canary_ok=$CANARY_OK +Spend: \$$TOTAL_SPEND (ceiling \$$TOTAL_BUDGET_USD) +Reports + JSON: \`$REPORT_DIR\` (on sh-secrev VM)" +SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)" + +log "ALARM conditions present — composing Slack post" +echo "$SLACK_TEXT" >&2 + +post_slack_alarm "$SLACK_TEXT" + +# An alarm is a reportable condition, not a script crash. Exit 0 so systemd shows success. +exit 0 diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..aa3d8b7 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,5 @@ +# Path B headless agentic runner (run_headless.py). Scanners (semgrep, gitleaks, +# checkov, cfn-lint, pip-audit) and npm are external binaries, installed separately +# (see README). The optional cross-model hook (ENABLE_XMODEL_HOOK=1) additionally +# needs langchain-openai, intentionally not pinned here since it is off by default. +claude-agent-sdk diff --git a/review.sh b/review.sh new file mode 100755 index 0000000..73d294a --- /dev/null +++ b/review.sh @@ -0,0 +1,253 @@ +#!/usr/bin/env bash +# review.sh — Sea Haven security-review gate. Trigger-agnostic (pre-commit / pre-push / on-demand / CI). +# Pure code: merges deterministic-scanner findings + agent findings, applies suppressions, +# and decides block. NO agent makes the merge/block decision — this script does, so no agent +# can shrug off a confirmed critical. See memory project-security-review-agent. +# +# Usage: +# review.sh [--scope "src infra web"] [--agent-findings FILE] [--suppressions FILE] +# [--json-out FILE] [--scanners-only] [TARGET_DIR] +# +# Exit codes: 0 = pass, 1 = BLOCK (unsuppressed confirmed critical/high), 2 = usage/setup error. +set -euo pipefail +export PATH="$HOME/.local/bin:/opt/homebrew/bin:$PATH" # find pipx/brew-installed scanners regardless of caller env + +TARGET="."; SCOPE=""; AGENT_FINDINGS=""; SUPPRESSIONS=""; JSON_OUT=""; SCANNERS_ONLY=0 +while [ $# -gt 0 ]; do + case "$1" in + --scope) SCOPE="$2"; shift 2;; + --agent-findings) AGENT_FINDINGS="$2"; shift 2;; + --suppressions) SUPPRESSIONS="$2"; shift 2;; + --json-out) JSON_OUT="$2"; shift 2;; + --scanners-only) SCANNERS_ONLY=1; shift;; + -h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0;; + *) TARGET="$1"; shift;; + esac +done +command -v jq >/dev/null || { echo "review.sh: jq is required" >&2; exit 2; } +[ -d "$TARGET" ] || { echo "review.sh: target dir not found: $TARGET" >&2; exit 2; } +TARGET="$(cd "$TARGET" && pwd)" + +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +ALL="$TMP/all.json"; echo '[]' > "$ALL" +add() { jq --argjson add "$1" '. + $add' "$ALL" > "$ALL.t" && mv "$ALL.t" "$ALL"; } + +# Map a scope list into find paths under TARGET (default: whole target). +scope_paths() { + if [ -n "$SCOPE" ]; then for d in $SCOPE; do [ -e "$TARGET/$d" ] && echo "$TARGET/$d"; done + else echo "$TARGET"; fi +} + +echo "== Deterministic scanners ==" >&2 +note_missing() { echo " [MISSING] $1 — not run. Install: $2" >&2; } + +# --- cfn-lint (installed): lint SAM/CFN templates. Normalize to findings. --- +if command -v cfn-lint >/dev/null; then + # Prune generated/vendored trees (cdk.out, node_modules, …): scanning synthesized output is + # wrong and, on CDK repos, explodes the arg list / stalls the scanners. + # `find -print0 | xargs -0 grep -lE` (was `… | xargs -I{} sh -c 'grep -l "{}"'`): the grep stage + # is the one that overflows. `xargs -I{}` packs every matched path — long, absolute, deep-worktree + # paths on this monorepo — into one assembled command and dies with "command line cannot be + # assembled, too long", emitting zero findings and blocking the push. NUL-delimited `xargs -0 grep` + # splits across invocations transparently (batches by ARG_MAX, never overflows), is safe for paths + # with spaces/newlines, and `grep -l` reports the same matching files as the old per-file grep. + # The first `xargs -I{} find {}` keeps the start path first (find needs it before the expression) + # and is bounded by the scope-path count, so it is not an overflow risk. `--no-run-if-empty` is + # GNU-only, so the trailing `|| true` absorbs grep's exit 1 on no-match / no-files, matching the + # old per-file `… || true` so TPLS is "list of templates, or empty" and never fails the gate. + TPLS="$(scope_paths | xargs -I{} find {} \( -type d \( -name cdk.out -o -name node_modules -o -name .git -o -name .claude -o -name .aws-sam -o -name .venv -o -name venv -o -name dist -o -name build \) -prune \) -o \( -type f \( -name '*.yaml' -o -name '*.yml' \) -print0 \) 2>/dev/null \ + | xargs -0 grep -lE "AWSTemplateFormatVersion|Transform: *AWS::Serverless" 2>/dev/null || true)" + if [ -n "$TPLS" ]; then + # shellcheck disable=SC2086 + RAW="$(cfn-lint -f json $TPLS 2>/dev/null || true)" + if [ -n "$RAW" ] && echo "$RAW" | jq -e 'type=="array"' >/dev/null 2>&1; then + NORM="$(echo "$RAW" | jq '[.[] | { + id: ("cfnlint-" + (.Rule.Id // "X") + "-" + ((.Location.Start.LineNumber // 0)|tostring)), + title: (.Rule.Id + ": " + (.Message // .Rule.Description // "")), + severity: (if (.Level=="Error") then "high" elif (.Level=="Warning") then "medium" else "low" end), + cwe: "n/a", file: (.Filename // ""), line: (.Location.Start.LineNumber // null), + category: "iac-iam", source: "cfn-lint", status: "confirmed", + data_flow: "cfn-lint rule violation", proof: {input: "deploy template", outcome: (.Message // "")} + }]')" + add "$NORM"; echo " [cfn-lint] $(echo "$NORM" | jq length) finding(s)" >&2 + else echo " [cfn-lint] 0 findings" >&2; fi + else echo " [cfn-lint] no CFN/SAM templates in scope" >&2; fi +else note_missing cfn-lint "pip install cfn-lint"; fi + +SCOPE_PATHS="$(scope_paths)" + +# --- semgrep (SAST: injection/authz/xss/secrets) --- +if command -v semgrep >/dev/null; then + # shellcheck disable=SC2086 + if SG="$(semgrep --config p/security-audit --config p/secrets --config p/javascript --json --metrics=off --exclude cdk.out --exclude node_modules --exclude .claude --exclude .venv --exclude venv --exclude .aws-sam --exclude dist --exclude build $SCOPE_PATHS 2>/dev/null)"; then + NORM="$(echo "$SG" | jq '[.results[] | { + id: ("semgrep-" + (.check_id|split(".")|last) + "-" + (.start.line|tostring)), + title: ((.check_id|split(".")|last) + ": " + ((.extra.message // "")[0:120])), + severity: (.extra.severity | if .=="ERROR" then "high" elif .=="WARNING" then "medium" else "low" end), + cwe: ((.extra.metadata.cwe // []) | if length>0 then (.[0]|split(":")[0]) else "n/a" end), + file: .path, line: .start.line, category: "other", source: "semgrep", status: "confirmed", + data_flow: "semgrep rule match", proof: {input: "see rule", outcome: (.extra.message // "")}}]')" + add "$NORM"; echo " [semgrep] $(echo "$NORM" | jq length) finding(s)" >&2 + else echo " [semgrep] run failed" >&2; fi +else note_missing semgrep "brew install semgrep"; fi + +# --- gitleaks (hardcoded secrets) — git-mode respects .gitignore (skips gitignored .env etc.) --- +if command -v gitleaks >/dev/null; then + GLALL="$TMP/gl.json"; echo '[]' > "$GLALL" + if git -C "$TARGET" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + # Scan committed content at the repo root; gitignored files (e.g. a local .env with real + # keys) are excluded by design, so the gate never false-blocks on them. Same JSON schema. + gitleaks git "$TARGET" --report-format json --report-path "$TMP/gl1.json" >/dev/null 2>&1 || true + if [ -s "$TMP/gl1.json" ]; then + jq -s '.[0]+(.[1] // [])' "$GLALL" "$TMP/gl1.json" > "$GLALL.t" && mv "$GLALL.t" "$GLALL"; rm -f "$TMP/gl1.json" + fi + else + for p in $SCOPE_PATHS; do + gitleaks dir "$p" --report-format json --report-path "$TMP/gl1.json" >/dev/null 2>&1 || true + if [ -s "$TMP/gl1.json" ]; then + jq -s '.[0]+(.[1] // [])' "$GLALL" "$TMP/gl1.json" > "$GLALL.t" && mv "$GLALL.t" "$GLALL"; rm -f "$TMP/gl1.json" + fi + done + fi + NORM="$(jq '[.[] | { + id: ("gitleaks-" + .RuleID + "-" + (.StartLine|tostring)), + title: ("secret: " + .Description), severity: "high", cwe: "CWE-798", + file: .File, line: .StartLine, category: "secrets-crypto", source: "gitleaks", status: "confirmed", + data_flow: "hardcoded secret in source", proof: {input: "read source", outcome: .Description}}]' "$GLALL")" + add "$NORM"; echo " [gitleaks] $(echo "$NORM" | jq length) finding(s)" >&2 +else note_missing gitleaks "brew install gitleaks"; fi + +# --- checkov (IaC/IAM misconfig) --- +if command -v checkov >/dev/null; then + CKALL="$TMP/ck.json"; echo '[]' > "$CKALL" + for p in $SCOPE_PATHS; do + # --skip-path is a regex over the file path. Skip only the cdk.out asset./ + # dependency bundles (the stall cause) + vendored/generated dirs — but KEEP + # scanning cdk.out/.template.json, which is the real deploy artifact + # (dropping it would silence genuine S3/IAM IaC findings). asset is anchored to + # cdk.out so a source file literally named asset.* is not also excluded. + if [ -d "$p" ]; then RAW="$(checkov -d "$p" --skip-path 'cdk\.out/asset\.' --skip-path node_modules --skip-path '\.claude' --skip-path '\.venv' --skip-path venv --skip-path '\.aws-sam' --skip-path dist --skip-path build -o json --compact --quiet 2>/dev/null || true)" + else RAW="$(checkov -f "$p" -o json --compact --quiet 2>/dev/null || true)"; fi + [ -z "$RAW" ] && continue + FC="$(echo "$RAW" | jq '[ (if type=="array" then .[] else . end).results.failed_checks // [] ] | add // []' 2>/dev/null || echo '[]')" + jq -s '.[0]+.[1]' "$CKALL" <(echo "$FC") > "$CKALL.t" && mv "$CKALL.t" "$CKALL" + done + # High-signal checkov checks (exposure/access/wildcard) -> high; everything else -> low (informational). + # checkov OSS rarely populates .severity, so without this every best-practice nit reads as medium and drowns signal. + CKHI='["CKV_AWS_53","CKV_AWS_54","CKV_AWS_55","CKV_AWS_56","CKV_AWS_57","CKV_AWS_20","CKV_AWS_24","CKV_AWS_25","CKV_AWS_260","CKV_AWS_1","CKV_AWS_40","CKV_AWS_49","CKV_AWS_62","CKV_AWS_70","CKV_AWS_107","CKV_AWS_108","CKV_AWS_109","CKV_AWS_110","CKV_AWS_111"]' + NORM="$(jq --argjson hi "$CKHI" '[.[] | { + id: ("checkov-" + .check_id + "-" + ((.file_line_range[0]) // 0 | tostring)), + title: (.check_id + ": " + (.check_name // "")), + severity: (if .severity != null then (.severity|ascii_downcase) + elif (.check_id as $c | $hi | index($c)) then "high" else "low" end), + cwe: "n/a", file: (.file_abs_path // .file_path), line: (.file_line_range[0] // null), + category: "iac-iam", source: "checkov", status: "confirmed", + data_flow: "checkov policy violation", proof: {input: "deploy template", outcome: (.check_name // "")}}]' "$CKALL")" + add "$NORM"; echo " [checkov] $(echo "$NORM" | jq length) finding(s)" >&2 +else note_missing checkov "pipx install checkov"; fi + +# --- pip-audit (vulnerable Python deps) — runs on requirements*.txt in scope --- +if command -v pip-audit >/dev/null; then + REQS="$(for p in $SCOPE_PATHS; do find "$p" -maxdepth 3 -name 'requirements*.txt' 2>/dev/null; done)" + if [ -n "$REQS" ]; then + PAALL="$TMP/pa.json"; echo '[]' > "$PAALL" + for r in $REQS; do + RAW="$(pip-audit -r "$r" -f json 2>/dev/null || true)"; [ -z "$RAW" ] && continue + NORM="$(echo "$RAW" | jq --arg f "$r" '[ (.dependencies // .)[] | . as $d | ($d.vulns // [])[] | { + id: ("pipaudit-" + $d.name + "-" + .id), + title: ("vulnerable dep " + $d.name + " " + $d.version + " (" + .id + ")"), + severity: "high", cwe: "n/a", file: $f, line: null, + category: "secrets-crypto", source: "pip-audit", status: "confirmed", + data_flow: "known-vulnerable dependency", proof: {input: "install", outcome: (.description // .id)}}]' 2>/dev/null || echo '[]')" + jq -s '.[0]+.[1]' "$PAALL" <(echo "$NORM") > "$PAALL.t" && mv "$PAALL.t" "$PAALL" + done + add "$(cat "$PAALL")"; echo " [pip-audit] $(jq length "$PAALL") finding(s)" >&2 + else echo " [pip-audit] no requirements*.txt in scope" >&2; fi +else note_missing pip-audit "pipx install pip-audit"; fi + +# --- npm audit (vulnerable Node deps) — runs at TARGET root if package.json present --- +if command -v npm >/dev/null; then + if [ -f "$TARGET/package.json" ]; then + RAW="$(cd "$TARGET" && npm audit --json 2>/dev/null || true)" + if [ -n "$RAW" ] && echo "$RAW" | jq -e '.vulnerabilities' >/dev/null 2>&1; then + NORM="$(echo "$RAW" | jq '[.vulnerabilities // {} | to_entries[] | .value as $v | { + id: ("npmaudit-" + $v.name), + title: ("vulnerable npm dep " + $v.name + " (" + ($v.range // "") + ")"), + severity: ($v.severity | if .=="moderate" then "medium" elif .=="critical" then "critical" elif .=="high" then "high" elif .=="low" then "low" else "info" end), + cwe: ([$v.via[]? | objects | .cwe[]?] | if length>0 then .[0] else "n/a" end), + file: "package.json", line: null, category: "secrets-crypto", source: "npm-audit", status: "confirmed", + data_flow: "known-vulnerable npm dependency", + proof: {input: "install", outcome: (([$v.via[]? | objects | .title] | join("; "))[0:140])}}]')" + add "$NORM"; echo " [npm-audit] $(echo "$NORM" | jq length) finding(s)" >&2 + else echo " [npm-audit] 0 findings / no lockfile" >&2; fi + else echo " [npm-audit] no package.json at target root" >&2; fi +else note_missing npm-audit "install Node.js"; fi + +# --- Agent findings (from interactive /sh-security-review, or Path B headless later) --- +if [ "$SCANNERS_ONLY" -eq 0 ] && [ -n "$AGENT_FINDINGS" ]; then + [ -f "$AGENT_FINDINGS" ] || { echo "review.sh: agent-findings file not found: $AGENT_FINDINGS" >&2; exit 2; } + AF="$(jq 'if type=="object" then (.findings // []) else . end' "$AGENT_FINDINGS")" + add "$AF"; echo "== Agent findings: $(echo "$AF" | jq length) ==" >&2 +fi + +# --- Normalize file paths to be repo-relative (scanners emit absolute) --- +TGT_ABS="$(cd "$TARGET" && pwd)" +jq --arg tgt "$TGT_ABS" '[.[] | .file |= ((. // "") | ltrimstr($tgt) | ltrimstr("/"))]' "$ALL" > "$ALL.t" && mv "$ALL.t" "$ALL" + +# --- Dedup by (file, cwe-or-id) keeping the highest severity --- +RANK='{"critical":4,"high":3,"medium":2,"low":1,"info":0,"unverified":-1}' +DEDUP="$(jq --argjson r "$RANK" ' + def rank: ($r[.severity] // 0); + group_by([.file, (if (.cwe // "n/a")=="n/a" then .id else .cwe end), (.line // 0)]) | map(max_by(rank))' "$ALL")" + +# --- Apply suppressions (require a written justification; surface them) --- +if [ -n "$SUPPRESSIONS" ] && [ -f "$SUPPRESSIONS" ]; then + DEDUP="$(jq --slurpfile s "$SUPPRESSIONS" ' + ($s[0].suppressions // []) as $sup + | map( . as $f + | ([ $sup[] | select(.id == $f.id) ] | first) as $m + | if $m then + if ($m.justification // "" | length) > 0 + then $f + {status:"suppressed", suppression_justification:$m.justification} + else $f + {status:"unverified", suppression_justification:"REJECTED: suppression missing justification"} + end + else $f end )' <<<"$DEDUP")" +fi + +# --- Gate (mechanical) --- +SUMMARY="$(jq -n --argjson f "$DEDUP" ' + def isconf(s): [ $f[] | select(.status=="confirmed" and .severity==s) ] | length; + { confirmed_critical: isconf("critical"), confirmed_high: isconf("high"), + confirmed_medium: isconf("medium"), + suppressed: ([ $f[] | select(.status=="suppressed") ] | length), + unverified: ([ $f[] | select(.status=="unverified") ] | length) } + | . + { block: ((.confirmed_critical + .confirmed_high) > 0) }')" + +OUT="$(jq -n --argjson findings "$DEDUP" --argjson summary "$SUMMARY" '{findings:$findings, summary:$summary}')" +[ -n "$JSON_OUT" ] && echo "$OUT" > "$JSON_OUT" + +# --- Human report --- +echo +echo "================ SECURITY REVIEW ================" +echo "$SUMMARY" | jq -r '" confirmed critical: \(.confirmed_critical) confirmed high: \(.confirmed_high) medium: \(.confirmed_medium) suppressed: \(.suppressed) unverified: \(.unverified)"' +echo "-------------------------------------------------" +echo "$DEDUP" | jq -r ' + def order: {critical:0,high:1,medium:2}[.severity] // 9; + [ .[] | select(.status=="confirmed" and (.severity|IN("critical","high","medium"))) ] | sort_by(order) | .[] + | " [\(.severity|ascii_upcase)] \(.cwe // "n/a") \(.file)\(if .line then ":"+( .line|tostring) else "" end) — \(.title // .id)"' +echo "$DEDUP" | jq -r ' + ([ .[] | select(.status=="confirmed" and .severity=="low") ] | length) as $lo + | ([ .[] | select(.status=="confirmed" and .severity=="info") ] | length) as $in + | if ($lo+$in)>0 then " (+\($lo) low, +\($in) info — informational, in JSON report only)" else empty end' +SUPN="$(echo "$SUMMARY" | jq '.suppressed')" +if [ "$SUPN" -gt 0 ]; then + echo " -- suppressed (logged) --" + echo "$DEDUP" | jq -r '.[] | select(.status=="suppressed") | " [SUPPRESSED] \(.file) \(.id) — \(.suppression_justification)"' +fi +echo "=================================================" + +if [ "$(echo "$SUMMARY" | jq '.block')" = "true" ]; then + echo "RESULT: BLOCK (unsuppressed confirmed critical/high)"; exit 1 +else + echo "RESULT: PASS"; exit 0 +fi diff --git a/run_headless.py b/run_headless.py new file mode 100644 index 0000000..178e249 --- /dev/null +++ b/run_headless.py @@ -0,0 +1,447 @@ +#!/usr/bin/env python3 +"""run_headless.py — Path B headless detector fan-out + proof-or-kill verifier. + +Reuses the /sh-security-review detector + verifier prompts, but runs them +unattended via the Claude Agent SDK instead of interactive Claude Code subagents. +Authenticates with the Claude subscription OAuth token (CLAUDE_CODE_OAUTH_TOKEN) +through the bundled `claude` CLI — NEVER a raw ANTHROPIC_API_KEY (which would be +metered and would silently win if both were set, so we pop it). + +Emits the finding-schema JSON ({findings, summary}) that +`review.sh --agent-findings` consumes. review.sh re-derives the gate, so this +script's job is high-recall candidate generation + proof-or-kill verification, +failing toward over-reporting (never silently drops a candidate or a parse error). + +See memory project-security-review-agent and security-review/DEPLOY-R720.md. + +Usage: + CLAUDE_CODE_OAUTH_TOKEN=... python3 run_headless.py TARGET_DIR \ + [--scope "src infra web"] [--out findings.json] [--model claude-...] \ + [--concurrency 3] [--detectors injection,authz] \ + [--detector-budget-usd 2.0] [--total-budget-usd 12.0] +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import re +import sys +import time +from pathlib import Path + +from claude_agent_sdk import ClaudeAgentOptions, query + +# Read-only surface: detectors reason over source, they don't mutate or fetch. +READONLY_TOOLS = ["Read", "Grep", "Glob"] +BLOCKED_TOOLS = ["Bash", "Write", "Edit", "NotebookEdit", "WebFetch", "WebSearch"] + +# Detector category -> closed checklist (verbatim intent from sh-security-review.md). +DETECTORS: dict[str, str] = { + "injection": "SQL/command/template injection, unsafe deserialization, SSRF, path/file traversal, XXE", + "authz": "broken object-level auth/IDOR, missing access checks, missing webhook/Slack signature verification, auth bypass", + "secrets-crypto": "hardcoded secrets/keys, weak/broken crypto (MD5/SHA1/unsalted), sensitive data in logs/errors/responses, wrong SSM-vs-Secrets-Manager placement", + "iac-iam": "wildcard IAM actions/resources, public buckets/endpoints, open security-group ingress (0.0.0.0/0), missing encryption, over-broad trust policies", + "web-client": "XSS (incl. dangerouslySetInnerHTML), CSRF, open redirect, client-side secret exposure", + "logic": "broken multi-step invariants, race conditions, missing tenant isolation, auth-state confusion", +} + +DETECTOR_TMPL = """You are a hostile {category} security auditor for Sea Haven. Assume this code is \ +hostile and the author missed something. Audit ONLY {category} issues in: {scope}. The repository root \ +is your current working directory; read the actual files with your tools. Ignore .git and any file that \ +is obviously an answer key. + +For each checklist item, either name a specific line that is provably safe, OR file a finding. Do not \ +hand-wave or give an open "looks fine" verdict. + +Checklist: {checklist} + +Return ONLY a JSON array (no prose, no markdown fences) where each element is: +{{"id": "", "title": "...", "claimed_severity": "critical|high|medium|low|info", + "cwe": "CWE-####", "file": "", "line": , "category": "{category}", + "data_flow": "numbered source->sink trace", "proof": {{"input": "concrete malicious input/trigger", + "outcome": "the specific bad result", "test": "optional failing-test sketch or null"}}, + "recommendation": "..."}} +If there are no findings, return [].""" + +VERIFIER_TMPL = """You are a skeptical exploitation verifier. You did NOT find these; your job is to \ +REFUTE weak claims. The repository root is your current working directory; read the real files to check \ +reachability before ruling. + +For each candidate finding decide whether there is a concrete, plausible proof-of-exploit (a specific \ +malicious input and the specific bad outcome, consistent with the code): +- If yes: set "status":"confirmed", keep "severity" equal to the claimed_severity, and tighten the proof. +- If no / speculative / not reachable: set "status":"unverified" and "severity":"unverified". Default to \ +unverified when uncertain. A confident assertion with no demonstrable input is NOT proof. + +Return ONLY a JSON array (no prose, no fences) of the SAME findings, each preserving its original "id", \ +"title", "cwe", "file", "line", "category", "claimed_severity", "data_flow", "recommendation", and adding \ +"severity" (final) plus "status" and the verified "proof". Return one element per candidate — do not drop \ +any. + +Candidates: +{candidates}""" + +SEV_RANK = { + "critical": 4, + "high": 3, + "medium": 2, + "low": 1, + "info": 0, + "unverified": -1, +} + + +def log(msg: str) -> None: + print(f"[run_headless] {msg}", file=sys.stderr, flush=True) + + +def extract_json(text: str): + """Best-effort parse of a JSON array/object from an agent's final text.""" + if not text: + return None + t = text.strip() + if t.startswith("```"): + t = re.sub(r"^```[a-zA-Z0-9]*\n?", "", t) + t = re.sub(r"\n?```\s*$", "", t).strip() + try: + return json.loads(t) + except Exception: + pass + # Fall back to the outermost [...] (or {...}) span. + for open_c, close_c in (("[", "]"), ("{", "}")): + start, end = t.find(open_c), t.rfind(close_c) + if 0 <= start < end: + try: + return json.loads(t[start : end + 1]) + except Exception: + continue + return None + + +async def run_agent( + prompt: str, *, cwd: Path, model: str | None, max_turns: int, budget_usd: float +) -> tuple[str, float, bool]: + """Run one fresh-context agent turn; return (final_text, cost_usd, is_error).""" + opts = ClaudeAgentOptions( + allowed_tools=READONLY_TOOLS, + disallowed_tools=BLOCKED_TOOLS, + permission_mode="bypassPermissions", + setting_sources=[], # hermetic: ignore user/project/local config + CLAUDE.md + cwd=str(cwd), + model=model, + max_turns=max_turns, + max_budget_usd=budget_usd, + ) + texts: list[str] = [] + result_text: str | None = None + cost = 0.0 + is_error = False + async for msg in query(prompt=prompt, options=opts): + name = type(msg).__name__ + if name == "AssistantMessage": + for block in getattr(msg, "content", []) or []: + t = getattr(block, "text", None) + if t: + texts.append(t) + elif name == "ResultMessage": + result_text = getattr(msg, "result", None) + cost = float(getattr(msg, "total_cost_usd", 0.0) or 0.0) + is_error = bool(getattr(msg, "is_error", False)) + return (result_text or "\n".join(texts)), cost, is_error + + +class Budget: + def __init__(self, total: float) -> None: + self.total = total + self.spent = 0.0 + self._lock = asyncio.Lock() + + async def add(self, amount: float) -> None: + async with self._lock: + self.spent += amount + + def exhausted(self) -> bool: + return self.total > 0 and self.spent >= self.total + + +async def run_detector( + category: str, + scope: str, + *, + cwd: Path, + model: str | None, + max_turns: int, + budget_usd: float, + sem: asyncio.Semaphore, + budget: Budget, +) -> tuple[str, list[dict], str | None]: + """Returns (category, candidate_findings, error_message_or_None).""" + async with sem: + if budget.exhausted(): + return category, [], "skipped: total budget exhausted" + prompt = DETECTOR_TMPL.format( + category=category, scope=scope, checklist=DETECTORS[category] + ) + t0 = time.monotonic() + try: + text, cost, is_error = await run_agent( + prompt, cwd=cwd, model=model, max_turns=max_turns, budget_usd=budget_usd + ) + except Exception as exc: # never let one detector kill the run + log(f"detector {category}: EXCEPTION {type(exc).__name__}: {exc}") + return category, [], f"exception: {type(exc).__name__}: {exc}" + await budget.add(cost) + dt = time.monotonic() - t0 + parsed = extract_json(text) + if not isinstance(parsed, list): + log( + f"detector {category}: UNPARSEABLE output ({dt:.0f}s, ${cost:.3f}) — over-reporting as error" + ) + return category, [], "unparseable detector output (NOT treated as clean)" + for f in parsed: + if isinstance(f, dict): + f.setdefault("category", category) + log( + f"detector {category}: {len(parsed)} candidate(s) ({dt:.0f}s, ${cost:.3f})" + + (" [is_error]" if is_error else "") + ) + return category, [f for f in parsed if isinstance(f, dict)], None + + +async def run_verifier( + candidates: list[dict], + *, + cwd: Path, + model: str | None, + max_turns: int, + budget_usd: float, + budget: Budget, +) -> tuple[dict[str, dict], float, str | None]: + """Returns ({id -> verdict}, cost, error). Empty verdicts on failure (caller keeps candidates).""" + prompt = VERIFIER_TMPL.format(candidates=json.dumps(candidates, indent=2)) + try: + text, cost, _ = await run_agent( + prompt, cwd=cwd, model=model, max_turns=max_turns, budget_usd=budget_usd + ) + except Exception as exc: + log(f"verifier: EXCEPTION {type(exc).__name__}: {exc}") + return {}, 0.0, f"exception: {type(exc).__name__}: {exc}" + await budget.add(cost) + parsed = extract_json(text) + if not isinstance(parsed, list): + log( + f"verifier: UNPARSEABLE output (${cost:.3f}) — keeping all candidates as unverified" + ) + return {}, cost, "unparseable verifier output" + verdicts = {f["id"]: f for f in parsed if isinstance(f, dict) and f.get("id")} + log(f"verifier: ruled on {len(verdicts)} finding(s) (${cost:.3f})") + return verdicts, cost, None + + +def merge(candidates: list[dict], verdicts: dict[str, dict]) -> list[dict]: + """Apply verifier verdicts to candidates. A candidate the verifier dropped or never + ruled on stays as 'unverified' — fail toward over-reporting, never silently delete.""" + out: list[dict] = [] + for c in candidates: + fid = c.get("id") or f"anon-{c.get('file', '?')}-{c.get('line', '?')}" + c.setdefault("id", fid) + v = verdicts.get(fid, {}) + status = v.get("status") + if status not in ("confirmed", "unverified", "suppressed"): + status = "unverified" + if status == "confirmed": + severity = v.get("severity") or c.get("claimed_severity") or "high" + else: + severity = "unverified" + out.append( + { + "id": fid, + "title": v.get("title") or c.get("title") or fid, + "severity": severity, + "claimed_severity": c.get("claimed_severity") or "high", + "cwe": c.get("cwe") or v.get("cwe") or "n/a", + "file": c.get("file") or v.get("file") or "", + "line": c.get("line", v.get("line")), + "category": c.get("category") or v.get("category") or "other", + "data_flow": v.get("data_flow") or c.get("data_flow") or "", + "proof": v.get("proof") + or c.get("proof") + or {"input": "", "outcome": ""}, + "status": status, + "recommendation": v.get("recommendation") + or c.get("recommendation") + or "", + } + ) + return out + + +def summarize(findings: list[dict]) -> dict: + conf_crit = sum( + 1 + for f in findings + if f["status"] == "confirmed" and f["severity"] == "critical" + ) + conf_high = sum( + 1 for f in findings if f["status"] == "confirmed" and f["severity"] == "high" + ) + return { + "confirmed_critical": conf_crit, + "confirmed_high": conf_high, + "block": (conf_crit + conf_high) > 0, + } + + +async def main_async(args: argparse.Namespace) -> int: + target = Path(args.target).resolve() + if not target.is_dir(): + log(f"target dir not found: {target}") + return 2 + scope = args.scope.strip() if args.scope else "the entire repository" + selected = ( + [d.strip() for d in args.detectors.split(",") if d.strip()] + if args.detectors + else list(DETECTORS) + ) + unknown = [d for d in selected if d not in DETECTORS] + if unknown: + log(f"unknown detector(s): {unknown}; valid: {list(DETECTORS)}") + return 2 + + budget = Budget(args.total_budget_usd) + sem = asyncio.Semaphore(max(1, args.concurrency)) + log( + f"target={target} scope='{scope}' detectors={selected} model={args.model or 'cli-default'} " + f"concurrency={args.concurrency} total_budget=${args.total_budget_usd}" + ) + + det_results = await asyncio.gather( + *[ + run_detector( + cat, + scope, + cwd=target, + model=args.model, + max_turns=args.max_turns, + budget_usd=args.detector_budget_usd, + sem=sem, + budget=budget, + ) + for cat in selected + ] + ) + + candidates: list[dict] = [] + errors: list[str] = [] + for cat, found, err in det_results: + candidates.extend(found) + if err: + errors.append(f"{cat}: {err}") + + log( + f"total candidates: {len(candidates)}; detector spend so far: ${budget.spent:.3f}" + ) + + if candidates and not budget.exhausted(): + verdicts, _, verr = await run_verifier( + candidates, + cwd=target, + model=args.model, + max_turns=args.max_turns, + budget_usd=args.detector_budget_usd, + budget=budget, + ) + if verr: + errors.append(f"verifier: {verr}") + else: + verdicts = {} + if budget.exhausted(): + errors.append( + "verifier: skipped (budget exhausted) — all candidates left unverified" + ) + + findings = merge(candidates, verdicts) + report = { + "findings": findings, + "summary": summarize(findings), + "_meta": { + "target": str(target), + "scope": scope, + "detectors": selected, + "model": args.model or "cli-default", + "spend_usd": round(budget.spent, 4), + "errors": errors, + }, + } + out = json.dumps(report, indent=2) + if args.out: + Path(args.out).write_text(out) + log(f"wrote {args.out}") + else: + print(out) + + s = report["summary"] + log( + f"DONE: {s['confirmed_critical']} confirmed-crit, {s['confirmed_high']} confirmed-high, " + f"block={s['block']}, spend=${budget.spent:.3f}, errors={len(errors)}" + ) + if errors: + for e in errors: + log(f" ERROR/NOTE: {e}") + return 0 + + +def main() -> None: + p = argparse.ArgumentParser( + description="Headless Sea Haven security detector fan-out + verifier" + ) + p.add_argument("target", help="target repository directory") + p.add_argument( + "--scope", default="", help="space-separated subdirs to restrict the audit" + ) + p.add_argument( + "--out", default="", help="write findings JSON here (default: stdout)" + ) + p.add_argument("--model", default=None, help="model id (default: CLI default)") + p.add_argument( + "--detectors", + default="", + help="comma list to restrict detectors (default: all 6)", + ) + p.add_argument( + "--concurrency", type=int, default=3, help="max concurrent detectors" + ) + p.add_argument( + "--max-turns", + type=int, + default=40, + help="max agent turns per detector/verifier", + ) + p.add_argument( + "--detector-budget-usd", type=float, default=2.0, help="per-call SDK spend cap" + ) + p.add_argument( + "--total-budget-usd", + type=float, + default=12.0, + help="overall spend cap (0 = unlimited)", + ) + args = p.parse_args() + + # Guarantee the subscription OAuth path: a raw API key would silently win, so remove it. + if os.environ.pop("ANTHROPIC_API_KEY", None): + log("removed ANTHROPIC_API_KEY from env to force the subscription OAuth path") + if not os.environ.get("CLAUDE_CODE_OAUTH_TOKEN"): + log( + "FATAL: CLAUDE_CODE_OAUTH_TOKEN not set (source ~/secrev.env). Refusing to run." + ) + sys.exit(2) + + sys.exit(asyncio.run(main_async(args))) + + +if __name__ == "__main__": + main() diff --git a/skill/sh-security-review.md b/skill/sh-security-review.md new file mode 100644 index 0000000..96dcc45 --- /dev/null +++ b/skill/sh-security-review.md @@ -0,0 +1,93 @@ +--- +name: sh-security-review +description: High-recall agentic security review with anti-complacency structure. Opus fans out N narrow fresh-context detectors over the target, a separate fresh-context verifier demands proof-of-exploit or downgrades to unverified, then findings are emitted in the structured schema with a block decision. Returns severity-ranked findings each carrying a concrete proof. +--- + +# Sea Haven Security Review (detector fan-out + proof-or-kill verifier) + +A high-recall security review built to resist reviewer complacency. Instead of one model judging the +whole surface (that's `sh-build-review`), this fans out **N narrow detectors, each fresh context and +adversarial**, then a **separate verifier** with the opposing incentive demands a concrete +proof-of-exploit for every candidate or downgrades it to `unverified`. Output is the structured +finding schema (`~/.claude/security-review/finding.schema.json`) plus a block decision. + +This is the interactive (Path A) entry point and runs under the Max subscription. The same prompts and +schema are reused by the automated `review.sh` (Path B) later. See memory `project-security-review-agent`. + +## When to Use +- Security review of a branch, working tree, file, or whole repo +- Before pushing payments/auth/IaC/input-handling changes +- As the high-recall pass; complements `/security-review`, `/code-review ultra`, and `sh-build-review` + +## Arguments +- Optional target: a path, file, branch, or diff. Default: the current working tree / branch diff vs main. +- Optional `--scope `: restrict the scan (e.g. `src/ infra/ web/`). + +## Mechanism (Opus runs these) + +Opus is the main loop. It scopes the target, runs the detector fan-out and verifier as subagents +(each with **fresh context** so no "we already passed 10 files" approval prior accumulates), then +applies the gate rule and reports. Opus does NOT soften the gate; the block condition is mechanical. + +### 1. Scope +Identify the files in scope (respect `--scope`; never scan an answer key or `.git`). Note the languages +present (Python/Lambda, .NET, React/JS, SAM/CDK IaC) so detectors apply the right checklist. + +### 2. Detector fan-out (parallel, fresh context, closed checklist, adversarial) +Spawn these detectors as **separate parallel subagents** (`subagent_type: general-purpose`). Each gets +ONLY its category, the schema, and the adversarial framing. Each must, per checklist item, either cite a +specific safe line OR file a finding — no open "looks fine" judgment. + +Detectors: +- **injection** — SQL/command/template injection, unsafe deserialization, SSRF, path/file traversal, XXE +- **authz** — broken object-level auth/IDOR, missing access checks, missing webhook/Slack signature verification, auth bypass +- **secrets-crypto** — hardcoded secrets/keys, weak/again crypto (MD5/SHA1/unsalted), sensitive data in logs/errors/responses, wrong SSM-vs-Secrets-Manager +- **iac-iam** — wildcard IAM actions/resources, public buckets/endpoints, open security-group ingress (0.0.0.0/0), missing encryption, over-broad trust policies +- **web-client** — XSS (incl. dangerouslySetInnerHTML), CSRF, open redirect, client-side secret exposure +- **logic** — broken multi-step invariants, race conditions, missing tenant isolation, auth-state confusion + +Detector prompt template (fill `{CATEGORY}`, `{CHECKLIST}`, `{SCOPE}`): +``` +You are a hostile {CATEGORY} security auditor for Sea Haven. Assume this code is hostile and the author +missed something. Audit ONLY {CATEGORY} issues in: {SCOPE}. Read the actual files. +For each checklist item, either name a specific line that is safe, OR file a finding. Do not hand-wave. +Checklist: {CHECKLIST} +Return a JSON array of findings, each: {id, title, claimed_severity (critical|high|medium|low|info), +cwe (CWE-####), file, line, category:"{CATEGORY}", data_flow (numbered source->sink trace), +proof:{input, outcome, test|null}, recommendation}. If none, return []. No prose outside the JSON. +``` + +### 3. Verifier (separate subagent, fresh context, proof-or-kill) +Spawn one or more **verifier** subagents with the OPPOSING incentive. The verifier did not find these and +is rewarded for killing weak claims. For each candidate it demands a concrete, plausible proof. +``` +You are a skeptical exploitation verifier. You did NOT find these; your job is to REFUTE weak claims. +For each candidate finding, decide: is there a concrete, plausible proof-of-exploit (a specific malicious +input and the specific bad outcome, consistent with the code)? +- If yes: status="confirmed", keep severity = claimed_severity, tighten the proof. +- If no / speculative / not reachable: status="unverified" and severity="unverified". Default to unverified + when uncertain. A confident assertion without a demonstrable input is NOT proof. +Return the findings array with status, severity, and the verified proof. No prose outside the JSON. +``` + +### 4. Merge + gate (mechanical, Opus does not soften) +- Dedup by (file, cwe, nearby line); keep the highest accepted severity. +- `summary.confirmed_critical` / `confirmed_high` = counts of status=confirmed at that severity. +- `summary.block = true` if any unsuppressed confirmed critical/high exists. +- A finding may be suppressed ONLY with a written `suppression_justification`, which is surfaced in the report. + No silent dismissal: a critical/high with no proof becomes `unverified` (still listed), never deleted. + +### 5. Report +Emit the schema JSON, then a human summary: BLOCK/PASS, confirmed findings highest-severity-first, each +with file:line, the numbered data-flow trace, and the proof. List unverified and suppressed separately so +nothing is silently dropped. The reader reviews proofs, not raw code. + +## Relationship to existing tooling +- `sh-build-review` — single deep Fable pass over a change surface (depth, one reasoner). +- `sh-security-review` — high-recall fan-out + proof-or-kill verifier (breadth + anti-complacency). +- IAM/policy and Lambda-signature changes still require the mandatory cross-family review per global instructions; this does not replace it. + +## Output +- Structured findings (schema) + block decision +- Confirmed findings with proofs, unverified and suppressed listed separately +- HARD STOP / BLOCK surfaced when `summary.block` is true diff --git a/sweep-targets.txt b/sweep-targets.txt new file mode 100644 index 0000000..9be820e --- /dev/null +++ b/sweep-targets.txt @@ -0,0 +1,15 @@ +# sweep-targets.txt — one repo path per line for the nightly Path B sweep. +# Lines starting with '#' and blank lines are ignored. ~ is expanded. +# Override at runtime with the TARGETS env var (space-separated paths). +# +# NOTE: the testbed canary corpus is ALWAYS scanned by nightly_sweep.sh as the +# anti-complacency check; do NOT list it here (it is handled separately). +# +# TODO (Phase 5): add the first real hardened repo here once it is cloned on the +# VM (candidates: payments-dashboard / proposal-system / procurement-ingest). +# +# Deliberately EMPTY by default = canary-only nights. Do NOT scan ~/orchestrator: +# it holds ~/orchestrator/.env with live provider API keys, which the agentic +# detector could surface into sweep reports / Slack. Only add repos with no +# plaintext secrets (or scrub/exclude secret files first). +# ~/orchestrator diff --git a/systemd/sea-haven-checkers.service b/systemd/sea-haven-checkers.service new file mode 100644 index 0000000..93dc6f1 --- /dev/null +++ b/systemd/sea-haven-checkers.service @@ -0,0 +1,52 @@ +# sea-haven-checkers.service — Plane-1 nightly checker coordinator (sh-secrev VM, user adam). +# +# Runs security-review/checker_coordinator.sh: the read-only Plane-1 checkers +# (compliance-drift, dependency-cve, doc-drift, plan-groomer) under ONE shared +# budget ledger + versioned rotation/coverage, ALARM-only to Slack. Reuses the +# secrev sweep's substrate ($MIRROR_DIR clones, budget discipline) — no re-clone. +# +# Install (on the VM, as root): +# sudo cp sea-haven-checkers.service /etc/systemd/system/ +# sudo cp sea-haven-checkers.timer /etc/systemd/system/ +# sudo systemctl daemon-reload +# sudo systemctl enable --now sea-haven-checkers.timer # the timer drives it +# systemctl list-timers sea-haven-checkers.timer +# +# Secrets/config come from the EnvironmentFiles (leading '-' = optional): +# ~/secrev.env -> CLAUDE_CODE_OAUTH_TOKEN, GH_TOKEN, SLACK_WEBHOOK_URL +# ~/orchestrator/.env -> OPENAI/etc. (only if a checker shells the cross-model run.py) +# +# COORDINATOR_SKIP_ROLES excludes roles whose creds are NOT provisioned: +# - aws-posture needs IAM Roles Anywhere / step-ca (not provisioned) +# confluence-doc is ONLINE (2026-06-22): authenticates via the OAuth 2.0 +# client-credentials service account in ~/secrev.env (CONFLUENCE_BASE_URL + +# CONFLUENCE_OAUTH_CLIENT_ID/_SECRET); PAGE_MAP_FILE points at the IT page-ID map +# generated from the live space. Remove a name from the skip list once its +# credential is provisioned to bring that checker online. + +[Unit] +Description=Sea Haven agent-team Plane-1 nightly checker coordinator +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=adam +WorkingDirectory=/home/adam/orchestrator/security-review +EnvironmentFile=-/home/adam/secrev.env +EnvironmentFile=-/home/adam/orchestrator/.env +Environment=GH_ORG=Sea-Haven-Industries +Environment=COORDINATOR_SKIP_ROLES=aws-posture +# IT page-ID map for confluence-doc (generated from the live space; regenerate +# periodically as pages change). +Environment=PAGE_MAP_FILE=/home/adam/confluence-page-map.json +# Tune the shared ceiling without editing the script (uncomment to override): +# Environment=TOTAL_BUDGET_USD=120 +# Environment=MAX_CYCLE_NIGHTS=4 +ExecStart=/home/adam/orchestrator/security-review/checker_coordinator.sh +# Bounded so a hung checker cannot run forever; spend is capped by TOTAL_BUDGET_USD. +TimeoutStartSec=10800 +Nice=10 + +[Install] +WantedBy=multi-user.target diff --git a/systemd/sea-haven-checkers.timer b/systemd/sea-haven-checkers.timer new file mode 100644 index 0000000..7307f36 --- /dev/null +++ b/systemd/sea-haven-checkers.timer @@ -0,0 +1,20 @@ +# sea-haven-checkers.timer — fires the Plane-1 checker coordinator nightly. +# +# 03:30 UTC — ~90 min after the sea-haven-secrev sweep (02:00) so the two do not +# contend on $MIRROR_DIR or the shared Claude subscription pool at the same instant. +# Persistent=true → if the VM was off, it runs at next boot. RandomizedDelaySec +# spreads load off an exact-minute spike. +# +# Install: see the header of sea-haven-checkers.service. + +[Unit] +Description=Run the Sea Haven Plane-1 checker coordinator nightly (~03:30 UTC) + +[Timer] +OnCalendar=*-*-* 03:30:00 +Persistent=true +RandomizedDelaySec=600 +Unit=sea-haven-checkers.service + +[Install] +WantedBy=timers.target diff --git a/systemd/sea-haven-secrev.service b/systemd/sea-haven-secrev.service new file mode 100644 index 0000000..cb53527 --- /dev/null +++ b/systemd/sea-haven-secrev.service @@ -0,0 +1,49 @@ +# sea-haven-secrev.service — Path B nightly security sweep (sh-secrev VM, user adam). +# +# Install (on the VM, as root): +# sudo cp sea-haven-secrev.service /etc/systemd/system/ +# sudo cp sea-haven-secrev.timer /etc/systemd/system/ +# sudo systemctl daemon-reload +# sudo systemctl enable --now sea-haven-secrev.timer # timer drives the run; do NOT enable the .service +# systemctl list-timers sea-haven-secrev.timer # confirm next run +# sudo systemctl start sea-haven-secrev.service # optional: run once now to smoke-test +# journalctl -u sea-haven-secrev.service -e # logs (also under ~/sweep-reports//) +# +# Secrets come from the EnvironmentFiles (the leading '-' = optional, no failure if absent): +# ~/secrev.env -> CLAUDE_CODE_OAUTH_TOKEN (required by run_headless.py), +# GH_TOKEN (read-only fine-grained PAT — REQUIRED for org auto-discovery), +# SLACK_WEBHOOK_URL +# ~/orchestrator/.env -> OPENAI_API_KEY etc. (only needed if ENABLE_XMODEL_HOOK=1) +# +# GH_TOKEN must be a fine-grained PAT scoped to the Sea-Haven-Industries org with READ-ONLY +# Contents (and Metadata) permission — nothing else. It enumerates repos and clones them into +# ~/repo-mirrors. Never give this unattended box a write-capable token. + +[Unit] +Description=Sea Haven Path B nightly security sweep +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=adam +WorkingDirectory=/home/adam/orchestrator +EnvironmentFile=-/home/adam/secrev.env +EnvironmentFile=-/home/adam/orchestrator/.env +# Tune ceilings/targets here without editing the script (uncomment to override defaults): +# Environment=TOTAL_BUDGET_USD=120 +# Environment=PER_TARGET_BUDGET_USD=12 +# Environment=CANARY_FLOOR=10 +# Environment=MAX_CYCLE_NIGHTS=4 +# Environment=MAX_AGENTIC_PER_NIGHT=0 +# Environment=GH_ORG=Sea-Haven-Industries +# Environment=MIRROR_DIR=/home/adam/repo-mirrors +# Environment=ENABLE_XMODEL_HOOK=0 +ExecStart=/home/adam/orchestrator/security-review/nightly_sweep.sh +# Two-tier sweep (scanners over every repo + a budget-bounded agentic rotation) runs for hours; +# 6h ceiling bounds a hang without killing a healthy long night. Spend is capped by TOTAL_BUDGET_USD. +TimeoutStartSec=21600 +Nice=10 + +[Install] +WantedBy=multi-user.target diff --git a/systemd/sea-haven-secrev.timer b/systemd/sea-haven-secrev.timer new file mode 100644 index 0000000..b377585 --- /dev/null +++ b/systemd/sea-haven-secrev.timer @@ -0,0 +1,20 @@ +# sea-haven-secrev.timer — fires the nightly sweep at ~02:00 local, user adam. +# +# Install: see the header of sea-haven-secrev.service. In short: +# sudo systemctl enable --now sea-haven-secrev.timer +# systemctl list-timers sea-haven-secrev.timer +# +# Persistent=true → if the VM was off at 02:00, the sweep runs at next boot. +# RandomizedDelaySec spreads load off an exact-minute spike. + +[Unit] +Description=Run the Sea Haven Path B security sweep nightly (~02:00) + +[Timer] +OnCalendar=*-*-* 02:00:00 +Persistent=true +RandomizedDelaySec=600 +Unit=sea-haven-secrev.service + +[Install] +WantedBy=timers.target