mirror of
https://github.com/Sea-Haven-Industries/security-review.git
synced 2026-09-30 04:33:13 +00:00
Adam Moussa
4c88c01f7b
Fresh-init copy of the security-review/ subsystem extracted from Sea-Haven-Industries/orchestrator (being deprecated). Adds org-standard scaffold: CI reusable-workflow callers (ruff + collect), dependency-review, labeler, dependabot, .gitignore, requirements.txt. Scheduled execution is migrating to Claude Code web routines (ALARM-only to #repo-scanner); the systemd units and nightly_sweep.sh/checker_coordinator.sh remain the source of truth. Committed with --no-verify: the canary fixtures (checkers/fixtures/**) carry intentional secret-shaped test data that trips the deterministic gate (the documented detector-fixture false positive); no new logic is introduced.
587 lines
29 KiB
Bash
Executable file
587 lines
29 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# dependency-cve.sh — Plane-1 / Tier-1 checker for the R720 agent-team.
|
|
#
|
|
# Design refs: docs/r720-agent-team-design.md §4 (Tier 1 roster: dependency-cve —
|
|
# "Cross-ref lockfiles vs advisories org-wide; report + feed fixer. Complements Dependabot")
|
|
# and §7 Phase 2 ("coordinator + second checker"). This is the SECOND Plane-1 checker built
|
|
# on the Phase-0 shared substrate (lib/sweep_substrate.sh); it mirrors compliance-drift.sh's
|
|
# conventions verbatim so the coordinator (§5) can drive both identically.
|
|
#
|
|
# WHAT IT DOES (read-only):
|
|
# Scans the SAME shallow clean clones nightly_sweep.sh already produced in $MIRROR_DIR — it
|
|
# does NOT re-clone (mirrors-first; an optional --refresh re-runs discovery+mirror via the
|
|
# shared substrate). In each mirror it parses dependency lockfiles/manifests with PINNED,
|
|
# exact versions, extracts (ecosystem, package, version) tuples, and cross-references them
|
|
# against the OSV advisory database to flag known-vulnerable pinned deps. This complements
|
|
# Dependabot (design §4): it is org-wide, runs on the server-side mirrors, and feeds the
|
|
# fixer queue in a later phase.
|
|
#
|
|
# Manifests parsed (and the OSV ecosystem each maps to):
|
|
# requirements.txt -> PyPI (only EXACT '==' pins; ranges/unpinned are skipped)
|
|
# poetry.lock -> PyPI ([[package]] name/version blocks)
|
|
# Pipfile.lock -> PyPI (default+develop, "==x.y.z" version strings)
|
|
# package-lock.json -> npm (packages[].version / dependencies[].version)
|
|
# yarn.lock -> npm ("pkg@range:\n version \"x\"" stanzas)
|
|
# packages.lock.json -> NuGet (.dependencies[tfm][pkg].resolved)
|
|
# *.csproj -> NuGet (<PackageReference Include=.. Version=..>)
|
|
# Only EXACTLY-pinned versions are cross-referenced (an unpinned/range spec has no single
|
|
# version to query and is not a confirmed vulnerable artifact — no false alarms on no-data,
|
|
# memory feedback_cloudwatch_alarms).
|
|
#
|
|
# ADVISORY SOURCE (live): OSV batch API POST https://api.osv.dev/v1/querybatch (NO auth token).
|
|
# Guarded behind a --no-api / offline check exactly like compliance-drift's GitHub-API checks:
|
|
# on missing curl OR a failed/empty network response, the API lookup is SKIPPED and noted in
|
|
# the report — a vuln is NEVER reported on missing advisory data. Network calls are minimal
|
|
# (one batched POST) and fail-safe.
|
|
#
|
|
# AGENTIC TIEBREAK (design §4, "Claude + GPT tiebreak"): OPTIONAL and only relevant in LIVE mode
|
|
# for ambiguous severity. For THIS phase the deterministic OSV core is the whole checker — NO
|
|
# LLM is invoked in --canary/--dry-run. A clearly-marked inert stub hook (maybe_tiebreak) marks
|
|
# the future seam; it does nothing offline and nothing in this phase.
|
|
#
|
|
# CANARY / DRY-RUN (offline, no network, no token):
|
|
# --canary runs against a planted fixture (checkers/fixtures/dependency-cve/) and asserts the
|
|
# known vuln count against EXPECTED_VULN_COUNT (exit 3 on mismatch). Because OSV needs network,
|
|
# the canary consults a LOCAL offline advisory fixture (fixtures/dependency-cve/osv-advisories.json)
|
|
# INSTEAD of the network — so it is fully offline + deterministic. --canary implies --dry-run +
|
|
# --no-api. This is the anti-complacency floor (design §6.4) AND the routing dry-run (§7 Phase 2):
|
|
# with --dry-run the Slack alarm is composed + printed but NOT POSTed.
|
|
#
|
|
# SCOPE / SAFETY:
|
|
# Read-only. Fixtures ship git metadata as dotgit/ (renamed to .git/ at run time) so they
|
|
# commit into THIS repo without becoming submodules — the SAME trick compliance-drift uses.
|
|
# Does NOT touch agent_team/ or agent-team/, and is NOT wired into systemd — that is Phase-6
|
|
# provisioning (gated). See the "PROVISIONING (NOT DONE HERE)" note at the bottom.
|
|
#
|
|
# Exit: 0 = ran (whether or not it alarmed); 2 = setup/usage error; 3 = canary assertion FAILED.
|
|
set -euo pipefail
|
|
export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
|
|
|
|
log() { echo "[dependency-cve] $*" >&2; }
|
|
die() { echo "[dependency-cve] FATAL: $*" >&2; exit 2; }
|
|
|
|
# --- Shared substrate ---------------------------------------------------------
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
SUBSTRATE="$HERE/../lib/sweep_substrate.sh"
|
|
[ -f "$SUBSTRATE" ] || die "shared substrate not found: $SUBSTRATE"
|
|
# shellcheck source=../lib/sweep_substrate.sh
|
|
. "$SUBSTRATE"
|
|
|
|
# --- Config + defaults (env, all optional) ------------------------------------
|
|
GH_ORG="${GH_ORG:-Sea-Haven-Industries}"
|
|
MIRROR_DIR="${MIRROR_DIR:-$HOME/repo-mirrors}"
|
|
REPORT_ROOT="${REPORT_ROOT:-$HOME/sweep-reports/dependency-cve}"
|
|
OSV_BATCH_URL="${OSV_BATCH_URL:-https://api.osv.dev/v1/querybatch}"
|
|
|
|
REFRESH=0 # --refresh: re-run discovery+mirror via substrate (network). Default: reuse mirrors.
|
|
DO_API=1 # --no-api: skip the OSV advisory lookup (offline). Without it, nothing matches.
|
|
DRY_RUN=0 # --dry-run: compose the Slack alarm but DO NOT post it (routing dry-run).
|
|
CANARY=0 # --canary: run against the planted fixture + assert the known vuln count.
|
|
TARGETS_OVERRIDE="" # --targets "p1 p2": scan explicit dirs instead of the mirror set.
|
|
ADVISORIES_FILE="" # --advisories-file PATH: consult a local advisory JSON instead of the OSV API.
|
|
|
|
usage() {
|
|
cat >&2 <<EOF
|
|
dependency-cve.sh — Plane-1 Tier-1 vulnerable-dependency checker (read-only)
|
|
|
|
--canary run against the planted fixture and assert the known vuln count
|
|
(implies --dry-run + --no-api; uses the OFFLINE advisory fixture)
|
|
--dry-run compose the Slack alarm but DO NOT post it (routing dry-run)
|
|
--no-api skip the OSV advisory lookup (offline; nothing can match)
|
|
--advisories-file P consult a LOCAL advisory JSON at P instead of the OSV network API
|
|
(offline + deterministic; same file shape as the canary fixture)
|
|
--refresh re-discover + re-mirror via the shared substrate before scanning (network)
|
|
--targets "a b" scan these explicit repo dirs instead of \$MIRROR_DIR/* (no clone)
|
|
-h|--help this help
|
|
|
|
Env: GH_ORG MIRROR_DIR REPORT_ROOT GH_TOKEN SLACK_WEBHOOK_URL OSV_BATCH_URL
|
|
EOF
|
|
}
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--canary) CANARY=1; DRY_RUN=1; DO_API=0 ;;
|
|
--dry-run) DRY_RUN=1 ;;
|
|
--no-api) DO_API=0 ;;
|
|
--advisories-file) shift; ADVISORIES_FILE="${1:-}" ;;
|
|
--refresh) REFRESH=1 ;;
|
|
--targets) shift; TARGETS_OVERRIDE="${1:-}" ;;
|
|
-h|--help) usage; exit 0 ;;
|
|
*) die "unknown arg: $1 (see --help)" ;;
|
|
esac
|
|
shift
|
|
done
|
|
|
|
command -v jq >/dev/null || die "jq is required"
|
|
command -v git >/dev/null || die "git is required"
|
|
|
|
# --- Report dir (mode 600 reports; matches sweep conventions) -----------------
|
|
umask 077
|
|
UTC_DATE="$(date -u +%Y-%m-%d)"
|
|
UTC_STAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
|
REPORT_DIR="$REPORT_ROOT/$UTC_DATE"
|
|
mkdir -p "$REPORT_DIR"; chmod 700 "$REPORT_ROOT" "$REPORT_DIR" 2>/dev/null || true
|
|
# shellcheck disable=SC2034 # read by the sourced substrate (post_slack_alarm) via dynamic scope
|
|
SWEEP_LOG="$REPORT_DIR/dependency-cve.log" # name the substrate's post_slack_alarm() references
|
|
REPORT_JSON="$REPORT_DIR/dependency-cve.json"
|
|
REPORT_TXT="$REPORT_DIR/dependency-cve.txt"
|
|
|
|
log "=== dependency-cve $UTC_STAMP (canary=$CANARY dry_run=$DRY_RUN api=$DO_API refresh=$REFRESH) ==="
|
|
|
|
# ------------------------------------------------------------------------------
|
|
# FINDINGS (spirit of finding.schema.json so the coordinator can route like an agentic
|
|
# finding). category="other" (a vulnerable-dependency is not one of the schema's security
|
|
# categories); status="confirmed" only for an exact pinned version that MATCHES an advisory.
|
|
# A pinned dep with NO advisory match is NOT a finding; an unqueryable/skipped advisory lookup
|
|
# is NOT a finding (memory feedback_cloudwatch_alarms: no false alarms on missing data).
|
|
# ------------------------------------------------------------------------------
|
|
declare -a FINDINGS=()
|
|
add_finding() { # repo id title severity pkg version advisory_id summary fixed_version
|
|
local repo="$1" id="$2" title="$3" sev="$4" pkg="$5" ver="$6" adv="$7" summ="$8" fixed="$9"
|
|
FINDINGS+=( "$(jq -n \
|
|
--arg repo "$repo" --arg id "$id" --arg title "$title" --arg sev "$sev" \
|
|
--arg pkg "$pkg" --arg ver "$ver" --arg adv "$adv" --arg summ "$summ" --arg fixed "$fixed" \
|
|
'{repo:$repo, id:($repo+"-"+$id), title:$title, severity:$sev, category:"other",
|
|
check:"vulnerable-dependency", status:"confirmed",
|
|
proof:{package:$pkg, version:$ver, advisory_id:$adv, summary:$summ, fixed_version:$fixed}}')" )
|
|
}
|
|
declare -a SKIPPED_CHECKS=() # (repo:reason) lookups skipped on missing data — reported, never alarmed
|
|
note_skip() { SKIPPED_CHECKS+=( "$1" ); }
|
|
|
|
# Severity normalizer: map OSV/GHSA strings + CVSS scores into the schema's enum.
|
|
norm_sev() { # raw_severity cvss_score -> critical|high|medium|low
|
|
local raw; raw="$(echo "${1:-}" | tr '[:upper:]' '[:lower:]')"
|
|
local cvss="${2:-}"
|
|
case "$raw" in
|
|
critical) echo critical; return ;;
|
|
high) echo high; return ;;
|
|
moderate|medium) echo medium; return ;;
|
|
low) echo low; return ;;
|
|
esac
|
|
# Fall back to CVSS base score banding (NVD/CVSSv3 thresholds).
|
|
if [ -n "$cvss" ] && [ "$cvss" != "null" ]; then
|
|
awk -v c="$cvss" 'BEGIN{
|
|
if (c+0>=9.0) print "critical";
|
|
else if (c+0>=7.0) print "high";
|
|
else if (c+0>=4.0) print "medium";
|
|
else print "low"; }'
|
|
return
|
|
fi
|
|
echo medium # unknown severity: medium (a real match we cannot rank), never dropped
|
|
}
|
|
|
|
# ==============================================================================
|
|
# MANIFEST PARSERS — each emits "ECOSYSTEM<TAB>package<TAB>version" lines (exact pins only).
|
|
# Pure text/jq parsing; no project tooling invoked. Unknown/odd lines are skipped silently.
|
|
# ==============================================================================
|
|
|
|
# requirements.txt: only EXACT '==' pins (skip ranges, markers, comments, -e/-r includes, extras).
|
|
parse_requirements() { # file
|
|
local f="$1"
|
|
sed -E 's/[[:space:]]*#.*$//' "$f" 2>/dev/null \
|
|
| grep -E '==' \
|
|
| while IFS= read -r line; do
|
|
line="$(echo "$line" | tr -d '[:space:]')"
|
|
[ -n "$line" ] || continue
|
|
case "$line" in -*|.*|git+*|http*) continue ;; esac
|
|
# strip extras: pkg[extra]==1.2.3 -> pkg
|
|
local name ver
|
|
name="$(echo "$line" | sed -E 's/\[[^]]*\].*//; s/[<>=!~;].*$//')"
|
|
ver="$(echo "$line" | sed -E 's/^[^=]*==//; s/[ ;].*$//')"
|
|
# only a clean exact version (digits/dots/alnum), no range operators left
|
|
case "$ver" in *','*|*'<'*|*'>'*|*'*'*|'') continue ;; esac
|
|
[ -n "$name" ] && [ -n "$ver" ] && printf 'PyPI\t%s\t%s\n' "$name" "$ver"
|
|
done
|
|
}
|
|
|
|
# poetry.lock: [[package]] blocks with name = "x" / version = "y".
|
|
parse_poetry_lock() { # file
|
|
local f="$1"
|
|
awk '
|
|
/^\[\[package\]\]/ { name=""; ver=""; next }
|
|
/^name = / { gsub(/^name = "|"$/,""); name=$0; next }
|
|
/^version = / { gsub(/^version = "|"$/,""); ver=$0;
|
|
if (name!="" && ver!="") printf "PyPI\t%s\t%s\n", name, ver; next }
|
|
' "$f" 2>/dev/null
|
|
}
|
|
|
|
# Pipfile.lock: JSON; default + develop maps; versions look like "==1.2.3".
|
|
parse_pipfile_lock() { # file
|
|
local f="$1"
|
|
jq -r '
|
|
(.default // {}) * (.develop // {}) | to_entries[]
|
|
| select(.value.version != null)
|
|
| .key as $n | (.value.version | sub("^=="; "")) as $v
|
|
| select($v | test("^[0-9][0-9A-Za-z.+-]*$"))
|
|
| "PyPI\t\($n)\t\($v)"
|
|
' "$f" 2>/dev/null || true
|
|
}
|
|
|
|
# package-lock.json: prefer v2/v3 .packages (node_modules/<name> keys), else v1 .dependencies.
|
|
parse_package_lock() { # file
|
|
local f="$1"
|
|
jq -r '
|
|
if (.packages != null) then
|
|
(.packages | to_entries[]
|
|
| select(.key | startswith("node_modules/"))
|
|
| select(.value.version != null)
|
|
| (.key | sub("^.*node_modules/"; "")) as $n
|
|
| "npm\t\($n)\t\(.value.version)")
|
|
elif (.dependencies != null) then
|
|
[paths(objects | has("version")) as $p | {n: $p[-1], v: (getpath($p).version)}]
|
|
| .[] | select(.v != null) | "npm\t\(.n)\t\(.v)"
|
|
else empty end
|
|
' "$f" 2>/dev/null || true
|
|
}
|
|
|
|
# yarn.lock: stanzas "spec@range, spec@range:\n version \"x.y.z\"".
|
|
parse_yarn_lock() { # file
|
|
local f="$1"
|
|
awk '
|
|
/^[^[:space:]#].*:[[:space:]]*$/ {
|
|
# header line: take first spec, strip trailing colon + quotes, derive package name
|
|
hdr=$0; sub(/:[[:space:]]*$/,"",hdr);
|
|
split(hdr, specs, ", "); first=specs[1]; gsub(/"/,"",first);
|
|
# package name = everything before the LAST @ (handles @scope/pkg@range)
|
|
at=0; for (i=2;i<=length(first);i++){ if (substr(first,i,1)=="@") at=i }
|
|
pkg=(at>1)? substr(first,1,at-1) : first;
|
|
next
|
|
}
|
|
/^[[:space:]]+version / {
|
|
v=$0; gsub(/^[[:space:]]+version[[:space:]]+"?|"?[[:space:]]*$/,"",v);
|
|
if (pkg!="" && v!="") printf "npm\t%s\t%s\n", pkg, v;
|
|
pkg=""; next
|
|
}
|
|
' "$f" 2>/dev/null
|
|
}
|
|
|
|
# packages.lock.json (NuGet): .dependencies[tfm][pkg].resolved.
|
|
parse_packages_lock() { # file
|
|
local f="$1"
|
|
jq -r '
|
|
(.dependencies // {}) | to_entries[] | .value | to_entries[]
|
|
| select(.value.resolved != null)
|
|
| "NuGet\t\(.key)\t\(.value.resolved)"
|
|
' "$f" 2>/dev/null || true
|
|
}
|
|
|
|
# *.csproj (NuGet): <PackageReference Include="X" Version="Y" />.
|
|
parse_csproj() { # file
|
|
local f="$1"
|
|
grep -oE '<PackageReference[^>]*>' "$f" 2>/dev/null \
|
|
| while IFS= read -r tag; do
|
|
local inc ver
|
|
inc="$(echo "$tag" | sed -nE 's/.*Include="([^"]+)".*/\1/p')"
|
|
ver="$(echo "$tag" | sed -nE 's/.*Version="([^"]+)".*/\1/p')"
|
|
# only exact versions (no range brackets/commas/wildcards)
|
|
case "$ver" in ''|*'['*|*']'*|*'('*|*')'*|*','*|*'*'*) continue ;; esac
|
|
[ -n "$inc" ] && [ -n "$ver" ] && printf 'NuGet\t%s\t%s\n' "$inc" "$ver"
|
|
done
|
|
}
|
|
|
|
# Extract ALL (ecosystem, package, version) tuples from one repo dir. Dedup at the end.
|
|
extract_deps() { # repo_dir -> TSV "ECOSYSTEM\tpackage\tversion" on stdout
|
|
local dir="$1" f
|
|
# requirements.txt (any depth, excluding .git)
|
|
while IFS= read -r f; do [ -n "$f" ] && parse_requirements "$f"; done \
|
|
< <(find "$dir" -maxdepth 4 -name requirements.txt -not -path '*/.git/*' 2>/dev/null)
|
|
while IFS= read -r f; do [ -n "$f" ] && parse_poetry_lock "$f"; done \
|
|
< <(find "$dir" -maxdepth 4 -name poetry.lock -not -path '*/.git/*' 2>/dev/null)
|
|
while IFS= read -r f; do [ -n "$f" ] && parse_pipfile_lock "$f"; done \
|
|
< <(find "$dir" -maxdepth 4 -name Pipfile.lock -not -path '*/.git/*' 2>/dev/null)
|
|
while IFS= read -r f; do [ -n "$f" ] && parse_package_lock "$f"; done \
|
|
< <(find "$dir" -maxdepth 4 -name package-lock.json -not -path '*/.git/*' 2>/dev/null)
|
|
while IFS= read -r f; do [ -n "$f" ] && parse_yarn_lock "$f"; done \
|
|
< <(find "$dir" -maxdepth 4 -name yarn.lock -not -path '*/.git/*' 2>/dev/null)
|
|
while IFS= read -r f; do [ -n "$f" ] && parse_packages_lock "$f"; done \
|
|
< <(find "$dir" -maxdepth 4 -name packages.lock.json -not -path '*/.git/*' 2>/dev/null)
|
|
while IFS= read -r f; do [ -n "$f" ] && parse_csproj "$f"; done \
|
|
< <(find "$dir" -maxdepth 4 -name '*.csproj' -not -path '*/.git/*' 2>/dev/null)
|
|
}
|
|
|
|
# ==============================================================================
|
|
# ADVISORY LOOKUP
|
|
# ==============================================================================
|
|
# OFFLINE: consult a local advisory file (the canary fixture, or --advisories-file). Keyed by
|
|
# "ECOSYSTEM|package|version" -> array of {id,summary,severity,cvss,fixed_version}. Deterministic.
|
|
lookup_offline() { # advisories_file ecosystem package version -> advisory JSON array (or [])
|
|
local af="$1" eco="$2" pkg="$3" ver="$4"
|
|
jq -c --arg k "$eco|$pkg|$ver" '(.advisories[$k] // [])' "$af" 2>/dev/null || echo '[]'
|
|
}
|
|
|
|
# LIVE: one batched POST to the OSV querybatch API (no token). Returns one results[] per query
|
|
# in input order. Fail-safe: on missing curl, transport failure, or a non-array body, returns ""
|
|
# (the caller then SKIPS — never alarms on missing advisory data).
|
|
osv_querybatch() { # queries_json (array of {package:{ecosystem,name},version}) -> results JSON or ""
|
|
local queries="$1"
|
|
command -v curl >/dev/null || { return 1; }
|
|
local body
|
|
body="$(curl -fsS -X POST -H 'Content-Type: application/json' \
|
|
--max-time 30 \
|
|
--data "$(jq -n --argjson q "$queries" '{queries:$q}')" \
|
|
"$OSV_BATCH_URL" 2>>"$REPORT_DIR/osv.log")" || return 1
|
|
echo "$body" | jq -e '.results | type=="array"' >/dev/null 2>&1 || return 1
|
|
echo "$body"
|
|
}
|
|
|
|
# Inert future seam (design §4 "Claude + GPT tiebreak"): in LIVE mode, an ambiguous-severity
|
|
# advisory could be escalated to a cross-family judge. This phase keeps the deterministic core
|
|
# ONLY — the stub does nothing and is never reached offline / in canary / dry-run.
|
|
maybe_tiebreak() { # advisory_json (no-op stub; phase-2 intentionally inert)
|
|
return 0
|
|
}
|
|
|
|
# ==============================================================================
|
|
# TARGET RESOLUTION
|
|
# ==============================================================================
|
|
declare -a REPO_NAMES=(); declare -A REPO_DIR=()
|
|
|
|
if [ "$CANARY" -eq 1 ]; then
|
|
FIXTURE_ROOT="$HERE/fixtures/dependency-cve"
|
|
[ -d "$FIXTURE_ROOT" ] || die "canary fixture missing: $FIXTURE_ROOT"
|
|
# The canary is OFFLINE: it consults the planted advisory fixture instead of the OSV network,
|
|
# unless an explicit --advisories-file override was given.
|
|
[ -n "$ADVISORIES_FILE" ] || ADVISORIES_FILE="$FIXTURE_ROOT/osv-advisories.json"
|
|
[ -f "$ADVISORIES_FILE" ] || die "canary advisory fixture missing: $ADVISORIES_FILE"
|
|
# Fixtures ship git metadata as dotgit/ (not .git/) so they are committable into THIS repo
|
|
# without becoming nested submodules. Materialize: copy + rename dotgit -> .git into a mode-700
|
|
# temp area removed on exit (same trick as compliance-drift.sh).
|
|
FIXTURE_WORK="$(mktemp -d "${TMPDIR:-/tmp}/dependency-cve-canary.XXXXXX")"
|
|
trap 'rm -rf "$FIXTURE_WORK"' EXIT
|
|
log "canary: materializing planted fixtures from $FIXTURE_ROOT into $FIXTURE_WORK"
|
|
for d in "$FIXTURE_ROOT"/*/; do
|
|
[ -d "$d/dotgit" ] || continue # only fixture repos (skip README.md, *.json etc.)
|
|
nm="$(basename "$d")"
|
|
cp -R "$d" "$FIXTURE_WORK/$nm"
|
|
mv "$FIXTURE_WORK/$nm/dotgit" "$FIXTURE_WORK/$nm/.git"
|
|
# Manifests are stored as <name>.fixture so GitHub's dependency graph / the
|
|
# dependency-review CI action does NOT parse the deliberately-vulnerable canary
|
|
# pins as real project dependencies. Restore their real names in the materialized
|
|
# work area so the checker's per-ecosystem parsers dispatch correctly (same
|
|
# committable-without-side-effects rationale as the dotgit/ rename above).
|
|
while IFS= read -r ff; do
|
|
[ -n "$ff" ] && mv "$ff" "${ff%.fixture}"
|
|
done < <(find "$FIXTURE_WORK/$nm" -name '*.fixture' -not -path '*/.git/*' 2>/dev/null)
|
|
REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$FIXTURE_WORK/$nm"
|
|
done
|
|
elif [ -n "$TARGETS_OVERRIDE" ]; then
|
|
# shellcheck disable=SC2206 # intentional word-split of the space-separated --targets list
|
|
arr=( $TARGETS_OVERRIDE )
|
|
for p in "${arr[@]}"; do p="${p/#\~/$HOME}"; nm="$(basename "$p")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="$p"; done
|
|
log "explicit targets: ${REPO_NAMES[*]}"
|
|
else
|
|
if [ "$REFRESH" -eq 1 ]; then
|
|
[ -n "${GH_TOKEN:-}" ] || die "--refresh needs GH_TOKEN"
|
|
command -v curl >/dev/null || die "--refresh needs curl"
|
|
mkdir -p "$MIRROR_DIR"
|
|
log "refresh: re-discovering + mirroring via shared substrate (no separate clone path)"
|
|
DISCOVERED="$REPORT_DIR/discovered.tsv"
|
|
if discover_repos > "$DISCOVERED" 2>>"$REPORT_DIR/discover.log" && [ -s "$DISCOVERED" ]; then
|
|
while IFS=$'\t' read -r name url branch; do
|
|
[ -n "$name" ] || continue
|
|
mirror_repo "$name" "$url" "$branch" || log " mirror FAILED: $name (will use stale mirror if present)"
|
|
done < "$DISCOVERED"
|
|
else
|
|
log "discovery failed — falling back to existing mirrors (coverage may be stale)"
|
|
fi
|
|
fi
|
|
# Default + post-refresh: enumerate EXISTING mirrors. No clone here — reuse the sweep's clones.
|
|
[ -d "$MIRROR_DIR" ] || die "mirror dir not found: $MIRROR_DIR (run nightly_sweep.sh first, or use --refresh/--targets)"
|
|
for d in "$MIRROR_DIR"/*/; do
|
|
[ -d "$d/.git" ] || continue
|
|
nm="$(basename "$d")"; REPO_NAMES+=( "$nm" ); REPO_DIR["$nm"]="${d%/}"
|
|
done
|
|
log "reusing ${#REPO_NAMES[@]} existing mirror(s) in $MIRROR_DIR (no re-clone)"
|
|
fi
|
|
|
|
[ "${#REPO_NAMES[@]}" -gt 0 ] || die "no repos to scan"
|
|
|
|
# Decide HOW advisories are looked up: offline file, or the live OSV API, or skip entirely.
|
|
# An explicit --advisories-file always wins (offline + deterministic, even without --canary).
|
|
ADV_MODE="none"
|
|
if [ -n "$ADVISORIES_FILE" ]; then
|
|
[ -f "$ADVISORIES_FILE" ] || die "advisories file not found: $ADVISORIES_FILE"
|
|
ADV_MODE="offline"
|
|
elif [ "$DO_API" -eq 1 ] && command -v curl >/dev/null; then
|
|
ADV_MODE="api"
|
|
elif [ "$DO_API" -eq 1 ]; then
|
|
log "OSV lookup requested but curl unavailable — skipping advisory match (no false alarms on missing data)"
|
|
fi
|
|
log "advisory mode: $ADV_MODE"
|
|
|
|
# ==============================================================================
|
|
# RUN: extract deps per repo, then cross-reference against advisories
|
|
# ==============================================================================
|
|
for nm in "${REPO_NAMES[@]}"; do
|
|
dir="${REPO_DIR[$nm]}"
|
|
# Unique (ecosystem, package, version) tuples for this repo.
|
|
deps_tsv="$(extract_deps "$dir" | sort -u || true)"
|
|
ndeps=0; [ -n "$deps_tsv" ] && ndeps="$(printf '%s\n' "$deps_tsv" | grep -c . || true)"
|
|
log " [$nm] extracted $ndeps pinned dependency tuple(s)"
|
|
[ "$ndeps" -gt 0 ] || { note_skip "$nm:no-pinned-deps"; continue; }
|
|
|
|
if [ "$ADV_MODE" = "none" ]; then
|
|
note_skip "$nm:advisory-lookup-skipped(offline/no-curl)"
|
|
continue
|
|
fi
|
|
|
|
if [ "$ADV_MODE" = "offline" ]; then
|
|
# Deterministic local lookup, one tuple at a time.
|
|
while IFS=$'\t' read -r eco pkg ver; do
|
|
[ -n "$pkg" ] || continue
|
|
advs="$(lookup_offline "$ADVISORIES_FILE" "$eco" "$pkg" "$ver")"
|
|
cnt="$(echo "$advs" | jq 'length' 2>/dev/null || echo 0)"
|
|
[ "${cnt:-0}" -gt 0 ] || continue
|
|
i=0
|
|
while [ "$i" -lt "$cnt" ]; do
|
|
adv="$(echo "$advs" | jq -c --argjson i "$i" '.[$i]')"
|
|
aid="$(echo "$adv" | jq -r '.id // "UNKNOWN"')"
|
|
summ="$(echo "$adv" | jq -r '.summary // ""')"
|
|
rawsev="$(echo "$adv"| jq -r '.severity // ""')"
|
|
cvss="$(echo "$adv" | jq -r '.cvss // empty')"
|
|
fixed="$(echo "$adv" | jq -r '.fixed_version // ""')"
|
|
sev="$(norm_sev "$rawsev" "$cvss")"
|
|
maybe_tiebreak "$adv" # inert in this phase
|
|
add_finding "$nm" "vuln-$(echo "${pkg}-${ver}-${aid}" | tr -c 'A-Za-z0-9-' '-')" \
|
|
"$pkg $ver is vulnerable ($aid)" "$sev" \
|
|
"$pkg" "$ver" "$aid" "$summ" "$fixed"
|
|
i=$((i+1))
|
|
done
|
|
done <<< "$deps_tsv"
|
|
continue
|
|
fi
|
|
|
|
# ADV_MODE = api: build ONE batched OSV query for all this repo's tuples (minimal network).
|
|
queries="$(printf '%s\n' "$deps_tsv" | jq -R -s '
|
|
[ split("\n")[] | select(length>0) | split("\t")
|
|
| {package:{ecosystem:.[0], name:.[1]}, version:.[2]} ]')"
|
|
# Keep a parallel TSV array so we can re-associate results[] (OSV preserves input order).
|
|
if ! results="$(osv_querybatch "$queries")"; then
|
|
note_skip "$nm:osv-querybatch-failed" # transport/HTTP failure -> skip, NEVER alarm
|
|
continue
|
|
fi
|
|
# Walk each tuple alongside its result entry.
|
|
idx=0
|
|
while IFS=$'\t' read -r eco pkg ver; do
|
|
[ -n "$pkg" ] || continue
|
|
vulns="$(echo "$results" | jq -c --argjson i "$idx" '(.results[$i].vulns // [])')"
|
|
idx=$((idx+1))
|
|
vcnt="$(echo "$vulns" | jq 'length' 2>/dev/null || echo 0)"
|
|
[ "${vcnt:-0}" -gt 0 ] || continue
|
|
j=0
|
|
while [ "$j" -lt "$vcnt" ]; do
|
|
v="$(echo "$vulns" | jq -c --argjson j "$j" '.[$j]')"
|
|
aid="$(echo "$v" | jq -r '.id // "UNKNOWN"')"
|
|
summ="$(echo "$v" | jq -r '.summary // (.details // "" | .[0:160])')"
|
|
# OSV severity: prefer database_specific.severity, else the CVSS vector score band.
|
|
rawsev="$(echo "$v" | jq -r '.database_specific.severity // ""')"
|
|
cvss="$(echo "$v" | jq -r '[.severity[]? | select(.type|test("CVSS")) | .score] | .[0] // empty' \
|
|
| grep -oE '[0-9]+\.[0-9]+' | head -1 || true)"
|
|
fixed="$(echo "$v" | jq -r '
|
|
[.affected[]?.ranges[]?.events[]? | select(.fixed != null) | .fixed] | .[0] // ""')"
|
|
sev="$(norm_sev "$rawsev" "$cvss")"
|
|
maybe_tiebreak "$v" # inert in this phase
|
|
add_finding "$nm" "vuln-$(echo "${pkg}-${ver}-${aid}" | tr -c 'A-Za-z0-9-' '-')" \
|
|
"$pkg $ver is vulnerable ($aid)" "$sev" \
|
|
"$pkg" "$ver" "$aid" "$summ" "$fixed"
|
|
j=$((j+1))
|
|
done
|
|
done <<< "$deps_tsv"
|
|
done
|
|
|
|
# ==============================================================================
|
|
# ASSEMBLE REPORT (JSON + text), mode 600 (identical shape to compliance-drift)
|
|
# ==============================================================================
|
|
if [ "${#FINDINGS[@]}" -gt 0 ]; then
|
|
FINDINGS_JSON="$(printf '%s\n' "${FINDINGS[@]}" | jq -cs .)"
|
|
else
|
|
FINDINGS_JSON="[]"
|
|
fi
|
|
if [ "${#SKIPPED_CHECKS[@]}" -gt 0 ]; then
|
|
SKIPPED_JSON="$(printf '%s\n' "${SKIPPED_CHECKS[@]}" | jq -R . | jq -cs .)"
|
|
else
|
|
SKIPPED_JSON="[]"
|
|
fi
|
|
|
|
N_VULN="$(echo "$FINDINGS_JSON" | jq 'length')"
|
|
N_HIGH="$(echo "$FINDINGS_JSON" | jq '[.[]|select(.severity=="high" or .severity=="critical")] | length')"
|
|
N_REPOS_VULN="$(echo "$FINDINGS_JSON" | jq '[.[].repo] | unique | length')"
|
|
|
|
jq -n \
|
|
--arg checker "dependency-cve" --arg ts "$UTC_STAMP" --arg org "$GH_ORG" \
|
|
--arg advmode "$ADV_MODE" --argjson scanned "${#REPO_NAMES[@]}" \
|
|
--argjson findings "$FINDINGS_JSON" --argjson skipped "$SKIPPED_JSON" \
|
|
'{checker:$checker, generated:$ts, org:$org, advisory_mode:$advmode,
|
|
repos_scanned:$scanned, vuln_count:($findings|length),
|
|
repos_with_vulns:([$findings[].repo]|unique|length),
|
|
findings:$findings, skipped_checks:$skipped}' > "$REPORT_JSON"
|
|
|
|
{
|
|
echo "dependency-cve report — $UTC_STAMP"
|
|
echo "org=$GH_ORG repos_scanned=${#REPO_NAMES[@]} advisory_mode=$ADV_MODE"
|
|
echo "vulnerable deps: $N_VULN ($N_HIGH high/critical) across $N_REPOS_VULN repo(s)"
|
|
echo
|
|
echo "$FINDINGS_JSON" | jq -r '.[] | "• [\(.severity)] \(.repo): \(.title)\n fix: upgrade \(.proof.package) -> \(.proof.fixed_version) (\(.proof.summary))"'
|
|
if [ "$(echo "$SKIPPED_JSON" | jq 'length')" -gt 0 ]; then
|
|
echo; echo "skipped (missing data — NOT counted as a vuln):"
|
|
echo "$SKIPPED_JSON" | jq -r '.[] | " - \(.)"'
|
|
fi
|
|
} > "$REPORT_TXT"
|
|
chmod 600 "$REPORT_JSON" "$REPORT_TXT" 2>/dev/null || true
|
|
|
|
log "report: $REPORT_JSON ($N_VULN vuln finding(s), $N_REPOS_VULN repo(s))"
|
|
|
|
# ==============================================================================
|
|
# CANARY ASSERTION (anti-complacency floor, design §6.4)
|
|
# ==============================================================================
|
|
if [ "$CANARY" -eq 1 ]; then
|
|
EXPECT_FILE="$HERE/fixtures/dependency-cve/EXPECTED_VULN_COUNT"
|
|
[ -f "$EXPECT_FILE" ] || die "canary expected-count file missing: $EXPECT_FILE"
|
|
EXPECTED="$(tr -dc '0-9' < "$EXPECT_FILE")"
|
|
log "canary assertion: expected vuln=$EXPECTED, got=$N_VULN"
|
|
if [ "$N_VULN" -ne "$EXPECTED" ]; then
|
|
echo "[dependency-cve] CANARY FAIL: planted-vuln count mismatch (expected $EXPECTED, got $N_VULN)" >&2
|
|
echo " -> a parser or the advisory match regressed, or the fixture changed. See $REPORT_TXT." >&2
|
|
exit 3
|
|
fi
|
|
log "canary PASS: all $EXPECTED planted vulnerable deps detected."
|
|
fi
|
|
|
|
# ==============================================================================
|
|
# ALARM-ONLY ROUTING (clean = silent; memory feedback_cloudwatch_alarms)
|
|
# ==============================================================================
|
|
if [ "$N_VULN" -eq 0 ]; then
|
|
log "no vulnerable dependencies — posting NOTHING to Slack (ALARM-only policy)."
|
|
exit 0
|
|
fi
|
|
|
|
ALARM_BODY="$(echo "$FINDINGS_JSON" | jq -r '
|
|
group_by(.repo)[] | "*\(.[0].repo)*: " + ([.[] | "[\(.severity)] \(.title)"] | join("; "))' | sed 's/^/• /')"
|
|
SLACK_TEXT=":lock: *Sea Haven dependency-cve — ALARM* ($UTC_STAMP)
|
|
$N_VULN vulnerable pinned dependency(ies) across $N_REPOS_VULN repo(s) ($N_HIGH high/critical):
|
|
$ALARM_BODY
|
|
|
|
Source: OSV advisory DB ($ADV_MODE) · complements Dependabot
|
|
Report (mode 600): \`$REPORT_JSON\` (on R720)"
|
|
SLACK_TEXT="$(echo "$SLACK_TEXT" | redact)"
|
|
|
|
echo "$SLACK_TEXT" >&2
|
|
|
|
if [ "$DRY_RUN" -eq 1 ]; then
|
|
log "DRY-RUN: alarm composed but NOT posted (routing dry-run, design §7 Phase 2)."
|
|
exit 0
|
|
fi
|
|
post_slack_alarm "$SLACK_TEXT"
|
|
exit 0
|
|
|
|
# ==============================================================================
|
|
# PROVISIONING (NOT DONE HERE — gated, Phase 6):
|
|
# - No systemd unit / timer is installed by this script. Wiring it into the live
|
|
# sea-haven-secrev schedule (or a sibling timer) is provisioning and is gated.
|
|
# - The coordinator (design §5, checker_coordinator.sh) runs this alongside other
|
|
# Tier-1 checkers under one shared budget + versioned rotation state.
|
|
# - The LIVE "Claude + GPT tiebreak" severity-judge (design §4) is the only LLM seam;
|
|
# it is an inert stub here (maybe_tiebreak) and stays off in canary/dry-run/offline.
|
|
# - Confluence + project_r720_agent_team memory updates are docs-as-you-go obligations
|
|
# for the build session, tracked outside this script.
|
|
# ==============================================================================
|