open-swe/.github/scripts/check-dev-green.sh
Adam Moussa 3224d7abf4
Some checks are pending
Build & publish app artifacts / Publish + deploy (dev) (push) Waiting to run
Build & publish app artifacts / Publish + deploy (prod) (push) Waiting to run
Infra CD / Infra CI (pre-deploy) (push) Waiting to run
Infra CD / Deploy open-swe-dev (push) Blocked by required conditions
Infra CD / Deploy open-swe-prod (push) Blocked by required conditions
Agent CI / Agent lint (push) Waiting to run
Agent CI / Agent format check (push) Waiting to run
Agent CI / Agent unit tests (push) Waiting to run
Agent CI / Playwright E2E (push) Waiting to run
ci: gate dev→prod promotion on green checks + add rollback safety net (#28)
T20 CD safety nets. Two gaps closed before the first real prod deploy:

1. Promotion gate. promote_dev_to_prod.yml previously fast-forwarded
   dev→main unconditionally. It now hard-gates on check-dev-green.sh:
   every check-run on the dev HEAD must be completed+passing AND the
   Agent CI suite (lint/format/unit/E2E) must be present+success, or the
   promotion blocks (fails safe on a missing/renamed check). The promote
   run excludes its OWN check-run by run-id (unforgeable), never by the
   mutable name "promote", so a colliding red check cannot hide. Fields
   are read with a 0x1F separator so an empty conclusion (every
   in_progress check) cannot shift columns. ci.yml now also runs on
   push:dev so dev HEAD actually carries that signal (a PR check alone
   can be admin-merged past).

2. Rollback + last-good. publish-and-deploy.sh advances
   releases/last-good/ only after a successful roll (deploy.sh gates on
   `systemctl is-active`), and makes releases/latest/ transactional —
   reverting to the prior release if the roll fails so a replaced box
   never self-deploys a broken release. New rollback.yml + rollback.sh
   re-point latest at last-good (or an explicit sha) and re-fire the
   deploy; prod is gated by the `prod` Environment approval, same as a
   deploy. The shared fire/wait/aggregate-gate logic is factored into
   roll-box.sh (used by both forward and backward rolls).

Least-privilege: drop the unused s3:DeleteObject from the app deploy
role — publish/rollback/deploy only Get+Put (S3-to-S3 copy), and the
rollback fallback now depends on immutable release history staying
intact. Lifecycle expiry (not CI) handles old-version cleanup.

Gate logic unit-tested (7 cases + jq round-trip). IAM change +
release-safety control cross-reviewed by GPT-4.1: APPROVE, no blocks.

Claude-Session: https://claude.ai/code/session_01DMhLf4G5V8MStJQyAW95hi
2026-06-27 20:21:59 -04:00

85 lines
3.3 KiB
Bash
Executable file

#!/usr/bin/env bash
# Promotion gate: refuse to promote a dev HEAD that is not fully green.
#
# Reads check-runs on stdin — one
# name<US>status<US>conclusion<US>details_url
# per line, fields separated by ASCII Unit Separator (0x1F) — so it is unit-testable
# WITHOUT GitHub. promote_dev_to_prod.yml pipes the live `gh api .../check-runs`
# output in. 0x1F (not TAB) is used deliberately: TAB is IFS-whitespace, so an empty
# conclusion (every in_progress check has a null conclusion) would collapse and shift
# the columns — which would make the promote run fail to exclude itself. 0x1F is
# non-whitespace, so `read` preserves empty fields and the columns stay aligned.
#
# A dev HEAD is promotable ONLY when BOTH hold:
# 1. every present check-run is completed with a passing conclusion
# (success/neutral/skipped); any pending/failed/cancelled/timed_out check BLOCKS.
# 2. every check named in REQUIRED_CHECKS is present AND concluded "success".
# This positive allow-list is what stops a partial-signal promotion — e.g. a
# commit pushed with [skip ci] (no CI check-runs) that still carries one
# unrelated green check, or a required check silently renamed/dropped. The gate
# FAILS SAFE: a missing required check blocks rather than promotes.
#
# Self-exclusion: the promotion workflow's OWN in-progress check-run is dropped by
# EXCLUDE_RUN_ID (its github.run_id, matched in the check-run details_url) — an
# unforgeable identity, NOT a mutable check name. A check that merely happens to be
# named "promote" can no longer hide a real failing/required check.
set -euo pipefail
EXCLUDE_RUN_ID="${EXCLUDE_RUN_ID:-}"
# Mandatory checks (one per line). Defaults to the Agent CI suite, which runs on
# every push to dev (see ci.yml). Keep in sync with those job names; if a name
# drifts the gate blocks (fails safe) until the list is updated.
REQUIRED_CHECKS="${REQUIRED_CHECKS:-Agent lint
Agent format check
Agent unit tests
Playwright E2E}"
declare -A GREEN
seen=0
bad=0
while IFS=$'\037' read -r name status conclusion details_url; do
[ -n "${name:-}" ] || continue
# drop the promotion run's own check-run by run id (never by name).
if [ -n "${EXCLUDE_RUN_ID}" ] && \
[ "${details_url}" != "${details_url#*/runs/${EXCLUDE_RUN_ID}/}" ]; then
continue
fi
seen=$((seen + 1))
if [ "${status}" != "completed" ]; then
echo "BLOCK: check '${name}' is '${status}' (not completed)" >&2
bad=1
continue
fi
case "${conclusion}" in
success)
GREEN["${name}"]=1
echo "ok: ${name} (success)" ;;
neutral | skipped)
echo "ok: ${name} (${conclusion})" ;;
*)
echo "BLOCK: check '${name}' concluded '${conclusion:-<none>}'" >&2
bad=1 ;;
esac
done
if [ "${seen}" -eq 0 ]; then
echo "BLOCK: no check-runs found for this commit — refusing to promote an unverified dev HEAD" >&2
exit 1
fi
missing=0
while IFS= read -r req; do
[ -n "${req}" ] || continue
if [ -z "${GREEN[${req}]:-}" ]; then
echo "BLOCK: required check '${req}' is missing or not successful on dev HEAD" >&2
missing=1
fi
done <<EOF
${REQUIRED_CHECKS}
EOF
if [ "${bad}" -ne 0 ] || [ "${missing}" -ne 0 ]; then
echo "BLOCK: dev HEAD is not promotable (see reasons above)" >&2
exit 1
fi
echo "PASS: ${seen} check(s) present, all green; all required checks present + successful."