adam/engineering-handbook
1
0
Fork 0
mirror of https://github.com/Sea-Haven-Industries/engineering-handbook.git synced 2026-09-30 06:53:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(cd): document Option A callers, ship-gate, and ci-complete

Browse source 
Converted HCP repos use org reusables, dual OIDC claims, and a ci-complete ruleset instead of copied target jobs and Mergify.
This commit is contained in:
Adam Moussa 2026-09-22 14:53:02 -04:00
parent 5a33f27a58
commit 245ef2e338
No known key found for this signature in database
3 changed files with 122 additions and 21 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

109
cicd.md
View file

@ -13,7 +13,7 @@ Every deployable repo must have a CI/CD pipeline. No manual deploys to productio
GitHub Actions is the CI/CD platform in both lanes. Infra details for the default lane are in [hcp-terraform.md](hcp-terraform.md).
Do not extract org reusable **deploy** workflows yet. Codify shared composites after a second repo copies the portal shape and the duplication is real. CI may still call org `ci-*` reusables.
HCP app repos call org reusables `cd-hcp-fargate.yaml` and `cd-hcp-spa.yaml` with one caller job per GitHub Environment. Remaining SAM/CDK stacks keep `cd-sam` / `cd-cdk`. Sequential `ci-typescript-frontend.yaml` stays for remaining-lane templates until they migrate.
## HCP Terraform lane (default)
@ -23,28 +23,63 @@ Terraform owns infrastructure and never touches application content. GitHub Acti
One workflow per deployable, kebab-case, in `.github/workflows/`. Examples: `deploy-web.yaml`, `deploy-api.yaml`. A single-deployable repo may use `deploy.yaml`.
| Trigger | Environment | Notes |
Each Environment is its own caller job. `environment` is a `with:` input. The reusable job owns `environment:`, concurrency, OIDC, and `vars.DEPLOY_ROLE_ARN`. GitHub rejects `environment:` beside `uses:`.
| Trigger | Caller job | Notes |
|---|---|---|
| `push` to `main` | `dev` | `paths-ignore` for `terraform/**`, docs, and the other deployable's paths. Mixed app+terraform merges still deploy (GitHub skips only when **every** changed file matches the ignore list). |
| `release: published` | `prod` | Human-cut GitHub Release. See [Releases](#releases). |
| `workflow_dispatch` with `environment` and `ref` | chosen | Redeploy or rollback at any prior tag. |
| `push` to `main` | `deploy-dev` | `paths-ignore` for `terraform/**`, docs, and the other deployable's paths. Mixed app+terraform merges still deploy (GitHub skips only when **every** changed file matches the ignore list). |
| `release: published` | `deploy-prod` (or `deploy-staging`) | Human-cut GitHub Release. `ship-gate: true`. See [Releases](#releases) and [Hotfix ship path](#hotfix-ship-path). |
| `workflow_dispatch` with `environment` and `ref` | matching job | Redeploy or rollback. Empty `ref` means `github.sha`. |
Do **not** put `paths` / `paths-ignore` on tag events. A tag create often has an empty file diff, so the job never starts.
CI stays a separate workflow (`ci.yaml` on `pull_request` to `main`). Org `ci-*` reusables are allowed. The required check is still `ci / ci`.
CI is a separate workflow. Converted repos trigger on `pull_request` to `main`, `hotfix/**`, and `release/**`; `merge_group`; and `push` to `hotfix/**` and `release/**`. No `push` CI on `main`. The required check is `ci-complete`. Unconverted remaining-lane repos still emit `ci / ci`.
### Target job
### Caller shape
A `target` job resolves `environment` and `ref` from the event.
```yaml
jobs:
deploy-dev:
name: Deploy API to dev
if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && inputs.environment == 'dev')
uses: Sea-Haven-Industries/.github/.github/workflows/cd-hcp-fargate.yaml@<sha> # vX.Y.Z
permissions: { contents: read, id-token: write }
secrets: inherit
with:
environment: dev
ref: ${{ inputs.ref }}
ssm-prefix: /<repo>/deploy
docker-platform: linux/amd64
On `release: published` it verifies the tag is an ancestor of `main` via `compare/main...<tag>` (status `behind` or `identical`). Anything else fails closed, so nothing un-reviewed ships.
deploy-prod:
name: Deploy API to prod
if: github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && inputs.environment == 'prod')
uses: Sea-Haven-Industries/.github/.github/workflows/cd-hcp-fargate.yaml@<sha> # vX.Y.Z
permissions: { contents: read, id-token: write }
secrets: inherit
with:
environment: prod
ref: ${{ github.event.release.tag_name || inputs.ref }}
ssm-prefix: /<repo>/deploy
docker-platform: linux/amd64
ship-gate: true
```
Environment-derived values (Sentry environment, stage) come from that output. Do not store them as per-environment variables that can silently be unset.
SPA callers use `cd-hcp-spa.yaml`. Add `deploy-staging` only where that Environment exists.
### Deploy job
### Ship-gate
- `environment: ${{ needs.target.outputs.environment }}`
- `concurrency: deploy-<name>-<env>` with `cancel-in-progress: false`
`ship-gate: true` on prod and staging replaces the copied `target` job. It is legal when:
1. `compare/main...<tag>` is `behind` or `identical`, or
2. The tag is a fast-forward of the previous matching Release tag, SemVer matches that Environment (`^v[0-9]+\.[0-9]+\.[0-9]+$` or `-staging`), and it was created from `hotfix/*` or `release/*`.
Anything else fails closed, so nothing un-reviewed ships. Environment-derived values (Sentry environment, stage) come from `inputs.environment`. Do not store them as per-environment variables that can silently be unset.
### Deploy job (inside the reusable)
- `environment: ${{ inputs.environment }}`
- `concurrency: deploy-${{ inputs.ssm-prefix }}-${{ inputs.environment }}` with `cancel-in-progress: false`
- Permissions: `id-token: write`, `contents: read`
- Assumes the GitHub Environment variable `DEPLOY_ROLE_ARN`
@ -52,11 +87,13 @@ Environment-derived values (Sentry environment, stage) come from that output. Do
**SPA / CloudFront.** Build, `s3 sync` hashed assets with immutable cache headers, `cp` `index.html` last with `no-store`, `s3 sync --delete` to prune, `CreateInvalidation /*`. Then verify live: distribution `Deployed`, served `index.html` hash equals the built hash, cache headers, hashed assets, forbidden URLs, and `/api/health` polled for up to five minutes when the SPA depends on an API.
**Fargate.** Docker build+push tagged `$sha` and `$environment`, patch `GIT_SHA` (and optional `extra-task-env`) on the task definition, `RegisterTaskDefinition` + `UpdateService` + `services-stable`, then poll SSM `api-url` until `/api/health` reports that SHA.
**Lambda zip.** esbuild (or equivalent) with `GIT_SHA` inlined at build time. The build constant wins over any runtime env var. Upload `functions/<name>/<sha>.zip` to the artifacts bucket, `update-function-code` on each function, then poll `/api/health` until it reports that exact SHA.
Verify against live state, not action success. Every check that depends on another deployable must poll, not probe once. Parallel deployables race each other on a fresh environment.
Reference: `internal-portal` `deploy-web.yaml` and `deploy-api.yaml`.
Reference: org reusables `cd-hcp-spa.yaml` and `cd-hcp-fargate.yaml`. Inline copies remain on unconverted callers until their cutover.
### Releases
@ -70,6 +107,21 @@ Not a workflow. Releases created with `GITHUB_TOKEN` do not fire `release: publi
One tag drives both prod infra and prod app. Approving the app deploys after the HCP apply lands is the operator's sequencing responsibility. SPA origin-path guards and health polls catch the common misorderings.
### Hotfix ship path
HCP, Environment `v*`, and OIDC tag refs already allow a hotfix tag that is not yet on `main`. No IAM change for this path. `ship-gate: true` accepts it when the tag is a fast-forward of the previous matching Release, SemVer matches the Environment, and it was created from `hotfix/*` or `release/*`.
```bash
git fetch --tags
git checkout -b hotfix/describe-the-break v1.2.3
# commit, push (CI on hotfix/**) or PR targeting the hotfix branch
gh release create v1.2.4 --target hotfix/describe-the-break --generate-notes
# approve prod Environment after the HCP prod apply
# merge hotfix/describe-the-break into main
```
Do not add GitFlow release trains, cherry-pick bots, or auto merge-back. Merge the hotfix branch into `main` after prod.
### GitHub Environments
| Environment | Reviewers | Deployment branch policy | Variables |
@ -85,7 +137,12 @@ Staging is not a default Environment. See [hcp-terraform.md](hcp-terraform.md#ac
IAM role `githubdeploy-<repo>`, owned by workload Terraform.
**Trust.** OIDC `sub` pinned to `repo:Sea-Haven-Industries/<repo>:environment:<env>` plus `job_workflow_ref` for each deploy workflow at `refs/heads/main`. Adding a workflow means adding its ref here. That is a cross-family review change.
**Trust.** OIDC `sub` stays `repo:Sea-Haven-Industries/<repo>:environment:<env>`. SHA-pinned org reusables change `job_workflow_ref` to the reusable file:
- `job_workflow_ref`: `Sea-Haven-Industries/.github/.github/workflows/cd-hcp-fargate.yaml@*` (and `cd-hcp-spa.yaml`)
- `workflow_ref`: the thin caller still `Sea-Haven-Industries/<repo>/.github/workflows/deploy-api.yaml@refs/heads/main` and `@refs/tags/v*`
Adding a reusable is a cross-family IAM change. Adding an app still adds `workflow_ref` for that caller. Do not pin trust to only `ref:refs/heads/main`.
**Permissions.** Only what the workflows write:
@ -94,14 +151,28 @@ IAM role `githubdeploy-<repo>`, owned by workload Terraform.
- `cloudfront:CreateInvalidation` and `GetDistribution` on the one distribution
- `lambda:UpdateFunctionCode` and `GetFunction` on the named functions
- `ssm:GetParameter` on `/<repo>/deploy/*`
- Fargate callers also need ECR push, `ecs:RegisterTaskDefinition` / `UpdateService` / `Describe*`, and `iam:PassRole` on the task roles
Nothing else.
`DEPLOY_ROLE_ARN` is a GitHub Environment **variable**, not a repo secret.
### Converted CI
Converted HCP callers use parallel portions plus a `ci-complete` aggregator. Autofix is a pull-request convenience that commits with a GitHub App token. `format:check` / lint in the portions stay the fail-closed gate.
- Autofix runs on `pull_request` only, skips forks, and skips when the actor is the App. If the tree is dirty it commits `style: apply formatter` and sets `committed=true` so this SHA skips build/test. The `synchronize` run must be green. Do not `--no-verify`. Do not push to `main`.
- `merge_group` and `push` skip autofix (`result == skipped`) so those paths still run portions.
- `always()` on later jobs is required so an autofix failure does not skip `static`.
- Org secrets: `AUTOFMT_APP_ID`, `AUTOFMT_APP_PRIVATE_KEY`. The App has `contents: write` and `metadata: read`. It is not on the main-branch ruleset bypass list.
The org ruleset **CI complete** requires the check-run name `ci-complete`. Do not put portion names (`frontend / static`, `unit (1)`, …) in a ruleset. Unconverted repos stay on **main branch protection** requiring `ci / ci`. A repo is on exactly one of those rulesets. Flip membership in the same window as the workflow merge. Do not remove or retarget native GitHub merge-queue rulesets.
Python callers pass `format-command: ruff format .` and `lint-fix-command: ruff check --fix .`. Frontend passes npm scripts. Do not run `eslint --fix` unless that repo's `lint` script is already fix-safe.
## SAM / CDK lane (remaining)
Existing SAM and CDK stacks keep thin callers of org reusables until they migrate. Do not start a new deployable on this lane.
Existing SAM and CDK stacks keep thin callers of org reusables until they migrate. Do not start a new deployable on this lane. Sequential `ci-typescript-frontend.yaml` remains for remaining-lane SPA templates until those repos migrate onto `ci-frontend.yaml` plus `ci-complete`.
Every remaining repo still has:
@ -203,15 +274,15 @@ CI runs produce no external side effects, so when a newer commit supersedes an o
### HCP lane
Declare concurrency on the deploy job:
Declare concurrency inside the reusable deploy job:
```yaml
concurrency:
group: deploy-<name>-${{ needs.target.outputs.environment }}
group: deploy-${{ inputs.ssm-prefix }}-${{ inputs.environment }}
cancel-in-progress: false
```
`<name>` is the deployable (`web`, `api`, …). Independent deployables must not share a group.
Independent deployables must not share an `ssm-prefix`. Unconverted inline callers still key `deploy-<name>-<env>` until they move.
### Remaining SAM / CDK reusables

17
git-workflow.md
View file

@ -55,7 +55,9 @@ Transition names are case-insensitive and match the issue's workflow (`#in-progr
Nobody deploys from a workstation. The sanctioned paths depend on the lane. See [cicd.md](cicd.md).
**HCP Terraform (default).** Merge to `main` deploys **dev**: HCP auto-applies if `terraform/**` changed, and GitHub Actions deploys the app unless the merge was terraform-only. **Prod** is a human GitHub Release (`gh release create vX.Y.Z --target main --generate-notes`). That tag applies prod infra and queues the prod app deploys behind Environment reviewers. Rollback is `workflow_dispatch` of the deploy workflow at a prior tag, not a Terraform revert of application content.
**HCP Terraform (default).** Merge to `main` deploys **dev**: HCP auto-applies if `terraform/**` changed, and GitHub Actions deploys the app unless the merge was terraform-only. **Prod** is a human GitHub Release (`gh release create vX.Y.Z --target main --generate-notes`). That tag applies prod infra and queues the prod app deploys behind Environment reviewers and `ship-gate`. Rollback is `workflow_dispatch` of the deploy workflow at a prior tag, not a Terraform revert of application content.
A production break that cannot wait for `main` uses a hotfix branch cut from the last prod tag. CI runs on `hotfix/**` and `release/**`. Cut the next patch Release from that branch, approve prod after the HCP apply, then merge the hotfix branch into `main`. See [cicd.md](cicd.md#hotfix-ship-path). Do not add GitFlow trains or auto merge-back.
**SAM / CDK (remaining).** Merge to `main` (or `workflow_dispatch` of that same pipeline) is still the only path.
@ -70,6 +72,17 @@ The standard flow for an HCP app repo:
7. Cut prod with `gh release create`. Approve the prod Environment gate after the HCP prod apply lands. Verify live state, not action success.
8. If prod is wrong, dispatch the deploy workflow at the previous tag and approve the gate.
Hotfix when prod is already broken:
```bash
git fetch --tags
git checkout -b hotfix/describe-the-break v1.2.3
# commit, push (CI on hotfix/**) or PR targeting the hotfix branch
gh release create v1.2.4 --target hotfix/describe-the-break --generate-notes
# approve prod Environment after the HCP prod apply
# merge hotfix/describe-the-break into main
```
A local deploy puts code into an environment that no reviewed commit describes, and its result depends on whoever ran it having the right credentials and a clean working tree. The pipeline deploys a known commit with the repo's own OIDC role every time.
### Legacy exception: deploy-then-merge
@ -118,6 +131,8 @@ For HCP app repos, cut a GitHub Release from `main` after the PR is merged. Do n
gh release create v1.2.0 --target main --generate-notes
```
A hotfix Release targets the hotfix branch instead of `main`. See [cicd.md](cicd.md#hotfix-ship-path).
For packages and libraries that are not HCP app repos, annotated tags remain fine:
```bash

17
github-standards.md
View file

@ -133,6 +133,8 @@ HCP app repos use Environments as the deploy gate. See [cicd.md](cicd.md#github-
`DEPLOY_ROLE_ARN` is an Environment variable, not a repo secret. Remaining SAM/CDK repos may still use `AWS_DEPLOY_ROLE_ARN` as a repo secret until they migrate.
Converted HCP CD callers pin OIDC with both claims: `job_workflow_ref` on the org reusable (`cd-hcp-fargate.yaml` / `cd-hcp-spa.yaml`) and `workflow_ref` on the thin repo caller at `refs/heads/main` and `refs/tags/v*`. `sub` stays `repo:.../<app>:environment:<env>`. Details are in [cicd.md](cicd.md#github-deploy-role).
## README Badges
Every repo's README carries a small badge block immediately under the H1. Use **static** badges only — dynamic badges (for example shields.io `last-commit` or `open-issues`) query the public GitHub API and render broken on private repos.
@ -153,7 +155,20 @@ Set them with `gh repo edit <repo> --add-topic a,b,c`. Adding topics is part of
## Required CI Status Check
The org ruleset requires the status check named **`ci / ci`** — the job name `ci` under the workflow named `CI`. This is what the reusable `ci-typescript-cdk.yaml` and `ci-python-sam.yaml` callers emit. If the check name in the ruleset does not match what CI actually emits, merges will be blocked by a phantom required check. Verify after any change to CI job names.
Two org rulesets. A repo is on exactly one of them. Never both.
| Ruleset | Required check | Who |
|---|---|---|
| **main branch protection** | `ci / ci` | Unconverted remaining-lane repos |
| **CI complete** | `ci-complete` | Converted HCP callers |
`CI complete` starts with no repos. Flip include/unexclude in the same window as the workflow merge that lands `name: ci-complete`. Do not put portion names (`frontend / static`, `unit (1)`, `browser-smoke`) in a ruleset.
If the check name in the ruleset does not match what CI actually emits, merges will be blocked by a phantom required check. Verify after any change to CI job names.
Mergify YAML is not used. Converted CI keeps a `merge_group` trigger so native GitHub merge queues still run. Do not treat a missing Mergify config as a gap. Do not remove or retarget native GitHub merge-queue rulesets when flipping CI membership.
The formatter GitHub App is not on the main-branch ruleset bypass list.
## Repo Hygiene