Commit graph

16 commits

Author SHA1 Message Date
Adam Moussa
76fd73a69f Add CloudWatch alarm for backup verification Lambda errors
Fires on any Lambda error and on missing data (missed schedule).
Catches silent failures where the Slack notification never fires.
2026-05-15 17:31:32 -04:00
Adam Moussa
4d9c30fbb3 Address cross-review findings for backup verification
- Add size guard before downloading dump in restore test (3.5 GB cap)
- Use paginator for list_objects_v2 in S3 checks and restore test
- Remove unnecessary overrideLogicalId on GcsTransferCredentials secret
- Pass explicit { mode: "daily" } to daily EventBridge rule target
- Add fallback for SSM parameter fetch in backup script
- Export replica bucket ARN/name from replica stack, consume via props
2026-05-15 17:25:25 -04:00
Adam Moussa
deaae36396 Read backup S3 prefix from SSM parameter at runtime
Adds /forgejo/backup-s3-prefix SSM parameter (value: archive)
and updates the backup script to fetch it instead of hardcoding
the prefix. Eliminates the manual post-deploy sed step.
2026-05-15 17:13:50 -04:00
Adam Moussa
7aaf8340a5 Add 4 GiB ephemeral storage to verification Lambda
Monthly restore-test downloads and extracts the full dump tarball
in /tmp. As the dump grows with LFS data, the default 512 MB will
eventually cause ENOSPC failures.
2026-05-14 15:03:36 -04:00
Adam Moussa
5cae537ee2 Fix restore runbook, DLM snapshot tagging, README cleanup, and gsutil prompt 2026-05-14 13:56:36 -04:00
Adam Moussa
bafb924cfc Fix EBS snapshot state check, drop unused GCS write grant and dead lifecycle rule 2026-05-14 13:24:24 -04:00
Adam Moussa
c771ed4f08 Rename SECRET_ARN env vars to SECRET_NAME to match actual values 2026-05-14 11:38:31 -04:00
Cursor Agent
b9726e5176
Fix backup strategy bug findings 2026-05-13 22:32:26 +00:00
Adam Moussa
fa51085578 Address code review findings for backup verification
Fix 4 critical issues:
- Add filter/priority/deleteMarkerReplication to S3 CRR rule (deploy would fail without)
- Add stack dependency so replica deploys before main stack
- Fix DB file extension matching (.sqlite3/.sql instead of .db)
- Replace nonexistent `forgejo restore` command with actual restore steps in README

Fix 4 moderate issues:
- Add timeout=10 to Slack webhook urlopen call
- Add filter='data' to tarfile.extract for PEP 706 compliance
- Add explicit ValueError for unknown handler mode
- Use date-scoped S3/GCS prefix instead of unbounded listing
2026-05-13 18:29:28 -04:00
Adam Moussa
d57aea19f3 Add 3-2-1 backup strategy with cross-region replication and GCS offsite
Implements a fully compliant 3-2-1 backup architecture:
- Copy 1 (live): Harden existing EBS snapshots to 30-day retention
- Copy 2 (near-site): S3 cross-region replication to us-west-2 with
  Object Lock (governance 90d) and versioning
- Copy 3 (offsite): GCS bucket in dedicated seahaven-backups GCP project
  with 2-year irreversible retention lock

Also adds a verification Lambda that checks all 3 locations daily and
runs monthly restore tests with SQLite integrity checks.
2026-05-13 18:02:50 -04:00
Adam Moussa
ba937f1b83 Add autodiscovery and token refresh crons
Autodiscovery runs hourly — creates Forgejo mirrors for new GitHub
org repos. Token refresh runs daily — propagates the current PAT
from Secrets Manager to all mirror git remotes.
2026-05-11 18:46:25 -04:00
Adam Moussa
2f344ac7c0 Add nightly S3 backups with Glacier lifecycle
Some checks are pending
Deploy / deploy (push) Waiting to run
S3 bucket for Forgejo dumps (Standard 30d → Glacier, expire 365d).
Cron runs forgejo dump at 5 AM UTC and uploads to S3.
2026-05-11 18:26:26 -04:00
Adam Moussa
ed41fd4eac Add ALB egress rule for Forgejo target group
ALB security group has restricted outbound — needed explicit
egress to the Forgejo SG on port 3000 for health checks.
2026-05-11 18:13:21 -04:00
Adam Moussa
9e0a14e5b8 Route through seahaven-com ALB for HTTPS
Add target group, listener rule (forgejo.seahaven.com), and alias
the Route53 record to the ALB. SSL termination via wildcard cert.
2026-05-11 18:08:36 -04:00
Adam Moussa
83c381b1da Fix subnet import to include availability zone
CDK requires availabilityZone when using fromSubnetAttributes
for EC2 instance placement.
2026-05-11 17:58:14 -04:00
Adam Moussa
a500d69716 Add Forgejo CDK stack
EC2 (t4g.small, arm64) in private subnet with VPN-only access,
DLM nightly snapshots, and Route53 DNS at forgejo.seahaven.com.
2026-05-11 17:52:37 -04:00