Commit graph

5 commits

Author SHA1 Message Date
Adam Moussa
76fd73a69f Add CloudWatch alarm for backup verification Lambda errors
Fires on any Lambda error and on missing data (missed schedule).
Catches silent failures where the Slack notification never fires.
2026-05-15 17:31:32 -04:00
Adam Moussa
4d9c30fbb3 Address cross-review findings for backup verification
- Add size guard before downloading dump in restore test (3.5 GB cap)
- Use paginator for list_objects_v2 in S3 checks and restore test
- Remove unnecessary overrideLogicalId on GcsTransferCredentials secret
- Pass explicit { mode: "daily" } to daily EventBridge rule target
- Add fallback for SSM parameter fetch in backup script
- Export replica bucket ARN/name from replica stack, consume via props
2026-05-15 17:25:25 -04:00
Adam Moussa
7aaf8340a5 Add 4 GiB ephemeral storage to verification Lambda
Monthly restore-test downloads and extracts the full dump tarball
in /tmp. As the dump grows with LFS data, the default 512 MB will
eventually cause ENOSPC failures.
2026-05-14 15:03:36 -04:00
Adam Moussa
c771ed4f08 Rename SECRET_ARN env vars to SECRET_NAME to match actual values 2026-05-14 11:38:31 -04:00
Adam Moussa
d57aea19f3 Add 3-2-1 backup strategy with cross-region replication and GCS offsite
Implements a fully compliant 3-2-1 backup architecture:
- Copy 1 (live): Harden existing EBS snapshots to 30-day retention
- Copy 2 (near-site): S3 cross-region replication to us-west-2 with
  Object Lock (governance 90d) and versioning
- Copy 3 (offsite): GCS bucket in dedicated seahaven-backups GCP project
  with 2-year irreversible retention lock

Also adds a verification Lambda that checks all 3 locations daily and
runs monthly restore tests with SQLite integrity checks.
2026-05-13 18:02:50 -04:00