Add 3-2-1 backup strategy (#2)

* Add 3-2-1 backup strategy with cross-region replication and GCS offsite

Implements a fully compliant 3-2-1 backup architecture:
- Copy 1 (live): Harden existing EBS snapshots to 30-day retention
- Copy 2 (near-site): S3 cross-region replication to us-west-2 with
  Object Lock (governance 90d) and versioning
- Copy 3 (offsite): GCS bucket in dedicated seahaven-backups GCP project
  with 2-year irreversible retention lock

Also adds a verification Lambda that checks all 3 locations daily and
runs monthly restore tests with SQLite integrity checks.

* Enable QEMU in CI for arm64 Lambda Docker builds

* Commit cdk.context.json for CI synth without AWS credentials

Vpc.fromLookup requires cached context to synthesize without
AWS credentials. Required for CI which runs cdk synth without
an OIDC role.

* Fix GCP project ID to sea-haven-backups

* Address code review findings for backup verification

Fix 4 critical issues:
- Add filter/priority/deleteMarkerReplication to S3 CRR rule (deploy would fail without)
- Add stack dependency so replica deploys before main stack
- Fix DB file extension matching (.sqlite3/.sql instead of .db)
- Replace nonexistent `forgejo restore` command with actual restore steps in README

Fix 4 moderate issues:
- Add timeout=10 to Slack webhook urlopen call
- Add filter='data' to tarfile.extract for PEP 706 compliance
- Add explicit ValueError for unknown handler mode
- Use date-scoped S3/GCS prefix instead of unbounded listing

* Fix backup strategy bug findings

* Handle SQL text dumps separately from binary SQLite in restore test

Forgejo dump produces gitea-db.sql as a text SQL dump (XORM export),
not a binary SQLite file. Opening it directly with sqlite3.connect()
throws DatabaseError. Now imports the SQL dump into a temp DB first.

* Fix GCS backup check: align staleness cutoff and add size validation

GCS check used a 72h cutoff but only listed 2 days of prefixes (~48h),
making the staleness check unreachable. Also added 1MB minimum file
size validation to match the S3 check.

* Rename SECRET_ARN env vars to SECRET_NAME to match actual values

* Fix EBS snapshot state check, drop unused GCS write grant and dead lifecycle rule

* Fix restore runbook, DLM snapshot tagging, README cleanup, and gsutil prompt

* Fix restore runbook: trailing-dot cp idiom and Glacier restore step

* Rename GCS service account to match read-only permissions

* Add 4 GiB ephemeral storage to verification Lambda

Monthly restore-test downloads and extracts the full dump tarball
in /tmp. As the dump grows with LFS data, the default 512 MB will
eventually cause ENOSPC failures.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

.github/workflows/ci.yaml

@@ -6,3 +6,5 @@ on:
 jobs:
   ci:
     uses: Sea-Haven-Industries/.github/.github/workflows/ci-typescript-cdk.yaml@main
+    with:
+      enable-qemu: true

.github/workflows/deploy.yaml

@@ -14,5 +14,7 @@ concurrency:
 jobs:
   deploy:
     uses: Sea-Haven-Industries/.github/.github/workflows/cd-cdk.yaml@main
+    with:
+      enable-qemu: true
     secrets:
       deploy-role-arn: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}

.gitignore

@@ -3,4 +3,4 @@ cdk.out/
 *.js
 *.d.ts
 *.js.map
-cdk.context.json
+docs/*.pdf

README.md

@@ -8,7 +8,7 @@ Self-hosted Forgejo git server for archiving GitHub repos and mirroring active o
 - **Network**: Private subnet (us-east-1a), behind `seahaven-com` ALB for SSL termination
 - **DNS**: `forgejo.seahaven.com` — Route53 alias record pointing to the `seahaven-com` ALB (not a direct A record)
 - **TLS**: Wildcard cert on ALB, HTTP internally on port 3000
-- **Backup**: Nightly `forgejo dump` to S3 + EBS snapshots via DLM (see [S3 Backups](#s3-backups))
+- **Backup**: Nightly `forgejo dump` to S3 + EBS snapshots via DLM (see [3-2-1 Backup Strategy](#3-2-1-backup-strategy))
 - **Admin access**: SSM Session Manager (no SSH port exposed)
 - **CI/CD**: GitHub Actions with OIDC role `githubdeploy-forgejo`
 
@@ -20,26 +20,87 @@ Self-hosted Forgejo git server for archiving GitHub repos and mirroring active o
 | 3000 | HTTP | ALB → instance | Internal traffic from ALB |
 | 2222 | SSH | VPC + VPN | Git SSH operations |
 
-## S3 Backups
+## 3-2-1 Backup Strategy
 
-A nightly `forgejo dump` runs at 5:00 UTC and uploads the archive to `s3://forgejo-backups-328440206208`.
+All backups follow a 3-2-1 strategy: 3 copies, 2 storage types, 1 offsite provider.
 
-**S3 lifecycle policy:**
+| Copy | Location | Type | Retention |
+|------|----------|------|-----------|
+| Live | EBS volume (us-east-1) | Block | N/A |
+| Near-site | S3 replica (us-west-2) | Object | Archive: indefinite, noncurrent versions: 90d |
+| Offsite | GCS `forgejo-backups-offsite-seahaven` (GCP us-central1) | Object | 2-year locked retention |
 
-| Phase | Duration |
+**Daily data flow:**
-|-------|----------|
-| Standard | First 30 days |
-| Glacier | Days 31–365 |
-| Expired | After 365 days |
 
-EBS snapshots are managed separately by DLM and run nightly at 6:00 UTC with a 7-day retention window.
+| Time (UTC) | Event |
+|------------|-------|
+| 05:00 | `forgejo dump` → `s3://forgejo-backups-328440206208/archive/{date}/` |
+| ~05:01 | S3 CRR replicates to `forgejo-backups-replica-328440206208` (us-west-2) |
+| 06:00 | DLM EBS snapshot (30-day retention) |
+| 08:00 | Verification Lambda checks all 3 locations, posts to Slack |
+| 10:00 | GCS Storage Transfer pulls from S3 to GCS offsite |
 
-To test the backup manually:
+**S3 source lifecycle:** Standard 30d → Glacier (no expiration).
+
+**Immutability layers:**
+- S3 Versioning on both source and replica buckets
+- S3 Object Lock (Governance, 90d) on the replica bucket
+- GCS Bucket Lock (2yr, irreversible) on the offsite bucket
+
+### Verification
+
+The `forgejo-backup-verification` Lambda runs daily at 08:00 UTC and checks:
+1. S3 source has a recent dump under `archive/`
+2. S3 replica has replicated the latest dump
+3. GCS offsite has received the latest transfer
+4. EBS snapshots exist within the last 48 hours
+
+On the 1st of each month at 09:00 UTC, it runs a restore test: downloads the latest dump, extracts the archive, and runs SQLite integrity checks.
+
+### Manual backup
 
 ```bash
 sudo /usr/local/bin/forgejo-backup.sh
 ```
 
+### Restore from S3
+
+For backups older than 30 days (Glacier), restore the object first:
+
+```bash
+aws s3api restore-object --bucket forgejo-backups-328440206208 \
+  --key "archive/<date>/forgejo-<date>.tar.gz" \
+  --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Standard"}}'
+# Wait ~3-5 hours for restore to complete, then:
+```
+
+Download and restore:
+
+```bash
+aws s3 cp s3://forgejo-backups-328440206208/archive/<date>/forgejo-<date>.tar.gz /tmp/
+systemctl stop forgejo
+mkdir -p /tmp/forgejo-restore && tar -xzf /tmp/forgejo-<date>.tar.gz -C /tmp/forgejo-restore
+cd /tmp/forgejo-restore
+cp app.ini /etc/forgejo/app.ini
+cp gitea-db.sqlite3 /var/lib/forgejo/data/forgejo.db
+rm -rf /var/lib/forgejo/data/repositories
+cp -a repos /var/lib/forgejo/data/repositories
+cp -a data/. /var/lib/forgejo/data/
+[ -d lfs ] && cp -a lfs/. /var/lib/forgejo/data/lfs/
+[ -d custom ] && cp -a custom/. /var/lib/forgejo/custom/
+chown -R forgejo:forgejo /var/lib/forgejo /etc/forgejo/app.ini
+systemctl start forgejo
+rm -rf /tmp/forgejo-restore /tmp/forgejo-<date>.tar.gz
+```
+
+### Restore from GCS (disaster recovery)
+
+```bash
+gcloud config set project sea-haven-backups
+gsutil cp gs://forgejo-backups-offsite-seahaven/archive/<date>/forgejo-<date>.tar.gz /tmp/
+# Then follow the same restore steps as S3 above
+```
+
 ## Autodiscovery
 
 An hourly cron job checks the `Sea-Haven-Industries` GitHub org for new repositories and mirrors them into Forgejo automatically.
@@ -78,6 +139,9 @@ sudo /usr/local/bin/forgejo-refresh-tokens.sh
 | `forgejo/admin-password` | Forgejo admin user password |
 | `forgejo/api-token` | Forgejo API token (used by autodiscovery and token refresh scripts) |
 | `forgejo/github-pat` | GitHub fine-grained PAT for mirroring |
+| `forgejo/gcs-sa-key` | GCP service account key for offsite backup verification |
+| `forgejo/gcs-transfer-credentials` | AWS IAM credentials for GCS Storage Transfer Service |
+| `forgejo/slack-webhook` | Slack webhook URL for backup verification alerts |
 
 ## First-time setup
 
@@ -108,15 +172,46 @@ In the Forgejo web UI: **New Migration → GitHub** → paste the GitHub repo UR
 
 Same migration flow, but check **This Repository Will Be A Mirror**. Forgejo polls GitHub hourly (`DEFAULT_INTERVAL = 1h` in app.ini) and keeps the mirror in sync.
 
+## GCP Offsite Setup (one-time)
+
+Run the setup script to create the GCS offsite bucket, service account, and store credentials:
+
+```bash
+./scripts/gcp-setup.sh
+```
+
+This creates the `sea-haven-backups` GCP project with a locked-retention GCS bucket. After running, configure the Storage Transfer job in the GCP Console using the AWS credentials from `forgejo/gcs-transfer-credentials`.
+
 ## Deployment
 
 ```bash
 npm install
-npx cdk deploy
+npx cdk deploy --all
 ```
 
+This deploys two stacks:
+- `forgejo-replica` (us-west-2) — S3 replica bucket with Object Lock
+- `forgejo` (us-east-1) — main stack with Forgejo instance, CRR, and verification Lambda
+
 CI/CD is handled by GitHub Actions — PRs run CI, merges to `main` deploy via the reusable CDK workflow.
 
+## Post-deploy: update running instance backup path
+
+After the first deploy with the 3-2-1 changes, the running instance's backup script still uses the old S3 path (without the `archive/` prefix). Update it via SSM:
+
+```bash
+aws ssm start-session --target i-0d3005fb3c36124cd
+sudo sed -i 's|s3://forgejo-backups-328440206208/${TIMESTAMP}/|s3://forgejo-backups-328440206208/archive/${TIMESTAMP}/|' /usr/local/bin/forgejo-backup.sh
+```
+
+Also store the Slack webhook URL for backup verification alerts:
+
+```bash
+aws secretsmanager create-secret --name forgejo/slack-webhook \
+  --secret-string "https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \
+  --region us-east-1
+```
+
 ## Updating Forgejo
 
 Update the `FORGEJO_VERSION` constant in `lib/forgejo-stack.ts` and deploy. This replaces the instance, so ensure the latest EBS snapshot is available for data recovery if needed. Alternatively, update in-place via SSM:

bin/app.ts

@@ -2,9 +2,18 @@
 import "source-map-support/register";
 import * as cdk from "aws-cdk-lib";
 import { ForgejoStack } from "../lib/forgejo-stack";
+import { ForgejoReplicaStack } from "../lib/forgejo-replica-stack";
 
 const app = new cdk.App();
-new ForgejoStack(app, "forgejo", {
+
+const replicaStack = new ForgejoReplicaStack(app, "forgejo-replica", {
+  stackName: "forgejo-replica",
+  env: { account: "328440206208", region: "us-west-2" },
+});
+
+const forgejoStack = new ForgejoStack(app, "forgejo", {
   stackName: "forgejo",
   env: { account: "328440206208", region: "us-east-1" },
 });
+
+forgejoStack.addDependency(replicaStack);

cdk.context.json

@@ -0,0 +1,47 @@
+{
+  "vpc-provider:account=328440206208:filter.vpc-id=vpc-0d3d4b67bd0cf8a68:region=us-east-1:returnAsymmetricSubnets=true": {
+    "vpcId": "vpc-0d3d4b67bd0cf8a68",
+    "vpcCidrBlock": "10.20.0.0/16",
+    "ownerAccountId": "328440206208",
+    "availabilityZones": [],
+    "vpnGatewayId": "vgw-073737d44762dffc2",
+    "subnetGroups": [
+      {
+        "name": "Private",
+        "type": "Private",
+        "subnets": [
+          {
+            "subnetId": "subnet-04e38c507e96f1926",
+            "cidr": "10.20.30.0/24",
+            "availabilityZone": "us-east-1a",
+            "routeTableId": "rtb-06a2f56f492b9b4de"
+          },
+          {
+            "subnetId": "subnet-0a0b4fc6f296dfba5",
+            "cidr": "10.20.40.0/24",
+            "availabilityZone": "us-east-1b",
+            "routeTableId": "rtb-01e152fe5cabca7d6"
+          }
+        ]
+      },
+      {
+        "name": "Public",
+        "type": "Public",
+        "subnets": [
+          {
+            "subnetId": "subnet-0eea820effe1b3ae5",
+            "cidr": "10.20.10.0/24",
+            "availabilityZone": "us-east-1a",
+            "routeTableId": "rtb-0f2232493a5c43fe8"
+          },
+          {
+            "subnetId": "subnet-0012f5895182c1580",
+            "cidr": "10.20.20.0/24",
+            "availabilityZone": "us-east-1b",
+            "routeTableId": "rtb-0f2232493a5c43fe8"
+          }
+        ]
+      }
+    ]
+  }
+}

lambda/backup-verification/app.py

@@ -0,0 +1,246 @@
+import json
+import os
+import tarfile
+import tempfile
+import urllib.request
+from datetime import datetime, timedelta, timezone
+
+import boto3
+from google.cloud import storage as gcs
+from google.oauth2 import service_account
+
+
+s3 = boto3.client("s3")
+s3_west = boto3.client("s3", region_name="us-west-2")
+ec2 = boto3.client("ec2")
+secrets = boto3.client("secretsmanager")
+
+SOURCE_BUCKET = os.environ["SOURCE_BUCKET"]
+REPLICA_BUCKET = os.environ["REPLICA_BUCKET"]
+GCS_BUCKET = os.environ["GCS_BUCKET"]
+GCS_SA_SECRET_NAME = os.environ["GCS_SA_SECRET_NAME"]
+SLACK_WEBHOOK_SECRET_NAME = os.environ["SLACK_WEBHOOK_SECRET_NAME"]
+
+_gcs_client = None
+
+
+def _get_gcs_client():
+    global _gcs_client
+    if _gcs_client is None:
+        raw = secrets.get_secret_value(SecretId=GCS_SA_SECRET_NAME)["SecretString"]
+        info = json.loads(raw)
+        creds = service_account.Credentials.from_service_account_info(info)
+        _gcs_client = gcs.Client(credentials=creds, project=info.get("project_id"))
+    return _gcs_client
+
+
+def _check_s3_bucket(client, bucket, label):
+    now = datetime.now(timezone.utc)
+    cutoff = now - timedelta(hours=48)
+    try:
+        today = now.strftime("%Y-%m-%d")
+        yesterday = (now - timedelta(days=1)).strftime("%Y-%m-%d")
+        contents = []
+        for date_prefix in [today, yesterday]:
+            resp = client.list_objects_v2(Bucket=bucket, Prefix=f"archive/{date_prefix}/")
+            contents.extend(resp.get("Contents", []))
+        if not contents:
+            return False, f"{label}: No objects found under archive/ for last 2 days"
+        latest = max(contents, key=lambda o: o["LastModified"])
+        if latest["LastModified"] < cutoff:
+            age = (now - latest["LastModified"]).total_seconds() / 3600
+            return False, f"{label}: Latest dump is {age:.0f}h old ({latest['Key']})"
+        if latest["Size"] < 1_000_000:
+            return False, f"{label}: Latest dump suspiciously small ({latest['Size']} bytes)"
+        return True, f"{label}: OK — {latest['Key']} ({latest['Size'] / 1_000_000:.1f} MB)"
+    except Exception as e:
+        return False, f"{label}: Error — {e}"
+
+
+def _check_gcs():
+    try:
+        client = _get_gcs_client()
+        bucket = client.bucket(GCS_BUCKET)
+        now = datetime.now(timezone.utc)
+        today = now.strftime("%Y-%m-%d")
+        yesterday = (now - timedelta(days=1)).strftime("%Y-%m-%d")
+        blobs = []
+        for date_prefix in [today, yesterday]:
+            blobs.extend(list(bucket.list_blobs(prefix=f"archive/{date_prefix}/")))
+        if not blobs:
+            return False, "GCS Offsite: No objects found under archive/ for last 2 days"
+        cutoff = now - timedelta(hours=48)
+        latest = max(blobs, key=lambda b: b.updated)
+        if latest.updated < cutoff:
+            age = (now - latest.updated).total_seconds() / 3600
+            return False, f"GCS Offsite: Latest object is {age:.0f}h old ({latest.name})"
+        if latest.size < 1_000_000:
+            return False, f"GCS Offsite: Latest dump suspiciously small ({latest.size} bytes)"
+        return True, f"GCS Offsite: OK — {latest.name} ({latest.size / 1_000_000:.1f} MB)"
+    except Exception as e:
+        return False, f"GCS Offsite: Error — {e}"
+
+
+def _check_ebs_snapshots():
+    try:
+        now = datetime.now(timezone.utc)
+        cutoff = now - timedelta(hours=48)
+        resp = ec2.describe_snapshots(
+            Filters=[{"Name": "tag:forgejo-backup", "Values": ["true"]}],
+            OwnerIds=["self"],
+        )
+        snapshots = resp.get("Snapshots", [])
+        if not snapshots:
+            return False, "EBS Snapshots: No snapshots found with forgejo-backup tag"
+        recent = [s for s in snapshots if s["StartTime"] >= cutoff and s.get("State") == "completed"]
+        if not recent:
+            pending = sum(1 for s in snapshots if s["StartTime"] >= cutoff and s.get("State") == "pending")
+            errored = sum(1 for s in snapshots if s["StartTime"] >= cutoff and s.get("State") == "error")
+            latest = max(snapshots, key=lambda s: s["StartTime"])
+            age = (now - latest["StartTime"]).total_seconds() / 3600
+            return False, (
+                f"EBS Snapshots: No completed snapshot in last 48h "
+                f"(latest {age:.0f}h old, state={latest.get('State')}; "
+                f"pending={pending}, error={errored})"
+            )
+        return True, f"EBS Snapshots: OK — {len(recent)} completed in last 48h"
+    except Exception as e:
+        return False, f"EBS Snapshots: Error — {e}"
+
+
+def _restore_test():
+    results = []
+    try:
+        now = datetime.now(timezone.utc)
+        contents = []
+        for days_ago in range(7):
+            date_prefix = (now - timedelta(days=days_ago)).strftime("%Y-%m-%d")
+            resp = s3.list_objects_v2(Bucket=SOURCE_BUCKET, Prefix=f"archive/{date_prefix}/")
+            contents.extend(resp.get("Contents", []))
+        if not contents:
+            return [{"pass": False, "msg": "Restore test: No dumps found in source bucket (last 7 days)"}]
+        latest = max(contents, key=lambda o: o["LastModified"])
+        results.append({"pass": True, "msg": f"Restore test: Using {latest['Key']} ({latest['Size'] / 1_000_000:.1f} MB)"})
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            local_path = os.path.join(tmpdir, "dump.tar.gz")
+            s3.download_file(SOURCE_BUCKET, latest["Key"], local_path)
+            results.append({"pass": True, "msg": "Restore test: Download OK"})
+
+            try:
+                with tarfile.open(local_path, "r:gz") as tf:
+                    names = tf.getnames()
+                    results.append({"pass": True, "msg": f"Restore test: Archive OK — {len(names)} entries"})
+
+                    sqlite_entries = [n for n in names if n.endswith(".sqlite3")]
+                    sql_entries = [n for n in names if n.endswith(".sql")]
+                    if sqlite_entries:
+                        import sqlite3 as sqlite_mod
+                        tf.extract(sqlite_entries[0], path=tmpdir, filter="data")
+                        db_path = os.path.join(tmpdir, sqlite_entries[0])
+                        conn = sqlite_mod.connect(db_path)
+                        result = conn.execute("PRAGMA integrity_check").fetchone()
+                        conn.close()
+                        if result[0] == "ok":
+                            results.append({"pass": True, "msg": "Restore test: SQLite integrity OK"})
+                        else:
+                            results.append({"pass": False, "msg": f"Restore test: SQLite integrity FAILED — {result[0]}"})
+                    elif sql_entries:
+                        import sqlite3 as sqlite_mod
+                        tf.extract(sql_entries[0], path=tmpdir, filter="data")
+                        sql_path = os.path.join(tmpdir, sql_entries[0])
+                        with open(sql_path, "r") as f:
+                            sql_text = f.read()
+                        if len(sql_text) < 100:
+                            results.append({"pass": False, "msg": f"Restore test: SQL dump suspiciously small ({len(sql_text)} bytes)"})
+                        else:
+                            db_path = os.path.join(tmpdir, "restore-test.db")
+                            conn = sqlite_mod.connect(db_path)
+                            conn.executescript(sql_text)
+                            result = conn.execute("PRAGMA integrity_check").fetchone()
+                            conn.close()
+                            if result[0] == "ok":
+                                results.append({"pass": True, "msg": "Restore test: SQL dump import + integrity OK"})
+                            else:
+                                results.append({"pass": False, "msg": f"Restore test: Integrity FAILED after SQL import — {result[0]}"})
+                    else:
+                        results.append({"pass": False, "msg": "Restore test: No database file found in archive"})
+            except tarfile.TarError as e:
+                results.append({"pass": False, "msg": f"Restore test: Archive extraction FAILED — {e}"})
+    except Exception as e:
+        results.append({"pass": False, "msg": f"Restore test: Error — {e}"})
+    return results
+
+
+def _post_slack(blocks):
+    raw = secrets.get_secret_value(SecretId=SLACK_WEBHOOK_SECRET_NAME)["SecretString"]
+    webhook_url = raw.strip()
+    payload = json.dumps({"blocks": blocks}).encode()
+    req = urllib.request.Request(
+        webhook_url,
+        data=payload,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    urllib.request.urlopen(req, timeout=10)
+
+
+def handler(event, context):
+    mode = event.get("mode", "daily")
+    results = []
+
+    if mode == "daily":
+        results.append(_check_s3_bucket(s3, SOURCE_BUCKET, "S3 Source (us-east-1)"))
+        results.append(_check_s3_bucket(s3_west, REPLICA_BUCKET, "S3 Replica (us-west-2)"))
+        results.append(_check_gcs())
+        results.append(_check_ebs_snapshots())
+
+        all_pass = all(r[0] for r in results)
+        header = "Forgejo Backup Verification"
+        blocks = [
+            {"type": "header", "text": {"type": "plain_text", "text": header}},
+            {"type": "section", "text": {"type": "mrkdwn", "text": f"*Date:* {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}"}},
+            {"type": "divider"},
+        ]
+        for passed, msg in results:
+            emoji = ":white_check_mark:" if passed else ":x:"
+            blocks.append({"type": "section", "text": {"type": "mrkdwn", "text": f"{emoji} {msg}"}})
+        blocks.append({"type": "divider"})
+        overall = ":white_check_mark: All checks passed" if all_pass else ":rotating_light: One or more checks failed"
+        blocks.append({"type": "section", "text": {"type": "mrkdwn", "text": f"*Overall:* {overall}"}})
+
+    elif mode == "restore-test":
+        test_results = _restore_test()
+        all_pass = all(r["pass"] for r in test_results)
+        header = "Forgejo Monthly Restore Test"
+        blocks = [
+            {"type": "header", "text": {"type": "plain_text", "text": header}},
+            {"type": "section", "text": {"type": "mrkdwn", "text": f"*Date:* {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}"}},
+            {"type": "divider"},
+        ]
+        for r in test_results:
+            emoji = ":white_check_mark:" if r["pass"] else ":x:"
+            blocks.append({"type": "section", "text": {"type": "mrkdwn", "text": f"{emoji} {r['msg']}"}})
+        blocks.append({"type": "divider"})
+        overall = ":white_check_mark: Restore test passed" if all_pass else ":rotating_light: Restore test failed"
+        blocks.append({"type": "section", "text": {"type": "mrkdwn", "text": f"*Overall:* {overall}"}})
+
+    else:
+        return {
+            "statusCode": 400,
+            "body": json.dumps({
+                "mode": mode,
+                "error": f"Unsupported backup verification mode: {mode}",
+            }),
+        }
+
+    _post_slack(blocks)
+
+    return {
+        "statusCode": 200,
+        "body": json.dumps({
+            "mode": mode,
+            "all_pass": all_pass,
+            "results": [{"pass": r[0], "msg": r[1]} for r in results] if mode == "daily" else test_results,
+        }),
+    }

lambda/backup-verification/requirements.txt

@@ -0,0 +1 @@
+google-cloud-storage>=2.18.0,<3.0.0

lib/constructs/backup-verification.ts

@@ -0,0 +1,95 @@
+import * as cdk from "aws-cdk-lib";
+import * as events from "aws-cdk-lib/aws-events";
+import * as events_targets from "aws-cdk-lib/aws-events-targets";
+import * as iam from "aws-cdk-lib/aws-iam";
+import * as lambda from "aws-cdk-lib/aws-lambda";
+import * as logs from "aws-cdk-lib/aws-logs";
+import * as s3 from "aws-cdk-lib/aws-s3";
+import { PythonFunction } from "@aws-cdk/aws-lambda-python-alpha";
+import { Construct } from "constructs";
+
+interface BackupVerificationProps {
+  sourceBucket: s3.IBucket;
+  replicaBucketName: string;
+  gcsBucket: string;
+  gcsSaSecretName: string;
+  slackWebhookSecretName: string;
+}
+
+export class BackupVerification extends Construct {
+  constructor(scope: Construct, id: string, props: BackupVerificationProps) {
+    super(scope, id);
+
+    const fn = new PythonFunction(this, "Function", {
+      functionName: "forgejo-backup-verification",
+      entry: "lambda/backup-verification",
+      runtime: lambda.Runtime.PYTHON_3_12,
+      architecture: lambda.Architecture.ARM_64,
+      handler: "handler",
+      index: "app.py",
+      memorySize: 512,
+      ephemeralStorageSize: cdk.Size.gibibytes(4),
+      timeout: cdk.Duration.minutes(5),
+      environment: {
+        SOURCE_BUCKET: props.sourceBucket.bucketName,
+        REPLICA_BUCKET: props.replicaBucketName,
+        GCS_BUCKET: props.gcsBucket,
+        GCS_SA_SECRET_NAME: props.gcsSaSecretName,
+        SLACK_WEBHOOK_SECRET_NAME: props.slackWebhookSecretName,
+      },
+      logRetention: logs.RetentionDays.TWO_MONTHS,
+    });
+
+    props.sourceBucket.grantRead(fn);
+
+    fn.addToRolePolicy(
+      new iam.PolicyStatement({
+        actions: ["s3:ListBucket", "s3:GetObject"],
+        resources: [
+          `arn:aws:s3:::${props.replicaBucketName}`,
+          `arn:aws:s3:::${props.replicaBucketName}/*`,
+        ],
+      })
+    );
+
+    const account = cdk.Stack.of(this).account;
+    const region = cdk.Stack.of(this).region;
+
+    fn.addToRolePolicy(
+      new iam.PolicyStatement({
+        actions: ["secretsmanager:GetSecretValue"],
+        resources: [
+          `arn:aws:secretsmanager:${region}:${account}:secret:${props.gcsSaSecretName}-*`,
+          `arn:aws:secretsmanager:${region}:${account}:secret:${props.slackWebhookSecretName}-*`,
+        ],
+      })
+    );
+
+    fn.addToRolePolicy(
+      new iam.PolicyStatement({
+        actions: ["ec2:DescribeSnapshots"],
+        resources: ["*"],
+      })
+    );
+
+    new events.Rule(this, "DailyCheck", {
+      ruleName: "forgejo-backup-daily-check",
+      schedule: events.Schedule.cron({ hour: "8", minute: "0" }),
+      targets: [new events_targets.LambdaFunction(fn)],
+    });
+
+    new events.Rule(this, "MonthlyRestoreTest", {
+      ruleName: "forgejo-backup-monthly-restore-test",
+      schedule: events.Schedule.cron({
+        hour: "9",
+        minute: "0",
+        day: "1",
+      }),
+      targets: [
+        new events_targets.LambdaFunction(fn, {
+          event: events.RuleTargetInput.fromObject({ mode: "restore-test" }),
+        }),
+      ],
+    });
+  }
+}

lib/forgejo-replica-stack.ts

@@ -0,0 +1,37 @@
+import * as cdk from "aws-cdk-lib";
+import * as s3 from "aws-cdk-lib/aws-s3";
+import { Construct } from "constructs";
+
+export class ForgejoReplicaStack extends cdk.Stack {
+  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
+    super(scope, id, props);
+
+    new s3.Bucket(this, "ReplicaBucket", {
+      bucketName: "forgejo-backups-replica-328440206208",
+      encryption: s3.BucketEncryption.S3_MANAGED,
+      blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
+      versioned: true,
+      objectLockEnabled: true,
+      objectLockDefaultRetention: s3.ObjectLockRetention.governance(
+        cdk.Duration.days(90)
+      ),
+      lifecycleRules: [
+        {
+          id: "archive-to-glacier",
+          prefix: "archive/",
+          transitions: [
+            {
+              storageClass: s3.StorageClass.GLACIER,
+              transitionAfter: cdk.Duration.days(30),
+            },
+          ],
+        },
+        {
+          id: "cleanup-noncurrent-versions",
+          noncurrentVersionExpiration: cdk.Duration.days(90),
+        },
+      ],
+      removalPolicy: cdk.RemovalPolicy.RETAIN,
+    });
+  }
+}

lib/forgejo-stack.ts

@@ -2,12 +2,14 @@ import * as cdk from "aws-cdk-lib";
 import * as ec2 from "aws-cdk-lib/aws-ec2";
 import * as iam from "aws-cdk-lib/aws-iam";
 import * as s3 from "aws-cdk-lib/aws-s3";
+import * as secretsmanager from "aws-cdk-lib/aws-secretsmanager";
 import * as elbv2 from "aws-cdk-lib/aws-elasticloadbalancingv2";
 import * as elbv2_targets from "aws-cdk-lib/aws-elasticloadbalancingv2-targets";
 import * as route53 from "aws-cdk-lib/aws-route53";
 import * as route53Targets from "aws-cdk-lib/aws-route53-targets";
 import * as dlm from "aws-cdk-lib/aws-dlm";
 import { Construct } from "constructs";
+import { BackupVerification } from "./constructs/backup-verification";
 
 const FORGEJO_VERSION = "10.0.1";
 
@@ -63,17 +65,97 @@ export class ForgejoStack extends cdk.Stack {
       bucketName: "forgejo-backups-328440206208",
       encryption: s3.BucketEncryption.S3_MANAGED,
       blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
-      lifecycleRules: [{
+      versioned: true,
-        transitions: [
+      lifecycleRules: [
-          { storageClass: s3.StorageClass.GLACIER, transitionAfter: cdk.Duration.days(30) },
+        {
-        ],
+          id: "archive-to-glacier",
-        expiration: cdk.Duration.days(365),
+          prefix: "archive/",
-      }],
+          transitions: [
+            { storageClass: s3.StorageClass.GLACIER, transitionAfter: cdk.Duration.days(30) },
+          ],
+        },
+        {
+          id: "cleanup-noncurrent-versions",
+          noncurrentVersionExpiration: cdk.Duration.days(90),
+        },
+      ],
       removalPolicy: cdk.RemovalPolicy.RETAIN,
     });
 
     backupBucket.grantReadWrite(role);
 
+    const replicaBucketArn = "arn:aws:s3:::forgejo-backups-replica-328440206208";
+
+    const replicationRole = new iam.Role(this, "ReplicationRole", {
+      roleName: "forgejo-s3-replication",
+      assumedBy: new iam.ServicePrincipal("s3.amazonaws.com"),
+    });
+
+    replicationRole.addToPolicy(new iam.PolicyStatement({
+      actions: [
+        "s3:GetReplicationConfiguration",
+        "s3:ListBucket",
+      ],
+      resources: [backupBucket.bucketArn],
+    }));
+
+    replicationRole.addToPolicy(new iam.PolicyStatement({
+      actions: [
+        "s3:GetObjectVersionForReplication",
+        "s3:GetObjectVersionAcl",
+        "s3:GetObjectVersionTagging",
+      ],
+      resources: [`${backupBucket.bucketArn}/*`],
+    }));
+
+    replicationRole.addToPolicy(new iam.PolicyStatement({
+      actions: [
+        "s3:ReplicateObject",
+        "s3:ReplicateDelete",
+        "s3:ReplicateTags",
+      ],
+      resources: [`${replicaBucketArn}/*`],
+    }));
+
+    const cfnBucket = backupBucket.node.defaultChild as s3.CfnBucket;
+    cfnBucket.replicationConfiguration = {
+      role: replicationRole.roleArn,
+      rules: [{
+        id: "replicate-to-west",
+        status: "Enabled",
+        priority: 1,
+        filter: { prefix: "" },
+        deleteMarkerReplication: { status: "Disabled" },
+        destination: {
+          bucket: replicaBucketArn,
+          storageClass: "STANDARD",
+        },
+      }],
+    };
+
+    const gcsTransferUser = new iam.User(this, "GcsTransferUser", {
+      userName: "forgejo-gcs-transfer",
+    });
+
+    gcsTransferUser.addToPolicy(new iam.PolicyStatement({
+      actions: ["s3:GetObject", "s3:ListBucket"],
+      resources: [backupBucket.bucketArn, `${backupBucket.bucketArn}/*`],
+    }));
+
+    const gcsTransferKey = new iam.AccessKey(this, "GcsTransferAccessKey", {
+      user: gcsTransferUser,
+    });
+
+    const gcsTransferCredentials = new secretsmanager.Secret(this, "GcsTransferCredentials", {
+      secretName: "forgejo/gcs-transfer-credentials",
+      secretObjectValue: {
+        accessKeyId: cdk.SecretValue.unsafePlainText(gcsTransferKey.accessKeyId),
+        secretAccessKey: gcsTransferKey.secretAccessKey,
+      },
+    });
+    const gcsTransferCredentialsResource = gcsTransferCredentials.node.defaultChild as secretsmanager.CfnSecret;
+    gcsTransferCredentialsResource.overrideLogicalId("GcsTransferCredentials");
+
     const userData = ec2.UserData.forLinux();
     userData.addCommands(
       "set -euxo pipefail",
@@ -161,7 +243,7 @@ export class ForgejoStack extends cdk.Stack {
       "chown forgejo:forgejo \"$DUMP_DIR\"",
       "cd \"$DUMP_DIR\"",
       "sudo -u forgejo /usr/local/bin/forgejo dump --config /etc/forgejo/app.ini --type tar.gz --file \"$DUMP_DIR/forgejo-${TIMESTAMP}.tar.gz\"",
-      "aws s3 cp \"$DUMP_DIR/forgejo-${TIMESTAMP}.tar.gz\" s3://forgejo-backups-328440206208/${TIMESTAMP}/forgejo-${TIMESTAMP}.tar.gz",
+      "aws s3 cp \"$DUMP_DIR/forgejo-${TIMESTAMP}.tar.gz\" s3://forgejo-backups-328440206208/archive/${TIMESTAMP}/forgejo-${TIMESTAMP}.tar.gz",
       "rm -rf \"$DUMP_DIR\"",
       "BAKEOF",
       "chmod +x /usr/local/bin/forgejo-backup.sh",
@@ -297,8 +379,9 @@ export class ForgejoStack extends cdk.Stack {
         schedules: [{
           name: "forgejo-nightly",
           createRule: { interval: 24, intervalUnit: "HOURS", times: ["06:00"] },
-          retainRule: { count: 7 },
+          retainRule: { count: 30 },
           copyTags: true,
+          tagsToAdd: [{ key: "forgejo-backup", value: "true" }],
         }],
       },
     });
@@ -356,6 +439,14 @@ export class ForgejoStack extends cdk.Stack {
       ),
     });
 
+    new BackupVerification(this, "BackupVerification", {
+      sourceBucket: backupBucket,
+      replicaBucketName: "forgejo-backups-replica-328440206208",
+      gcsBucket: "forgejo-backups-offsite-seahaven",
+      gcsSaSecretName: "forgejo/gcs-sa-key",
+      slackWebhookSecretName: "forgejo/slack-webhook",
+    });
+
     new cdk.CfnOutput(this, "ForgejoUrl", {
       value: "https://forgejo.seahaven.com",
     });

package-lock.json

@@ -8,6 +8,7 @@
       "name": "forgejo",
       "version": "1.0.0",
       "dependencies": {
+        "@aws-cdk/aws-lambda-python-alpha": "^2.252.0-alpha.0",
         "aws-cdk-lib": "^2.252.0",
         "constructs": "^10.0.0"
       },
@@ -33,6 +34,19 @@
       "integrity": "sha512-We4bmHaowOPHr+IQR4/FyTGjRfjgBj4ICMjtqmJeBDWad3Q/6St12NT07leNtyuukv2qMhtSZJQorD8KpKTwRA==",
       "license": "Apache-2.0"
     },
+    "node_modules/@aws-cdk/aws-lambda-python-alpha": {
+      "version": "2.252.0-alpha.0",
+      "resolved": "https://registry.npmjs.org/@aws-cdk/aws-lambda-python-alpha/-/aws-lambda-python-alpha-2.252.0-alpha.0.tgz",
+      "integrity": "sha512-hVcursqZQ6tjToN4AvzOTfoeiN9epJGbUVi5VCGbIT88ide4hUzKsOda29+vw1Ket8nLi5i/xNB9odWU5QWdkw==",
+      "license": "Apache-2.0",
+      "engines": {
+        "node": ">= 20.0.0"
+      },
+      "peerDependencies": {
+        "aws-cdk-lib": "^2.252.0",
+        "constructs": "^10.5.0"
+      }
+    },
     "node_modules/@aws-cdk/cloud-assembly-schema": {
       "version": "53.22.0",
       "resolved": "https://registry.npmjs.org/@aws-cdk/cloud-assembly-schema/-/cloud-assembly-schema-53.22.0.tgz",

package.json

@@ -18,6 +18,7 @@
     "typescript": "~5.7.0"
   },
   "dependencies": {
+    "@aws-cdk/aws-lambda-python-alpha": "^2.252.0-alpha.0",
     "aws-cdk-lib": "^2.252.0",
     "constructs": "^10.0.0"
   }

scripts/gcp-setup.sh

@@ -0,0 +1,150 @@
+#!/bin/bash
+set -euo pipefail
+
+PROJECT_ID="sea-haven-backups"
+BUCKET_NAME="forgejo-backups-offsite-seahaven"
+LOCATION="us-central1"
+SA_NAME="forgejo-backup-verifier"
+SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
+RETENTION_SECONDS=$((2 * 365 * 24 * 3600))  # 2 years
+AWS_REGION="us-east-1"
+AWS_SOURCE_BUCKET="forgejo-backups-328440206208"
+
+GCLOUD="${GCLOUD:-gcloud}"
+GSUTIL="${GSUTIL:-gsutil}"
+
+echo "=== Forgejo 3-2-1 Offsite Backup — GCP Setup ==="
+
+# --- Project ---
+echo ""
+echo "--- Step 1: Create GCP project ---"
+if $GCLOUD projects describe "$PROJECT_ID" &>/dev/null; then
+  echo "Project $PROJECT_ID already exists."
+else
+  $GCLOUD projects create "$PROJECT_ID" --name="Sea Haven Backups"
+  echo "Created project $PROJECT_ID."
+fi
+$GCLOUD config set project "$PROJECT_ID"
+
+echo ""
+echo "--- Step 2: Enable required APIs ---"
+$GCLOUD services enable storage.googleapis.com storagetransfer.googleapis.com
+
+# --- Bucket ---
+echo ""
+echo "--- Step 3: Create GCS bucket ---"
+if $GSUTIL ls -b "gs://$BUCKET_NAME" &>/dev/null; then
+  echo "Bucket gs://$BUCKET_NAME already exists."
+else
+  $GSUTIL mb -p "$PROJECT_ID" -l "$LOCATION" -c NEARLINE -b on "gs://$BUCKET_NAME"
+  echo "Created bucket gs://$BUCKET_NAME."
+fi
+
+echo ""
+echo "--- Step 4: Set lifecycle rules ---"
+LIFECYCLE_JSON=$(cat <<'LCEOF'
+{
+  "rule": [
+    {
+      "action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
+      "condition": {"age": 90}
+    },
+    {
+      "action": {"type": "SetStorageClass", "storageClass": "ARCHIVE"},
+      "condition": {"age": 180}
+    }
+  ]
+}
+LCEOF
+)
+echo "$LIFECYCLE_JSON" | $GSUTIL lifecycle set /dev/stdin "gs://$BUCKET_NAME"
+echo "Lifecycle rules applied."
+
+echo ""
+echo "--- Step 5: Enable object versioning ---"
+$GSUTIL versioning set on "gs://$BUCKET_NAME"
+
+echo ""
+echo "--- Step 6: Set retention policy (2 years) ---"
+$GSUTIL retention set "${RETENTION_SECONDS}s" "gs://$BUCKET_NAME"
+echo "Retention policy set to 2 years."
+
+echo ""
+echo "!!! IMPORTANT: Locking the retention policy is IRREVERSIBLE. !!!"
+echo "Once locked, objects cannot be deleted before the retention period expires."
+echo "Even the project owner cannot shorten or remove the policy."
+echo ""
+read -p "Lock the retention policy now? (yes/no): " CONFIRM
+if [ "$CONFIRM" = "yes" ]; then
+  echo y | $GSUTIL retention lock "gs://$BUCKET_NAME"
+  echo "Retention policy LOCKED."
+else
+  echo "Retention policy set but NOT locked. Run 'gsutil retention lock gs://$BUCKET_NAME' when ready."
+fi
+
+# --- Service Account ---
+echo ""
+echo "--- Step 7: Create service account ---"
+if $GCLOUD iam service-accounts describe "$SA_EMAIL" &>/dev/null 2>&1; then
+  echo "Service account $SA_EMAIL already exists."
+else
+  $GCLOUD iam service-accounts create "$SA_NAME" \
+    --display-name="Forgejo Backup Verifier" \
+    --description="Read-only access to forgejo offsite backup bucket (verification Lambda)"
+  echo "Created service account $SA_EMAIL."
+fi
+
+echo ""
+echo "--- Step 8: Grant bucket permissions ---"
+$GSUTIL iam ch "serviceAccount:${SA_EMAIL}:objectViewer" "gs://$BUCKET_NAME"
+echo "Granted objectViewer to $SA_EMAIL."
+
+echo ""
+echo "--- Step 9: Create and store service account key ---"
+KEY_FILE=$(mktemp)
+$GCLOUD iam service-accounts keys create "$KEY_FILE" --iam-account="$SA_EMAIL"
+echo "Service account key created."
+
+if aws secretsmanager describe-secret --secret-id forgejo/gcs-sa-key --region "$AWS_REGION" &>/dev/null 2>&1; then
+  aws secretsmanager put-secret-value \
+    --secret-id forgejo/gcs-sa-key \
+    --secret-string "file://$KEY_FILE" \
+    --region "$AWS_REGION"
+  echo "Updated existing secret forgejo/gcs-sa-key."
+else
+  aws secretsmanager create-secret \
+    --name forgejo/gcs-sa-key \
+    --secret-string "file://$KEY_FILE" \
+    --region "$AWS_REGION"
+  echo "Created secret forgejo/gcs-sa-key."
+fi
+rm -f "$KEY_FILE"
+echo "Key stored in AWS Secrets Manager, local copy deleted."
+
+# --- Storage Transfer ---
+echo ""
+echo "--- Step 10: Configure Storage Transfer Service ---"
+echo ""
+echo "Storage Transfer Service requires AWS credentials to read from S3."
+echo "Create a read-only IAM user 'forgejo-gcs-transfer' in AWS with access to:"
+echo "  s3://forgejo-backups-328440206208 (GetObject, ListBucket)"
+echo ""
+echo "Then configure the transfer job in the GCP Console:"
+echo "  1. Go to: https://console.cloud.google.com/transfer/jobs"
+echo "  2. Source: Amazon S3 — bucket '$AWS_SOURCE_BUCKET'"
+echo "  3. Destination: GCS — bucket '$BUCKET_NAME'"
+echo "  4. Schedule: Daily at 10:00 UTC"
+echo "  5. Enter the AWS access key ID and secret for the read-only user"
+echo ""
+echo "The CDK stack creates the 'forgejo-gcs-transfer' IAM user automatically."
+echo "Retrieve its credentials from Secrets Manager: forgejo/gcs-transfer-credentials"
+
+echo ""
+echo "=== Setup complete ==="
+echo ""
+echo "Summary:"
+echo "  GCP Project:     $PROJECT_ID"
+echo "  GCS Bucket:      gs://$BUCKET_NAME"
+echo "  Service Account: $SA_EMAIL"
+echo "  SA Key Secret:   forgejo/gcs-sa-key (AWS Secrets Manager)"
+echo "  Retention:       2 years (check lock status above)"