forgejo/scripts/gcp-setup.sh

152 lines
5 KiB
Bash
Raw Permalink Normal View History

Add 3-2-1 backup strategy (#2) * Add 3-2-1 backup strategy with cross-region replication and GCS offsite Implements a fully compliant 3-2-1 backup architecture: - Copy 1 (live): Harden existing EBS snapshots to 30-day retention - Copy 2 (near-site): S3 cross-region replication to us-west-2 with Object Lock (governance 90d) and versioning - Copy 3 (offsite): GCS bucket in dedicated seahaven-backups GCP project with 2-year irreversible retention lock Also adds a verification Lambda that checks all 3 locations daily and runs monthly restore tests with SQLite integrity checks. * Enable QEMU in CI for arm64 Lambda Docker builds * Commit cdk.context.json for CI synth without AWS credentials Vpc.fromLookup requires cached context to synthesize without AWS credentials. Required for CI which runs cdk synth without an OIDC role. * Fix GCP project ID to sea-haven-backups * Address code review findings for backup verification Fix 4 critical issues: - Add filter/priority/deleteMarkerReplication to S3 CRR rule (deploy would fail without) - Add stack dependency so replica deploys before main stack - Fix DB file extension matching (.sqlite3/.sql instead of .db) - Replace nonexistent `forgejo restore` command with actual restore steps in README Fix 4 moderate issues: - Add timeout=10 to Slack webhook urlopen call - Add filter='data' to tarfile.extract for PEP 706 compliance - Add explicit ValueError for unknown handler mode - Use date-scoped S3/GCS prefix instead of unbounded listing * Fix backup strategy bug findings * Handle SQL text dumps separately from binary SQLite in restore test Forgejo dump produces gitea-db.sql as a text SQL dump (XORM export), not a binary SQLite file. Opening it directly with sqlite3.connect() throws DatabaseError. Now imports the SQL dump into a temp DB first. * Fix GCS backup check: align staleness cutoff and add size validation GCS check used a 72h cutoff but only listed 2 days of prefixes (~48h), making the staleness check unreachable. Also added 1MB minimum file size validation to match the S3 check. * Rename SECRET_ARN env vars to SECRET_NAME to match actual values * Fix EBS snapshot state check, drop unused GCS write grant and dead lifecycle rule * Fix restore runbook, DLM snapshot tagging, README cleanup, and gsutil prompt * Fix restore runbook: trailing-dot cp idiom and Glacier restore step * Rename GCS service account to match read-only permissions * Add 4 GiB ephemeral storage to verification Lambda Monthly restore-test downloads and extracts the full dump tarball in /tmp. As the dump grows with LFS data, the default 512 MB will eventually cause ENOSPC failures. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-05-14 18:08:06 -04:00
#!/bin/bash
set -euo pipefail
PROJECT_ID="sea-haven-backups"
BUCKET_NAME="forgejo-backups-offsite-seahaven"
LOCATION="us-central1"
SA_NAME="forgejo-backup-verifier"
SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
RETENTION_SECONDS=$((2 * 365 * 24 * 3600)) # 2 years
AWS_REGION="us-east-1"
AWS_SOURCE_BUCKET="forgejo-backups-328440206208"
GCLOUD="${GCLOUD:-gcloud}"
GSUTIL="${GSUTIL:-gsutil}"
echo "=== Forgejo 3-2-1 Offsite Backup — GCP Setup ==="
# --- Project ---
echo ""
echo "--- Step 1: Create GCP project ---"
if $GCLOUD projects describe "$PROJECT_ID" &>/dev/null; then
echo "Project $PROJECT_ID already exists."
else
$GCLOUD projects create "$PROJECT_ID" --name="Sea Haven Backups"
echo "Created project $PROJECT_ID."
fi
$GCLOUD config set project "$PROJECT_ID"
echo ""
echo "--- Step 2: Enable required APIs ---"
$GCLOUD services enable storage.googleapis.com storagetransfer.googleapis.com
# --- Bucket ---
echo ""
echo "--- Step 3: Create GCS bucket ---"
if $GSUTIL ls -b "gs://$BUCKET_NAME" &>/dev/null; then
echo "Bucket gs://$BUCKET_NAME already exists."
else
$GSUTIL mb -p "$PROJECT_ID" -l "$LOCATION" -c NEARLINE -b on "gs://$BUCKET_NAME"
echo "Created bucket gs://$BUCKET_NAME."
fi
echo ""
echo "--- Step 4: Set lifecycle rules ---"
LIFECYCLE_JSON=$(cat <<'LCEOF'
{
"rule": [
{
"action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
"condition": {"age": 90}
},
{
"action": {"type": "SetStorageClass", "storageClass": "ARCHIVE"},
"condition": {"age": 180}
}
]
}
LCEOF
)
echo "$LIFECYCLE_JSON" | $GSUTIL lifecycle set /dev/stdin "gs://$BUCKET_NAME"
echo "Lifecycle rules applied."
echo ""
echo "--- Step 5: Enable object versioning ---"
$GSUTIL versioning set on "gs://$BUCKET_NAME"
echo ""
echo "--- Step 6: Set retention policy (2 years) ---"
$GSUTIL retention set "${RETENTION_SECONDS}s" "gs://$BUCKET_NAME"
echo "Retention policy set to 2 years."
echo ""
echo "!!! IMPORTANT: Locking the retention policy is IRREVERSIBLE. !!!"
echo "Once locked, objects cannot be deleted before the retention period expires."
echo "Even the project owner cannot shorten or remove the policy."
echo ""
read -p "Lock the retention policy now? (yes/no): " CONFIRM
if [ "$CONFIRM" = "yes" ]; then
echo y | $GSUTIL retention lock "gs://$BUCKET_NAME"
echo "Retention policy LOCKED."
else
echo "Retention policy set but NOT locked. Run 'gsutil retention lock gs://$BUCKET_NAME' when ready."
fi
# --- Service Account ---
echo ""
echo "--- Step 7: Create service account ---"
if $GCLOUD iam service-accounts describe "$SA_EMAIL" &>/dev/null 2>&1; then
echo "Service account $SA_EMAIL already exists."
else
$GCLOUD iam service-accounts create "$SA_NAME" \
--display-name="Forgejo Backup Verifier" \
--description="Read-only access to forgejo offsite backup bucket (verification Lambda)"
echo "Created service account $SA_EMAIL."
fi
echo ""
echo "--- Step 8: Grant bucket permissions ---"
$GSUTIL iam ch "serviceAccount:${SA_EMAIL}:objectViewer" "gs://$BUCKET_NAME"
echo "Granted objectViewer to $SA_EMAIL."
echo ""
echo "--- Step 9: Create and store service account key ---"
KEY_FILE=$(mktemp)
fix(scripts): ensure temp GCP SA key cleanup via EXIT trap (SEC-28) (#70) * chore(deps): bump aws-cdk-lib from 2.262.2 to 2.263.0 Bumps [aws-cdk-lib](https://github.com/aws/aws-cdk/tree/HEAD/packages/aws-cdk-lib) from 2.262.2 to 2.263.0. - [Release notes](https://github.com/aws/aws-cdk/releases) - [Changelog](https://github.com/aws/aws-cdk/blob/main/CHANGELOG.v2.alpha.md) - [Commits](https://github.com/aws/aws-cdk/commits/v2.263.0/packages/aws-cdk-lib) --- updated-dependencies: - dependency-name: aws-cdk-lib dependency-version: 2.263.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps-dev): bump tsx from 4.23.1 to 4.23.5 Bumps [tsx](https://github.com/privatenumber/tsx) from 4.23.1 to 4.23.5. - [Release notes](https://github.com/privatenumber/tsx/releases) - [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs) - [Commits](https://github.com/privatenumber/tsx/compare/v4.23.1...v4.23.5) --- updated-dependencies: - dependency-name: tsx dependency-version: 4.23.5 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * fix: add trap to ensure temp key file removal after GCP SA key creation * fix(scripts): delete temp GCP key before success message * chore(deps): drop cdk-lib and tsx bumps from this PR --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-07 10:58:30 -04:00
trap 'rm -f "$KEY_FILE"' EXIT
Add 3-2-1 backup strategy (#2) * Add 3-2-1 backup strategy with cross-region replication and GCS offsite Implements a fully compliant 3-2-1 backup architecture: - Copy 1 (live): Harden existing EBS snapshots to 30-day retention - Copy 2 (near-site): S3 cross-region replication to us-west-2 with Object Lock (governance 90d) and versioning - Copy 3 (offsite): GCS bucket in dedicated seahaven-backups GCP project with 2-year irreversible retention lock Also adds a verification Lambda that checks all 3 locations daily and runs monthly restore tests with SQLite integrity checks. * Enable QEMU in CI for arm64 Lambda Docker builds * Commit cdk.context.json for CI synth without AWS credentials Vpc.fromLookup requires cached context to synthesize without AWS credentials. Required for CI which runs cdk synth without an OIDC role. * Fix GCP project ID to sea-haven-backups * Address code review findings for backup verification Fix 4 critical issues: - Add filter/priority/deleteMarkerReplication to S3 CRR rule (deploy would fail without) - Add stack dependency so replica deploys before main stack - Fix DB file extension matching (.sqlite3/.sql instead of .db) - Replace nonexistent `forgejo restore` command with actual restore steps in README Fix 4 moderate issues: - Add timeout=10 to Slack webhook urlopen call - Add filter='data' to tarfile.extract for PEP 706 compliance - Add explicit ValueError for unknown handler mode - Use date-scoped S3/GCS prefix instead of unbounded listing * Fix backup strategy bug findings * Handle SQL text dumps separately from binary SQLite in restore test Forgejo dump produces gitea-db.sql as a text SQL dump (XORM export), not a binary SQLite file. Opening it directly with sqlite3.connect() throws DatabaseError. Now imports the SQL dump into a temp DB first. * Fix GCS backup check: align staleness cutoff and add size validation GCS check used a 72h cutoff but only listed 2 days of prefixes (~48h), making the staleness check unreachable. Also added 1MB minimum file size validation to match the S3 check. * Rename SECRET_ARN env vars to SECRET_NAME to match actual values * Fix EBS snapshot state check, drop unused GCS write grant and dead lifecycle rule * Fix restore runbook, DLM snapshot tagging, README cleanup, and gsutil prompt * Fix restore runbook: trailing-dot cp idiom and Glacier restore step * Rename GCS service account to match read-only permissions * Add 4 GiB ephemeral storage to verification Lambda Monthly restore-test downloads and extracts the full dump tarball in /tmp. As the dump grows with LFS data, the default 512 MB will eventually cause ENOSPC failures. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-05-14 18:08:06 -04:00
$GCLOUD iam service-accounts keys create "$KEY_FILE" --iam-account="$SA_EMAIL"
echo "Service account key created."
if aws secretsmanager describe-secret --secret-id forgejo/gcs-sa-key --region "$AWS_REGION" &>/dev/null 2>&1; then
aws secretsmanager put-secret-value \
--secret-id forgejo/gcs-sa-key \
--secret-string "file://$KEY_FILE" \
--region "$AWS_REGION"
echo "Updated existing secret forgejo/gcs-sa-key."
else
aws secretsmanager create-secret \
--name forgejo/gcs-sa-key \
--secret-string "file://$KEY_FILE" \
--region "$AWS_REGION"
echo "Created secret forgejo/gcs-sa-key."
fi
rm -f "$KEY_FILE"
echo "Key stored in AWS Secrets Manager, local copy deleted."
# --- Storage Transfer ---
echo ""
echo "--- Step 10: Configure Storage Transfer Service ---"
echo ""
echo "Storage Transfer Service requires AWS credentials to read from S3."
echo "Create a read-only IAM user 'forgejo-gcs-transfer' in AWS with access to:"
echo " s3://forgejo-backups-328440206208 (GetObject, ListBucket)"
echo ""
echo "Then configure the transfer job in the GCP Console:"
echo " 1. Go to: https://console.cloud.google.com/transfer/jobs"
echo " 2. Source: Amazon S3 — bucket '$AWS_SOURCE_BUCKET'"
echo " 3. Destination: GCS — bucket '$BUCKET_NAME'"
echo " 4. Schedule: Daily at 10:00 UTC"
echo " 5. Enter the AWS access key ID and secret for the read-only user"
echo ""
echo "The CDK stack creates the 'forgejo-gcs-transfer' IAM user automatically."
echo "Retrieve its credentials from Secrets Manager: forgejo/gcs-transfer-credentials"
echo ""
echo "=== Setup complete ==="
echo ""
echo "Summary:"
echo " GCP Project: $PROJECT_ID"
echo " GCS Bucket: gs://$BUCKET_NAME"
echo " Service Account: $SA_EMAIL"
echo " SA Key Secret: forgejo/gcs-sa-key (AWS Secrets Manager)"
echo " Retention: 2 years (check lock status above)"