mirror of
https://github.com/Sea-Haven-Industries/forgejo.git
synced 2026-09-30 06:33:11 +00:00
236 lines
10 KiB
Markdown
236 lines
10 KiB
Markdown
# forgejo
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Self-hosted Forgejo git server for archiving GitHub repos and mirroring active ones. Runs on a single EC2 instance within the Sea Haven VPC, fronted by the `seahaven-com` ALB for HTTPS.
|
|
|
|
## Architecture
|
|
|
|
- **EC2**: t4g.small (arm64), Amazon Linux 2023, 20 GiB gp3 root (no state lives on it) — look up instance ID with:
|
|
```
|
|
aws cloudformation describe-stacks --stack-name forgejo --query 'Stacks[0].Outputs[?OutputKey==`InstanceId`].OutputValue' --output text
|
|
```
|
|
The AMI is cached in the committed `cdk.context.json` (`cachedInContext: true`) so deploys never pick up a new AL2023 release implicitly — an AMI change forces instance replacement and must be deliberate (`cdk context --reset <ami key> && cdk synth`).
|
|
- **Data volume**: standalone 50 GiB gp3 encrypted EBS volume (`RemovalPolicy.RETAIN`) mounted at `/var/lib/forgejo` — sqlite database, repositories, and logs all live here and **survive instance replacement and stack deletion**. UserData waits for the volume attachment, mounts the existing filesystem (a `blkid` guard prevents formatting a disk that has one), and tags it `forgejo-backup=true` for DLM snapshots.
|
|
- **Restore-on-boot**: if the data volume has no database on boot (first boot or total volume loss), UserData automatically downloads the latest S3 dump and restores it before starting the service — volume loss self-heals to ≤24h-old state with no manual steps.
|
|
- **Network**: Private subnet (us-east-1a), behind `seahaven-com` ALB for SSL termination
|
|
- **DNS**: `forgejo.seahaven.com` — Route53 alias record pointing to the `seahaven-com` ALB (not a direct A record)
|
|
- **TLS**: Wildcard cert on ALB, HTTP internally on port 3000
|
|
- **Backup**: Nightly `forgejo dump` to S3 + EBS snapshots via DLM (see [3-2-1 Backup Strategy](#3-2-1-backup-strategy))
|
|
- **Admin access**: SSM Session Manager (no SSH port exposed)
|
|
- **CI/CD**: GitHub Actions with OIDC role `githubdeploy-forgejo`
|
|
|
|
### Ports
|
|
|
|
| Port | Protocol | Source | Purpose |
|
|
|------|----------|--------|---------|
|
|
| 443 | HTTPS | ALB (public) | Web UI + HTTP git clone |
|
|
| 3000 | HTTP | ALB → instance | Internal traffic from ALB |
|
|
| 2222 | SSH | VPC + VPN | Git SSH operations |
|
|
|
|
## 3-2-1 Backup Strategy
|
|
|
|
All backups follow a 3-2-1 strategy: 3 copies, 2 storage types, 1 offsite provider.
|
|
|
|
| Copy | Location | Type | Retention |
|
|
|------|----------|------|-----------|
|
|
| Live | Standalone EBS data volume, RETAIN (us-east-1) | Block | N/A |
|
|
| Near-site | S3 replica (us-west-2) | Object | Archive: indefinite, noncurrent versions: 90d |
|
|
| Offsite | GCS `forgejo-backups-offsite-seahaven` (GCP us-central1) | Object | 2-year locked retention |
|
|
|
|
**Daily data flow:**
|
|
|
|
| Time (UTC) | Event |
|
|
|------------|-------|
|
|
| 05:00 | `forgejo dump` → `s3://forgejo-backups-328440206208/archive/{date}/` |
|
|
| ~05:01 | S3 CRR replicates to `forgejo-backups-replica-328440206208` (us-west-2) |
|
|
| 06:00 | DLM EBS snapshot (30-day retention) |
|
|
| 08:00 | Verification Lambda checks all 3 locations, posts to Slack |
|
|
| 10:00 | GCS Storage Transfer pulls from S3 to GCS offsite |
|
|
|
|
**S3 source lifecycle:** Standard 30d → Glacier (no expiration).
|
|
|
|
**Immutability layers:**
|
|
- S3 Versioning on both source and replica buckets
|
|
- S3 Object Lock (Governance, 90d) on the replica bucket
|
|
- GCS Bucket Lock (2yr, irreversible) on the offsite bucket
|
|
|
|
### Verification
|
|
|
|
The `forgejo-backup-verification` Lambda runs daily at 08:00 UTC and checks:
|
|
1. S3 source has a recent dump under `archive/`
|
|
2. S3 replica has replicated the latest dump
|
|
3. GCS offsite has received the latest transfer
|
|
4. EBS snapshots exist within the last 48 hours
|
|
|
|
On the 1st of each month at 09:00 UTC, it runs a restore test: downloads the latest dump, extracts the archive, and runs SQLite integrity checks.
|
|
|
|
Two CloudWatch alarms watch the verification Lambda (both notify the `site-alerts` SNS topic, ALARM action only):
|
|
- `forgejo-backup-verification-errors` — Errors ≥ 1 in an hour, missing data = not breaching ("when it runs, did it fail")
|
|
- `forgejo-backup-verification-not-running` — Invocations < 1 over 24h, missing data = breaching ("did it run at all")
|
|
|
|
### Manual backup
|
|
|
|
```bash
|
|
sudo /usr/local/bin/forgejo-backup.sh
|
|
```
|
|
|
|
### Restore from S3
|
|
|
|
For backups older than 30 days (Glacier), restore the object first:
|
|
|
|
```bash
|
|
aws s3api restore-object --bucket forgejo-backups-328440206208 \
|
|
--key "archive/<date>/forgejo-<date>.tar.gz" \
|
|
--restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Standard"}}'
|
|
# Wait ~3-5 hours for restore to complete, then:
|
|
```
|
|
|
|
Download and restore:
|
|
|
|
```bash
|
|
aws s3 cp s3://forgejo-backups-328440206208/archive/<date>/forgejo-<date>.tar.gz /tmp/
|
|
systemctl stop forgejo
|
|
mkdir -p /tmp/forgejo-restore && tar -xzf /tmp/forgejo-<date>.tar.gz -C /tmp/forgejo-restore
|
|
cd /tmp/forgejo-restore
|
|
cp app.ini /etc/forgejo/app.ini
|
|
cp gitea-db.sqlite3 /var/lib/forgejo/data/forgejo.db
|
|
rm -rf /var/lib/forgejo/data/repositories
|
|
cp -a repos /var/lib/forgejo/data/repositories
|
|
cp -a data/. /var/lib/forgejo/data/
|
|
[ -d lfs ] && cp -a lfs/. /var/lib/forgejo/data/lfs/
|
|
[ -d custom ] && cp -a custom/. /var/lib/forgejo/custom/
|
|
chown -R forgejo:forgejo /var/lib/forgejo /etc/forgejo/app.ini
|
|
systemctl start forgejo
|
|
rm -rf /tmp/forgejo-restore /tmp/forgejo-<date>.tar.gz
|
|
```
|
|
|
|
### Restore from GCS (disaster recovery)
|
|
|
|
```bash
|
|
gcloud config set project sea-haven-backups
|
|
gsutil cp gs://forgejo-backups-offsite-seahaven/archive/<date>/forgejo-<date>.tar.gz /tmp/
|
|
# Then follow the same restore steps as S3 above
|
|
```
|
|
|
|
## Autodiscovery
|
|
|
|
An hourly cron job checks the `Sea-Haven-Industries` GitHub org for new repositories and mirrors them into Forgejo automatically.
|
|
|
|
- **Active repos** are created as mirrors (ongoing sync).
|
|
- **Archived repos** are created as static one-time imports.
|
|
- **Script**: `/usr/local/bin/forgejo-autodiscover.sh`
|
|
- **Log**: `/var/log/forgejo-autodiscover.log`
|
|
|
|
## Token Refresh
|
|
|
|
A daily cron at 4:30 UTC reads the GitHub PAT from Secrets Manager (`forgejo/github-pat`) and updates the git remote URL on every mirror repository so credentials stay current.
|
|
|
|
- **Script**: `/usr/local/bin/forgejo-refresh-tokens.sh`
|
|
|
|
## PAT Rotation
|
|
|
|
The GitHub personal access token used for mirroring is a fine-grained PAT scoped to `Sea-Haven-Industries` with **Contents: Read-only** permissions and a 1-year expiration. It is stored in Secrets Manager at `forgejo/github-pat`.
|
|
|
|
To rotate:
|
|
|
|
1. Create a new fine-grained PAT on GitHub with the same scope.
|
|
2. Update the secret value in Secrets Manager (`forgejo/github-pat`).
|
|
3. The daily token-refresh cron will pick it up automatically.
|
|
|
|
To force immediate propagation:
|
|
|
|
```bash
|
|
sudo /usr/local/bin/forgejo-refresh-tokens.sh
|
|
```
|
|
|
|
## Secrets Manager
|
|
|
|
| Secret | Purpose |
|
|
|--------|---------|
|
|
| `forgejo/admin-password` | Forgejo admin user password |
|
|
| `forgejo/api-token` | Forgejo API token (used by autodiscovery and token refresh scripts) |
|
|
| `forgejo/github-pat` | GitHub fine-grained PAT for mirroring |
|
|
| `forgejo/gcs-sa-key` | GCP service account key for offsite backup verification |
|
|
| `forgejo/gcs-transfer-credentials` | AWS IAM credentials for GCS Storage Transfer Service |
|
|
| `forgejo/slack-webhook` | Slack webhook URL for backup verification alerts |
|
|
|
|
## First-time setup
|
|
|
|
After the stack deploys, connect via SSM and create the admin user:
|
|
|
|
```bash
|
|
INSTANCE_ID=$(aws cloudformation describe-stacks --stack-name forgejo --query 'Stacks[0].Outputs[?OutputKey==`InstanceId`].OutputValue' --output text)
|
|
aws ssm start-session --target "$INSTANCE_ID"
|
|
|
|
sudo -u forgejo /usr/local/bin/forgejo admin user create \
|
|
--admin \
|
|
--username adam \
|
|
--password '<password>' \
|
|
--email adam@seahavenind.com \
|
|
--config /etc/forgejo/app.ini
|
|
```
|
|
|
|
Admin password is stored in Secrets Manager at `forgejo/admin-password`.
|
|
|
|
Access the web UI at `https://forgejo.seahaven.com`.
|
|
|
|
## Migrating repos from GitHub
|
|
|
|
### Archived repos (one-time import)
|
|
|
|
In the Forgejo web UI: **New Migration → GitHub** → paste the GitHub repo URL. Use a GitHub personal access token for private repos. These are full imports (code, issues, PRs, releases).
|
|
|
|
### Active repos (mirror sync)
|
|
|
|
Same migration flow, but check **This Repository Will Be A Mirror**. Forgejo polls GitHub hourly (`DEFAULT_INTERVAL = 1h` in app.ini) and keeps the mirror in sync.
|
|
|
|
## GCP Offsite Setup (one-time)
|
|
|
|
Run the setup script to create the GCS offsite bucket, service account, and store credentials:
|
|
|
|
```bash
|
|
./scripts/gcp-setup.sh
|
|
```
|
|
|
|
This creates the `sea-haven-backups` GCP project with a locked-retention GCS bucket. After running, configure the Storage Transfer job in the GCP Console using the AWS credentials from `forgejo/gcs-transfer-credentials`.
|
|
|
|
## Deployment
|
|
|
|
```bash
|
|
npm install
|
|
npx cdk deploy --all
|
|
```
|
|
|
|
This deploys two stacks:
|
|
- `forgejo-replica` (us-west-2) — S3 replica bucket with Object Lock
|
|
- `forgejo` (us-east-1) — main stack with Forgejo instance, CRR, and verification Lambda
|
|
|
|
CI/CD is handled by GitHub Actions — PRs run CI, merges to `main` deploy via the reusable CDK workflow.
|
|
|
|
## Post-deploy: store Slack webhook
|
|
|
|
Store the Slack webhook URL for backup verification alerts:
|
|
|
|
```bash
|
|
aws secretsmanager create-secret --name forgejo/slack-webhook \
|
|
--secret-string "https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \
|
|
--region us-east-1
|
|
```
|
|
|
|
## Updating Forgejo
|
|
|
|
Update the `FORGEJO_VERSION` constant in `lib/forgejo-stack.ts` and deploy. This replaces the instance, so ensure the latest EBS snapshot is available for data recovery if needed. Alternatively, update in-place via SSM:
|
|
|
|
```bash
|
|
INSTANCE_ID=$(aws cloudformation describe-stacks --stack-name forgejo --query 'Stacks[0].Outputs[?OutputKey==`InstanceId`].OutputValue' --output text)
|
|
aws ssm start-session --target "$INSTANCE_ID"
|
|
|
|
sudo systemctl stop forgejo
|
|
sudo curl -Lo /usr/local/bin/forgejo "https://codeberg.org/forgejo/forgejo/releases/download/v<NEW_VERSION>/forgejo-<NEW_VERSION>-linux-arm64"
|
|
sudo chmod +x /usr/local/bin/forgejo
|
|
sudo systemctl start forgejo
|
|
```
|