forgejo/README.md

# forgejo

Self-hosted Forgejo git server for archiving GitHub repos and mirroring active ones. Runs on a single EC2 instance within the Sea Haven VPC, fronted by the `seahaven-com` ALB for HTTPS.

## Architecture

- **EC2**: t4g.small (arm64), Amazon Linux 2023, 50GB gp3 EBS — instance `i-0d3005fb3c36124cd`
- **Network**: Private subnet (us-east-1a), behind `seahaven-com` ALB for SSL termination
- **DNS**: `forgejo.seahaven.com` — Route53 alias record pointing to the `seahaven-com` ALB (not a direct A record)
- **TLS**: Wildcard cert on ALB, HTTP internally on port 3000
- **Backup**: Nightly `forgejo dump` to S3 + EBS snapshots via DLM (see [S3 Backups](#s3-backups))
- **Admin access**: SSM Session Manager (no SSH port exposed)
- **CI/CD**: GitHub Actions with OIDC role `githubdeploy-forgejo`

### Ports

| Port | Protocol | Source | Purpose |
|------|----------|--------|---------|
| 443 | HTTPS | ALB (public) | Web UI + HTTP git clone |
| 3000 | HTTP | ALB → instance | Internal traffic from ALB |
| 2222 | SSH | VPC + VPN | Git SSH operations |

## 3-2-1 Backup Strategy

All backups follow a 3-2-1 strategy: 3 copies, 2 storage types, 1 offsite provider.

| Copy | Location | Type | Retention |
|------|----------|------|-----------|
| Live | EBS volume (us-east-1) | Block | N/A |
| Near-site | S3 replica (us-west-2) | Object | Archive: indefinite, noncurrent versions: 90d |
| Offsite | GCS `forgejo-backups-offsite-seahaven` (GCP us-central1) | Object | 2-year locked retention |

**Daily data flow:**

| Time (UTC) | Event |
|------------|-------|
| 05:00 | `forgejo dump` → `s3://forgejo-backups-328440206208/archive/{date}/` |
| ~05:01 | S3 CRR replicates to `forgejo-backups-replica-328440206208` (us-west-2) |
| 06:00 | DLM EBS snapshot (30-day retention) |
| 08:00 | Verification Lambda checks all 3 locations, posts to Slack |
| 10:00 | GCS Storage Transfer pulls from S3 to GCS offsite |

**S3 source lifecycle:** Standard 30d → Glacier (no expiration).

**Immutability layers:**
- S3 Versioning on both source and replica buckets
- S3 Object Lock (Governance, 90d) on the replica bucket
- GCS Bucket Lock (2yr, irreversible) on the offsite bucket

### Verification

The `forgejo-backup-verification` Lambda runs daily at 08:00 UTC and checks:
1. S3 source has a recent dump under `archive/`
2. S3 replica has replicated the latest dump
3. GCS offsite has received the latest transfer
4. EBS snapshots exist within the last 48 hours

On the 1st of each month at 09:00 UTC, it runs a restore test: downloads the latest dump, extracts the archive, and runs SQLite integrity checks.

### Manual backup

```bash
sudo /usr/local/bin/forgejo-backup.sh
```

### Restore from S3

```bash
aws s3 cp s3://forgejo-backups-328440206208/archive/<date>/forgejo-<date>.tar.gz /tmp/
systemctl stop forgejo
cd /tmp && tar xzf forgejo-<date>.tar.gz
forgejo restore --config /etc/forgejo/app.ini --from /tmp/forgejo-dump-*
chown -R forgejo:forgejo /var/lib/forgejo
systemctl start forgejo
```

### Restore from GCS (disaster recovery)

```bash
gcloud config set project seahaven-backups
gsutil cp gs://forgejo-backups-offsite-seahaven/archive/<date>/forgejo-<date>.tar.gz /tmp/
# Then follow the same restore steps as S3
```

To test the backup manually:

```bash
sudo /usr/local/bin/forgejo-backup.sh
```

## Autodiscovery

An hourly cron job checks the `Sea-Haven-Industries` GitHub org for new repositories and mirrors them into Forgejo automatically.

- **Active repos** are created as mirrors (ongoing sync).
- **Archived repos** are created as static one-time imports.
- **Script**: `/usr/local/bin/forgejo-autodiscover.sh`
- **Log**: `/var/log/forgejo-autodiscover.log`

## Token Refresh

A daily cron at 4:30 UTC reads the GitHub PAT from Secrets Manager (`forgejo/github-pat`) and updates the git remote URL on every mirror repository so credentials stay current.

- **Script**: `/usr/local/bin/forgejo-refresh-tokens.sh`

## PAT Rotation

The GitHub personal access token used for mirroring is a fine-grained PAT scoped to `Sea-Haven-Industries` with **Contents: Read-only** permissions and a 1-year expiration. It is stored in Secrets Manager at `forgejo/github-pat`.

To rotate:

1. Create a new fine-grained PAT on GitHub with the same scope.
2. Update the secret value in Secrets Manager (`forgejo/github-pat`).
3. The daily token-refresh cron will pick it up automatically.

To force immediate propagation:

```bash
sudo /usr/local/bin/forgejo-refresh-tokens.sh
```

## Secrets Manager

| Secret | Purpose |
|--------|---------|
| `forgejo/admin-password` | Forgejo admin user password |
| `forgejo/api-token` | Forgejo API token (used by autodiscovery and token refresh scripts) |
| `forgejo/github-pat` | GitHub fine-grained PAT for mirroring |
| `forgejo/gcs-sa-key` | GCP service account key for offsite backup verification |
| `forgejo/gcs-transfer-credentials` | AWS IAM credentials for GCS Storage Transfer Service |
| `forgejo/slack-webhook` | Slack webhook URL for backup verification alerts |

## First-time setup

After the stack deploys, connect via SSM and create the admin user:

```bash
aws ssm start-session --target i-0d3005fb3c36124cd

sudo -u forgejo /usr/local/bin/forgejo admin user create \
  --admin \
  --username adam \
  --password '<password>' \
  --email adam@seahavenind.com \
  --config /etc/forgejo/app.ini
```

Admin password is stored in Secrets Manager at `forgejo/admin-password`.

Access the web UI at `https://forgejo.seahaven.com`.

## Migrating repos from GitHub

### Archived repos (one-time import)

In the Forgejo web UI: **New Migration → GitHub** → paste the GitHub repo URL. Use a GitHub personal access token for private repos. These are full imports (code, issues, PRs, releases).

### Active repos (mirror sync)

Same migration flow, but check **This Repository Will Be A Mirror**. Forgejo polls GitHub hourly (`DEFAULT_INTERVAL = 1h` in app.ini) and keeps the mirror in sync.

## GCP Offsite Setup (one-time)

Run the setup script to create the GCS offsite bucket, service account, and store credentials:

```bash
./scripts/gcp-setup.sh
```

This creates the `seahaven-backups` GCP project with a locked-retention GCS bucket. After running, configure the Storage Transfer job in the GCP Console using the AWS credentials from `forgejo/gcs-transfer-credentials`.

## Deployment

```bash
npm install
npx cdk deploy --all
```

This deploys two stacks:
- `forgejo-replica` (us-west-2) — S3 replica bucket with Object Lock
- `forgejo` (us-east-1) — main stack with Forgejo instance, CRR, and verification Lambda

CI/CD is handled by GitHub Actions — PRs run CI, merges to `main` deploy via the reusable CDK workflow.

## Post-deploy: update running instance backup path

After the first deploy with the 3-2-1 changes, the running instance's backup script still uses the old S3 path (without the `archive/` prefix). Update it via SSM:

```bash
aws ssm start-session --target i-0d3005fb3c36124cd
sudo sed -i 's|s3://forgejo-backups-328440206208/${TIMESTAMP}/|s3://forgejo-backups-328440206208/archive/${TIMESTAMP}/|' /usr/local/bin/forgejo-backup.sh
```

Also store the Slack webhook URL for backup verification alerts:

```bash
aws secretsmanager create-secret --name forgejo/slack-webhook \
  --secret-string "https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \
  --region us-east-1
```

## Updating Forgejo

Update the `FORGEJO_VERSION` constant in `lib/forgejo-stack.ts` and deploy. This replaces the instance, so ensure the latest EBS snapshot is available for data recovery if needed. Alternatively, update in-place via SSM:

```bash
aws ssm start-session --target i-0d3005fb3c36124cd

sudo systemctl stop forgejo
sudo curl -Lo /usr/local/bin/forgejo "https://codeberg.org/forgejo/forgejo/releases/download/v<NEW_VERSION>/forgejo-<NEW_VERSION>-linux-arm64"
sudo chmod +x /usr/local/bin/forgejo
sudo systemctl start forgejo
```