Update README with backup, autodiscovery, and token management docs (#1 )

Add documentation for S3 backups with Glacier lifecycle, hourly autodiscovery of new GitHub org repos, daily PAT token refresh, PAT rotation procedure, and Secrets Manager secret inventory.
Add autodiscovery and token refresh crons
2026-09-30 04:13:11 +00:00 · 2026-05-11 19:03:42 -04:00 · 2026-05-11 18:46:25 -04:00
2 changed files with 157 additions and 5 deletions
--- a/README.md
+++ b/README.md
@ -4,12 +4,13 @@ Self-hosted Forgejo git server for archiving GitHub repos and mirroring active o

 ## Architecture

- **EC2**: t4g.small (arm64), Amazon Linux 2023, 50GB gp3 EBS
+- **EC2**: t4g.small (arm64), Amazon Linux 2023, 50GB gp3 EBS — instance `i-0d3005fb3c36124cd`
 - **Network**: Private subnet (us-east-1a), behind `seahaven-com` ALB for SSL termination
- **DNS**: `forgejo.seahaven.com` (Route53 alias → ALB)
+- **DNS**: `forgejo.seahaven.com` — Route53 alias record pointing to the `seahaven-com` ALB (not a direct A record)
 - **TLS**: Wildcard cert on ALB, HTTP internally on port 3000
- **Backup**: Nightly EBS snapshots via DLM, 7-day retention
+- **Backup**: Nightly `forgejo dump` to S3 + EBS snapshots via DLM (see [S3 Backups](#s3-backups))
 - **Admin access**: SSM Session Manager (no SSH port exposed)
+- **CI/CD**: GitHub Actions with OIDC role `githubdeploy-forgejo`

 ### Ports

@ -19,12 +20,71 @@ Self-hosted Forgejo git server for archiving GitHub repos and mirroring active o
 | 3000 | HTTP | ALB → instance | Internal traffic from ALB |
 | 2222 | SSH | VPC + VPN | Git SSH operations |

+## S3 Backups
+
+A nightly `forgejo dump` runs at 5:00 UTC and uploads the archive to `s3://forgejo-backups-328440206208`.
+
+**S3 lifecycle policy:**
+
+| Phase | Duration |
+|-------|----------|
+| Standard | First 30 days |
+| Glacier | Days 31–365 |
+| Expired | After 365 days |
+
+EBS snapshots are managed separately by DLM and run nightly at 6:00 UTC with a 7-day retention window.
+
+To test the backup manually:
+
+```bash
+sudo /usr/local/bin/forgejo-backup.sh
+```
+
+## Autodiscovery
+
+An hourly cron job checks the `Sea-Haven-Industries` GitHub org for new repositories and mirrors them into Forgejo automatically.
+
+- **Active repos** are created as mirrors (ongoing sync).
+- **Archived repos** are created as static one-time imports.
+- **Script**: `/usr/local/bin/forgejo-autodiscover.sh`
+- **Log**: `/var/log/forgejo-autodiscover.log`
+
+## Token Refresh
+
+A daily cron at 4:30 UTC reads the GitHub PAT from Secrets Manager (`forgejo/github-pat`) and updates the git remote URL on every mirror repository so credentials stay current.
+
+- **Script**: `/usr/local/bin/forgejo-refresh-tokens.sh`
+
+## PAT Rotation
+
+The GitHub personal access token used for mirroring is a fine-grained PAT scoped to `Sea-Haven-Industries` with **Contents: Read-only** permissions and a 1-year expiration. It is stored in Secrets Manager at `forgejo/github-pat`.
+
+To rotate:
+
+1. Create a new fine-grained PAT on GitHub with the same scope.
+2. Update the secret value in Secrets Manager (`forgejo/github-pat`).
+3. The daily token-refresh cron will pick it up automatically.
+
+To force immediate propagation:
+
+```bash
+sudo /usr/local/bin/forgejo-refresh-tokens.sh
+```
+
+## Secrets Manager
+
+| Secret | Purpose |
+|--------|---------|
+| `forgejo/admin-password` | Forgejo admin user password |
+| `forgejo/api-token` | Forgejo API token (used by autodiscovery and token refresh scripts) |
+| `forgejo/github-pat` | GitHub fine-grained PAT for mirroring |
+
 ## First-time setup

 After the stack deploys, connect via SSM and create the admin user:

 ```bash
-aws ssm start-session --target <instance-id>
+aws ssm start-session --target i-0d3005fb3c36124cd

 sudo -u forgejo /usr/local/bin/forgejo admin user create \
  --admin \
@ -62,7 +122,7 @@ CI/CD is handled by GitHub Actions — PRs run CI, merges to `main` deploy via t
 Update the `FORGEJO_VERSION` constant in `lib/forgejo-stack.ts` and deploy. This replaces the instance, so ensure the latest EBS snapshot is available for data recovery if needed. Alternatively, update in-place via SSM:

 ```bash
-aws ssm start-session --target <instance-id>
+aws ssm start-session --target i-0d3005fb3c36124cd

 sudo systemctl stop forgejo
 sudo curl -Lo /usr/local/bin/forgejo "https://codeberg.org/forgejo/forgejo/releases/download/v<NEW_VERSION>/forgejo-<NEW_VERSION>-linux-arm64"
--- a/lib/forgejo-stack.ts
+++ b/lib/forgejo-stack.ts
@ -52,6 +52,13 @@ export class ForgejoStack extends cdk.Stack {
      ],
    });

+    role.addToPolicy(new iam.PolicyStatement({
+      actions: ["secretsmanager:GetSecretValue"],
+      resources: [
+        `arn:aws:secretsmanager:us-east-1:328440206208:secret:forgejo/*`,
+      ],
+    }));
+
    const backupBucket = new s3.Bucket(this, "BackupBucket", {
      bucketName: "forgejo-backups-328440206208",
      encryption: s3.BucketEncryption.S3_MANAGED,
@ -161,6 +168,91 @@ export class ForgejoStack extends cdk.Stack {
      "",
      "echo '0 5 * * * root /usr/local/bin/forgejo-backup.sh >> /var/log/forgejo-backup.log 2>&1' > /etc/cron.d/forgejo-backup",
      "chmod 644 /etc/cron.d/forgejo-backup",
+      "",
+      "cat > /usr/local/bin/forgejo-autodiscover.sh << 'ADEOF'",
+      "#!/bin/bash",
+      "set -euo pipefail",
+      "GH_PAT=$(aws secretsmanager get-secret-value --secret-id forgejo/github-pat --query SecretString --output text --region us-east-1)",
+      "FORGEJO_TOKEN=$(aws secretsmanager get-secret-value --secret-id forgejo/api-token --query SecretString --output text --region us-east-1)",
+      "FORGEJO_URL=https://forgejo.seahaven.com/api/v1",
+      "GH_ORG=Sea-Haven-Industries",
+      "",
+      "gh_repos=$(curl -sf -H \"Authorization: token $GH_PAT\" \"https://api.github.com/orgs/$GH_ORG/repos?per_page=100&type=all\" | python3 -c \"",
+      "import sys, json",
+      "for r in json.load(sys.stdin):",
+      "    print(f\\\"{r['name']}\\\\t{r['archived']}\\\")",
+      "\")",
+      "",
+      "forgejo_repos=$(curl -sf -H \"Authorization: token $FORGEJO_TOKEN\" \"$FORGEJO_URL/repos/search?limit=100\" | python3 -c \"",
+      "import sys, json",
+      "data = json.load(sys.stdin)",
+      "repos = data.get('data', data) if isinstance(data, dict) else data",
+      "for r in repos:",
+      "    print(r['name'])",
+      "\")",
+      "",
+      "while IFS=$'\\t' read -r name archived; do",
+      "  if ! echo \"$forgejo_repos\" | grep -qx \"$name\"; then",
+      "    mirror=true",
+      "    [ \"$archived\" = \"True\" ] && mirror=false",
+      "    echo \"$(date -Is) Discovering: $name (mirror=$mirror)\"",
+      "    curl -sf -X POST \"$FORGEJO_URL/repos/migrate\" \\",
+      "      -H \"Authorization: token $FORGEJO_TOKEN\" \\",
+      "      -H \"Content-Type: application/json\" \\",
+      "      -d \"{",
+      "        \\\"clone_addr\\\": \\\"https://github.com/$GH_ORG/${name}.git\\\",",
+      "        \\\"auth_token\\\": \\\"${GH_PAT}\\\",",
+      "        \\\"repo_name\\\": \\\"${name}\\\",",
+      "        \\\"repo_owner\\\": \\\"adam\\\",",
+      "        \\\"service\\\": \\\"github\\\",",
+      "        \\\"mirror\\\": ${mirror},",
+      "        \\\"issues\\\": true,",
+      "        \\\"labels\\\": true,",
+      "        \\\"milestones\\\": true,",
+      "        \\\"pull_requests\\\": true,",
+      "        \\\"releases\\\": true,",
+      "        \\\"wiki\\\": true",
+      "      }\" > /dev/null",
+      "  fi",
+      "done <<< \"$gh_repos\"",
+      "ADEOF",
+      "chmod +x /usr/local/bin/forgejo-autodiscover.sh",
+      "",
+      "cat > /usr/local/bin/forgejo-refresh-tokens.sh << 'RTEOF'",
+      "#!/bin/bash",
+      "set -euo pipefail",
+      "GH_PAT=$(aws secretsmanager get-secret-value --secret-id forgejo/github-pat --query SecretString --output text --region us-east-1)",
+      "FORGEJO_TOKEN=$(aws secretsmanager get-secret-value --secret-id forgejo/api-token --query SecretString --output text --region us-east-1)",
+      "FORGEJO_URL=https://forgejo.seahaven.com/api/v1",
+      "REPO_ROOT=/var/lib/forgejo/data/repositories/adam",
+      "",
+      "export GIT_CONFIG_COUNT=1",
+      "export GIT_CONFIG_KEY_0=safe.directory",
+      "export GIT_CONFIG_VALUE_0='*'",
+      "",
+      "mirrors=$(curl -sf -H \"Authorization: token $FORGEJO_TOKEN\" \"$FORGEJO_URL/repos/search?limit=100\" | python3 -c \"",
+      "import sys, json",
+      "data = json.load(sys.stdin)",
+      "repos = data.get('data', data) if isinstance(data, dict) else data",
+      "for r in repos:",
+      "    if r.get('mirror', False):",
+      "        print(r['name'])",
+      "\")",
+      "",
+      "while read -r repo_name; do",
+      "  [ -z \"$repo_name\" ] && continue",
+      "  repo_dir=\"$REPO_ROOT/${repo_name}.git\"",
+      "  if [ -d \"$repo_dir\" ]; then",
+      "    new_url=\"https://${GH_PAT}@github.com/Sea-Haven-Industries/${repo_name}.git\"",
+      "    git -C \"$repo_dir\" remote set-url origin \"$new_url\" 2>/dev/null && echo \"$(date -Is) Refreshed: $repo_name\"",
+      "  fi",
+      "done <<< \"$mirrors\"",
+      "RTEOF",
+      "chmod +x /usr/local/bin/forgejo-refresh-tokens.sh",
+      "",
+      "printf '%s\\n' '0 * * * * root /usr/local/bin/forgejo-autodiscover.sh >> /var/log/forgejo-autodiscover.log 2>&1' > /etc/cron.d/forgejo-autodiscover",
+      "printf '%s\\n' '30 4 * * * root /usr/local/bin/forgejo-refresh-tokens.sh >> /var/log/forgejo-refresh-tokens.log 2>&1' > /etc/cron.d/forgejo-refresh-tokens",
+      "chmod 644 /etc/cron.d/forgejo-autodiscover /etc/cron.d/forgejo-refresh-tokens",
    );

    const instance = new ec2.Instance(this, "Instance", {