sh-openswe-traces/README.md

# sh-openswe-traces

LangSmith Bulk Export destination for the Open SWE deployment. A **storage-only**
SAM stack — no compute. LangSmith runs the export on its own schedule and writes
Parquet run/trace data into this bucket; we retain it for periodic auditing and
prompt/instruction improvement (query with Athena).

## Why this exists

LangSmith retains traces for ~14 days. To audit agent behavior over longer
horizons and mine it for prompt improvements, we own the data in S3 rather than
paying for extended LangSmith retention. Capture is LangSmith-native Bulk Export;
this repo is just the destination and its access controls.

## Architecture

```
LangSmith (Bulk Export, scheduled LangSmith-side)
        │  s3:PutObject  (IAM user access key, least-privilege)
        ▼
  s3://sh-openswe-traces/langsmith/…    SSE-KMS (alias/sh-openswe-traces)
        │  lifecycle: → DEEP_ARCHIVE @ 90d
        ▼
  Athena / manual audit
```

| Resource | Name | Notes |
|---|---|---|
| S3 bucket | `sh-openswe-traces` | BPA all-on, SSE-KMS (default), versioned, TLS-only, `Retain` on delete |
| Log bucket | `sh-openswe-traces-logs` | S3 server access logs (SSE-S3) for read attribution; logs expire 365d |
| KMS CMK | `alias/sh-openswe-traces` | Rotation on; bucket default + writer encrypt through it |
| IAM user | auto-named (tag `sh-openswe-langsmith-export`) | Write-only LangSmith writer; `PutObject` bucket-wide, no read/delete |
| Secret | `sh-openswe/langsmith-export-s3` | Writer's access key (aws/secretsmanager key); populated post-deploy |

The IAM user is **not** given an explicit name so the app stack deploys under
`CAPABILITY_IAM`. It is referenced by ARN.

## Deploy

Two stacks:

- **`bootstrap.yaml`** → `sh-openswe-traces-bootstrap` — the CI IAM roles (OIDC deploy
  role + a least-privilege CFN exec role). Deployed **once, manually, under admin**
  (`CAPABILITY_NAMED_IAM`); rarely changes. Kept separate so CI never touches the shared
  `github-cfn-execution-role`, which is roles-only and can't create this stack's KMS key,
  secret, or IAM user.
- **`template.yaml`** → `sh-openswe-traces` — the app (buckets, CMK, writer, secret). CI
  (`cd-sam.yaml`) deploys it on merge to `main`, assuming the OIDC deploy role and passing
  `sh-openswe-traces-cfn-exec-role` as `--role-arn`.

```bash
# One-time bootstrap (admin):
aws cloudformation deploy --template-file bootstrap.yaml \
  --stack-name sh-openswe-traces-bootstrap --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

# App stack — CI does this on merge to main; for a local/admin deploy:
cp samconfig.toml.example samconfig.toml
sam validate --lint && sam build
sam deploy --capabilities CAPABILITY_IAM --resolve-s3
```

### Wire CI (after bootstrap + repo exist)

- Set repo secret **`AWS_DEPLOY_ROLE_ARN`** to the bootstrap `DeployRoleArn` output
  (`arn:aws:iam::328440206208:role/githubdeploy-sh-openswe-traces`).
- `deploy.yaml` already passes `cfn-role-arn = sh-openswe-traces-cfn-exec-role`; the OIDC
  trust is pinned to this repo's `main` ref.

### Post-deploy: mint and store the writer access key

The stack creates the IAM user and an empty secret; the access key is minted
out-of-band so it never lands in CloudFormation state. **Run this only on a trusted
single-user workstation, never in CI** — it handles a live credential.

```bash
USER=$(aws cloudformation describe-stacks --stack-name sh-openswe-traces \
  --query "Stacks[0].Outputs[?OutputKey=='ExportUserName'].OutputValue" --output text)

KEY_JSON=$(aws iam create-access-key --user-name "$USER" \
  | jq '{AccessKeyId: .AccessKey.AccessKeyId, SecretAccessKey: .AccessKey.SecretAccessKey}')

# Pass the secret via stdin, not argv, so it never appears in the process table.
aws secretsmanager put-secret-value \
  --secret-id sh-openswe/langsmith-export-s3 \
  --secret-string file:///dev/stdin <<<"$KEY_JSON"

unset KEY_JSON
```

### Configure LangSmith Bulk Export

Driven by the LangSmith API (Plus/Enterprise only). Needs `LS_API_KEY` (LangSmith API
key) and `LS_TENANT` (workspace id). Run **after** the writer key is minted/stored — the
destination call validates by test-writing to the bucket.

**1. Create the destination** (creds pulled from Secrets Manager, never pasted):

```bash
CREDS=$(aws secretsmanager get-secret-value --secret-id sh-openswe/langsmith-export-s3 \
  --query SecretString --output text)
AKID=$(jq -r .AccessKeyId <<<"$CREDS"); SAK=$(jq -r .SecretAccessKey <<<"$CREDS")

curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports/destinations' \
  -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
  --data @- <<JSON | jq .
{ "destination_type": "s3", "display_name": "sh-openswe-traces us-east-1",
  "config": { "bucket_name": "sh-openswe-traces", "prefix": "langsmith", "region": "us-east-1" },
  "credentials": { "access_key_id": "$AKID", "secret_access_key": "$SAK" } }
JSON
```

`display_name` must match `^[a-zA-Z0-9\-_ ']+$` (no parens); omit `endpoint_url` (S3-native,
not GCS/MinIO). Save the returned destination `id`. LangSmith writes header-less, so
bucket-default SSE-KMS encrypts every object under the CMK (confirm via `head-object`).

**2. One scheduled export per tracing project** (this workspace has 4). Get project UUIDs
from `GET /api/v1/sessions`, then:

```bash
export LS_DEST_ID='<destination id>'
PROJECTS=( '<uuid-1>' '<uuid-2>' '<uuid-3>' '<uuid-4>' )   # all 4, or the subset you audit
for PID in "${PROJECTS[@]}"; do
  curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports' \
    -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
    --data @- <<JSON | jq '{id, session_id, status}'
{ "bulk_export_destination_id": "$LS_DEST_ID", "session_id": "$PID",
  "start_time": "2026-06-26T00:00:00Z", "interval_hours": 24, "format_version": "v2_beta" }
JSON
done
```

`start_time` ~14d back backfills each project's retained window, then it continues daily.
Keep `inputs`/`outputs` (omit `export_fields`) — they're the point of the audit. Data lands
partitioned per project: `langsmith/export_id=…/…/session_id=<id>/…`.

### Monitor exports

```bash
# Every export in the workspace: id, project, schedule, status
curl -sS 'https://api.smith.langchain.com/api/v1/bulk-exports' \
  -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
  | jq -r '(.bulk_exports // .exports // .)[]
           | [.id,
              (.session_id // "all_experiments"),
              (if .interval_hours then "every \(.interval_hours)h" else "one-off" end),
              .status] | @tsv' | column -t
```

- A recurring schedule's own status is `IntervalScheduled`; the daily child exports it
  spawns have `CREATED` / `RUNNING` / `COMPLETED` / `FAILED` / `CANCELLED` / `TIMEDOUT`
  (child exports carry `source_bulk_export_id`).
- One export's detail: `GET /api/v1/bulk-exports/<id>` — its per-run rows:
  `GET /api/v1/bulk-exports/<id>/runs`.
- **Stop a schedule:** `PATCH /api/v1/bulk-exports/<id>` with `{"status":"Cancelled"}`.
  Already-spawned child exports must be cancelled separately, and a cancelled job can't be
  restarted — create a new one.

## Operations

- **Rotate** the writer access key quarterly: `aws iam create-access-key`, update the
  secret + LangSmith destination (`PATCH`/recreate), then delete the old key. Key age is
  monitored account-wide by the Security Hub `ACCESS_KEYS_ROTATED` Config rule (flags at
  90d) — rotation keeps it compliant.
- **Audit**: point Athena at `s3://sh-openswe-traces/langsmith/` (Parquet). Objects older
  than 90 days are in Deep Archive — restore before querying.
- **Read attribution**: object access is logged to `s3://sh-openswe-traces-logs/s3-access/`
  (S3 server access logging).
- **Cost**: Deep Archive ≈ $1/TB/mo; expect the archive to dominate storage cost.

## Gotchas (from IAM cross-review)

- **Writer needs `kms:Decrypt`.** SSE-KMS multipart uploads call `kms:Decrypt` at
  `CompleteMultipartUpload`; without it every multipart export fails `AccessDenied`.
  It's granted and safe — the writer has no `s3:GetObject`, so nothing to exfiltrate.
- **No ACL headers.** The bucket is `BucketOwnerEnforced`; any PutObject carrying an
  ACL header is rejected with `AccessControlListNotSupported` (not fixable in policy).
  boto3/most SDKs send none by default — verify LangSmith's exporter likewise.
- **Downstream readers need their own CMK grant.** An Athena/Glue role reading the
  Parquet needs explicit `kms:Decrypt` (+ `kms:GenerateDataKey`) on
  `alias/sh-openswe-traces` — see `reference_kms_cmk_grant_migration`.

## Security

Traces can contain source code and secrets surfaced in tool I/O. Controls: SSE-KMS at
rest (customer-managed CMK, bucket default), versioning (overwrite recovery), Block Public
Access, TLS-only bucket policy, a write-only least-privilege writer (bucket-wide `PutObject`
+ `kms:Decrypt` gated to `kms:ViaService=s3`, no read/delete), the credential secret on a
separate managed key, and S3 access logging. CI deploys via a dedicated least-privilege
exec role (see `bootstrap.yaml`), not the shared execution role.

The IAM surface passed GPT-4.1 cross-review and a `/sh-security-review` fan-out +
proof-or-kill verifier (no blocking findings). Deferred, non-blocking: CloudTrail S3
data-events (org trail carries none; access logging covers attribution for now).