sh-openswe-traces/README.md

# sh-openswe-traces

LangSmith Bulk Export destination for the Open SWE deployment. A **storage-only**
stack — no compute. LangSmith runs the export on its own schedule and writes
Parquet run/trace data into this bucket; we retain it for periodic auditing and
prompt/instruction improvement (query with Athena).

Owned by **HCP Terraform** in seahaven-prod (`sh-openswe-traces-prod`). Migrated
from mgmt SAM under [PLAT-73](https://seahaven.atlassian.net/browse/PLAT-73).

## Why this exists

LangSmith retains traces for ~14 days. To audit agent behavior over longer
horizons and mine it for prompt improvements, we own the data in S3 rather than
paying for extended LangSmith retention. Capture is LangSmith-native Bulk Export;
this repo is just the destination and its access controls.

## Architecture

```
LangSmith (Bulk Export, scheduled LangSmith-side)
        │  s3:PutObject  (IAM user access key, least-privilege)
        ▼
  s3://sh-openswe-traces-<account-id>/langsmith/…
        │  SSE-KMS (alias/sh-openswe-traces)
        │  lifecycle: → DEEP_ARCHIVE @ 90d
        ▼
  Athena / manual audit
```

| Resource | Name | Notes |
|---|---|---|
| S3 bucket | `sh-openswe-traces-<account-id>` | BPA all-on, SSE-KMS (default), versioned, TLS-only, `prevent_destroy` |
| Log bucket | `sh-openswe-traces-logs-<account-id>` | S3 server access logs (SSE-S3); logs expire 365d |
| KMS CMK | `alias/sh-openswe-traces` | Rotation on; bucket default + writer encrypt through it |
| IAM user | `sh-openswe-langsmith-export` | Write-only LangSmith writer; `PutObject` bucket-wide, no read/delete |
| Secret | `sh-openswe/langsmith-export-s3` | Writer's access key (aws/secretsmanager key); populated post-apply |

Account-suffixed bucket names avoid global S3 name collision with the retired mgmt
buckets during cutover.

## Deploy (HCP Terraform)

Workspace: `sh-openswe-traces-prod` (org `seahaven`, project `seahaven-prod`).
Working directory: `terraform/`. Apply method: **Manual** until sealed.
Roles: `hcptf-sh-openswe-traces-plan` / `hcptf-sh-openswe-traces` (org-baseline
terraform-substrate).

```bash
cd terraform
terraform init
terraform plan
# First apply is Manual from the HCP UI (or terraform apply after plan confirm).
```

If the secret shell was created out-of-band (checklist step 1), import before the
first apply that manages it:

```bash
terraform import aws_secretsmanager_secret.export_key \
  'arn:aws:secretsmanager:us-east-1:011934824531:secret:sh-openswe/langsmith-export-s3-OQoqqU'
```

### Workspace env (once)

Workspace-level only (never project-scoped variable sets):

- `TFC_AWS_PROVIDER_AUTH=true`
- `TFC_AWS_PLAN_ROLE_ARN=arn:aws:iam::011934824531:role/hcptf-sh-openswe-traces-plan`
- `TFC_AWS_APPLY_ROLE_ARN=arn:aws:iam::011934824531:role/hcptf-sh-openswe-traces`

### Post-apply: mint and store the writer access key

The stack creates the IAM user and an empty secret; the access key is minted
out-of-band so it never lands in Terraform state. **Run this only on a trusted
single-user workstation, never in CI** — it handles a live credential.

```bash
USER=$(terraform -chdir=terraform output -raw export_user_name)

KEY_JSON=$(aws iam create-access-key --user-name "$USER" --profile seahaven-prod \
  | jq '{AccessKeyId: .AccessKey.AccessKeyId, SecretAccessKey: .AccessKey.SecretAccessKey}')

# Pass the secret via stdin, not argv, so it never appears in the process table.
# Strip trailing newlines — a trailing \n breaks HTTP headers at runtime.
printf '%s' "$KEY_JSON" | aws secretsmanager put-secret-value \
  --secret-id sh-openswe/langsmith-export-s3 \
  --secret-string file:///dev/stdin \
  --profile seahaven-prod \
  --region us-east-1

unset KEY_JSON
```

### Configure LangSmith Bulk Export

Driven by the LangSmith API (Plus/Enterprise only). Needs `LS_API_KEY` (LangSmith API
key) and `LS_TENANT` (workspace id). Run **after** the writer key is minted/stored — the
destination call validates by test-writing to the bucket.

**1. Create the destination** (creds pulled from Secrets Manager, never pasted):

```bash
BUCKET=$(terraform -chdir=terraform output -raw bucket_name)
CREDS=$(aws secretsmanager get-secret-value --secret-id sh-openswe/langsmith-export-s3 \
  --profile seahaven-prod --region us-east-1 \
  --query SecretString --output text)
AKID=$(jq -r .AccessKeyId <<<"$CREDS"); SAK=$(jq -r .SecretAccessKey <<<"$CREDS")

curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports/destinations' \
  -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
  --data @- <<JSON | jq .
{ "destination_type": "s3", "display_name": "sh-openswe-traces us-east-1",
  "config": { "bucket_name": "$BUCKET", "prefix": "langsmith", "region": "us-east-1" },
  "credentials": { "access_key_id": "$AKID", "secret_access_key": "$SAK" } }
JSON
```

`display_name` must match `^[a-zA-Z0-9\-_ ']+$` (no parens); omit `endpoint_url` (S3-native,
not GCS/MinIO). Save the returned destination `id`. LangSmith writes header-less, so
bucket-default SSE-KMS encrypts every object under the CMK (confirm via `head-object`).

**2. One scheduled export per tracing project** (this workspace has 4). Get project UUIDs
from `GET /api/v1/sessions`, then:

```bash
export LS_DEST_ID='<destination id>'
PROJECTS=( '<uuid-1>' '<uuid-2>' '<uuid-3>' '<uuid-4>' )   # all 4, or the subset you audit
for PID in "${PROJECTS[@]}"; do
  curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports' \
    -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
    --data @- <<JSON | jq '{id, session_id, status}'
{ "bulk_export_destination_id": "$LS_DEST_ID", "session_id": "$PID",
  "start_time": "2026-06-26T00:00:00Z", "interval_hours": 24, "format_version": "v2_beta" }
JSON
done
```

`start_time` ~14d back backfills each project's retained window, then it continues daily.
Keep `inputs`/`outputs` (omit `export_fields`) — they're the point of the audit. Data lands
partitioned per project: `langsmith/export_id=…/…/session_id=<id>/…`.

### Monitor exports

```bash
# Every export in the workspace: id, project, schedule, status
curl -sS 'https://api.smith.langchain.com/api/v1/bulk-exports' \
  -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
  | jq -r '(.bulk_exports // .exports // .)[]
           | [.id,
              (.session_id // "all_experiments"),
              (if .interval_hours then "every \(.interval_hours)h" else "one-off" end),
              .status] | @tsv' | column -t
```

- A recurring schedule's own status is `IntervalScheduled`; the daily child exports it
  spawns have `CREATED` / `RUNNING` / `COMPLETED` / `FAILED` / `CANCELLED` / `TIMEDOUT`
  (child exports carry `source_bulk_export_id`).
- One export's detail: `GET /api/v1/bulk-exports/<id>` — its per-run rows:
  `GET /api/v1/bulk-exports/<id>/runs`.
- **Stop a schedule:** `PATCH /api/v1/bulk-exports/<id>` with `{"status":"Cancelled"}`.
  Already-spawned child exports must be cancelled separately, and a cancelled job can't be
  restarted — create a new one.

## Operations

- **Rotate** the writer access key quarterly: `aws iam create-access-key`, update the
  secret + LangSmith destination (`PATCH`/recreate), then delete the old key. Key age is
  monitored account-wide by the Security Hub `ACCESS_KEYS_ROTATED` Config rule (flags at
  90d) — rotation keeps it compliant.
- **Audit**: point Athena at `s3://sh-openswe-traces-<account-id>/langsmith/` (Parquet).
  Objects older than 90 days are in Deep Archive — restore before querying.
- **Read attribution**: object access is logged to
  `s3://sh-openswe-traces-logs-<account-id>/s3-access/` (S3 server access logging).
- **Cost**: Deep Archive ≈ $1/TB/mo; expect the archive to dominate storage cost.

## Gotchas (from IAM cross-review)

- **Writer needs `kms:Decrypt`.** SSE-KMS multipart uploads call `kms:Decrypt` at
  `CompleteMultipartUpload`; without it every multipart export fails `AccessDenied`.
  It's granted and safe — the writer has no `s3:GetObject`, so nothing to exfiltrate.
- **No ACL headers.** The bucket is `BucketOwnerEnforced`; any PutObject carrying an
  ACL header is rejected with `AccessControlListNotSupported` (not fixable in policy).
  boto3/most SDKs send none by default — verify LangSmith's exporter likewise.
- **Downstream readers need their own CMK grant.** An Athena/Glue role reading the
  Parquet needs explicit `kms:Decrypt` (+ `kms:GenerateDataKey`) on
  `alias/sh-openswe-traces` — see `reference_kms_cmk_grant_migration`.

## Security

Traces can contain source code and secrets surfaced in tool I/O. Controls: SSE-KMS at
rest (customer-managed CMK, bucket default), versioning (overwrite recovery), Block Public
Access, TLS-only bucket policy, a write-only least-privilege writer (bucket-wide `PutObject`
+ `kms:Decrypt` gated to `kms:ViaService=s3`, no read/delete), the credential secret on a
separate managed key, and S3 access logging.

SAM `template.yaml` / `bootstrap.yaml` and the frozen GitHub `deploy.yaml` are retained
only until mgmt cutover completes; do not re-enable SAM CD.

Deferred, non-blocking: CloudTrail S3 data-events (org trail carries none; access logging
covers attribution for now).