# sh-openswe-traces LangSmith Bulk Export destination for the Open SWE deployment. A **storage-only** SAM stack — no compute. LangSmith runs the export on its own schedule and writes Parquet run/trace data into this bucket; we retain it for periodic auditing and prompt/instruction improvement (query with Athena). ## Why this exists LangSmith retains traces for ~14 days. To audit agent behavior over longer horizons and mine it for prompt improvements, we own the data in S3 rather than paying for extended LangSmith retention. Capture is LangSmith-native Bulk Export; this repo is just the destination and its access controls. ## Architecture ``` LangSmith (Bulk Export, scheduled LangSmith-side) │ s3:PutObject (IAM user access key, least-privilege) ▼ s3://sh-openswe-traces/langsmith/… SSE-KMS (alias/sh-openswe-traces) │ lifecycle: → DEEP_ARCHIVE @ 90d ▼ Athena / manual audit ``` | Resource | Name | Notes | |---|---|---| | S3 bucket | `sh-openswe-traces` | BPA all-on, SSE-KMS (default), versioned, TLS-only, `Retain` on delete | | Log bucket | `sh-openswe-traces-logs` | S3 server access logs (SSE-S3) for read attribution; logs expire 365d | | KMS CMK | `alias/sh-openswe-traces` | Rotation on; bucket default + writer encrypt through it | | IAM user | auto-named (tag `sh-openswe-langsmith-export`) | Write-only LangSmith writer; `PutObject` bucket-wide, no read/delete | | Secret | `sh-openswe/langsmith-export-s3` | Writer's access key (aws/secretsmanager key); populated post-deploy | The IAM user is **not** given an explicit name so the app stack deploys under `CAPABILITY_IAM`. It is referenced by ARN. ## Deploy Two stacks: - **`bootstrap.yaml`** → `sh-openswe-traces-bootstrap` — the CI IAM roles (OIDC deploy role + a least-privilege CFN exec role). Deployed **once, manually, under admin** (`CAPABILITY_NAMED_IAM`); rarely changes. Kept separate so CI never touches the shared `github-cfn-execution-role`, which is roles-only and can't create this stack's KMS key, secret, or IAM user. - **`template.yaml`** → `sh-openswe-traces` — the app (buckets, CMK, writer, secret). CI (`cd-sam.yaml`) deploys it on merge to `main`, assuming the OIDC deploy role and passing `sh-openswe-traces-cfn-exec-role` as `--role-arn`. ```bash # One-time bootstrap (admin): aws cloudformation deploy --template-file bootstrap.yaml \ --stack-name sh-openswe-traces-bootstrap --capabilities CAPABILITY_NAMED_IAM \ --region us-east-1 # App stack — CI does this on merge to main; for a local/admin deploy: cp samconfig.toml.example samconfig.toml sam validate --lint && sam build sam deploy --capabilities CAPABILITY_IAM --resolve-s3 ``` ### Wire CI (after bootstrap + repo exist) - Set repo secret **`AWS_DEPLOY_ROLE_ARN`** to the bootstrap `DeployRoleArn` output (`arn:aws:iam::328440206208:role/githubdeploy-sh-openswe-traces`). - `deploy.yaml` already passes `cfn-role-arn = sh-openswe-traces-cfn-exec-role`; the OIDC trust is pinned to this repo's `main` ref. ### Post-deploy: mint and store the writer access key The stack creates the IAM user and an empty secret; the access key is minted out-of-band so it never lands in CloudFormation state. **Run this only on a trusted single-user workstation, never in CI** — it handles a live credential. ```bash USER=$(aws cloudformation describe-stacks --stack-name sh-openswe-traces \ --query "Stacks[0].Outputs[?OutputKey=='ExportUserName'].OutputValue" --output text) KEY_JSON=$(aws iam create-access-key --user-name "$USER" \ | jq '{AccessKeyId: .AccessKey.AccessKeyId, SecretAccessKey: .AccessKey.SecretAccessKey}') # Pass the secret via stdin, not argv, so it never appears in the process table. aws secretsmanager put-secret-value \ --secret-id sh-openswe/langsmith-export-s3 \ --secret-string file:///dev/stdin <<<"$KEY_JSON" unset KEY_JSON ``` ### Configure LangSmith Bulk Export Driven by the LangSmith API (Plus/Enterprise only). Needs `LS_API_KEY` (LangSmith API key) and `LS_TENANT` (workspace id). Run **after** the writer key is minted/stored — the destination call validates by test-writing to the bucket. **1. Create the destination** (creds pulled from Secrets Manager, never pasted): ```bash CREDS=$(aws secretsmanager get-secret-value --secret-id sh-openswe/langsmith-export-s3 \ --query SecretString --output text) AKID=$(jq -r .AccessKeyId <<<"$CREDS"); SAK=$(jq -r .SecretAccessKey <<<"$CREDS") curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports/destinations' \ -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \ --data @- <' '' '' '' ) # all 4, or the subset you audit for PID in "${PROJECTS[@]}"; do curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports' \ -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \ --data @- </…`. ### Monitor exports ```bash # Every export in the workspace: id, project, schedule, status curl -sS 'https://api.smith.langchain.com/api/v1/bulk-exports' \ -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \ | jq -r '(.bulk_exports // .exports // .)[] | [.id, (.session_id // "all_experiments"), (if .interval_hours then "every \(.interval_hours)h" else "one-off" end), .status] | @tsv' | column -t ``` - A recurring schedule's own status is `IntervalScheduled`; the daily child exports it spawns have `CREATED` / `RUNNING` / `COMPLETED` / `FAILED` / `CANCELLED` / `TIMEDOUT` (child exports carry `source_bulk_export_id`). - One export's detail: `GET /api/v1/bulk-exports/` — its per-run rows: `GET /api/v1/bulk-exports//runs`. - **Stop a schedule:** `PATCH /api/v1/bulk-exports/` with `{"status":"Cancelled"}`. Already-spawned child exports must be cancelled separately, and a cancelled job can't be restarted — create a new one. ## Operations - **Rotate** the writer access key quarterly: `aws iam create-access-key`, update the secret + LangSmith destination (`PATCH`/recreate), then delete the old key. Key age is monitored account-wide by the Security Hub `ACCESS_KEYS_ROTATED` Config rule (flags at 90d) — rotation keeps it compliant. - **Audit**: point Athena at `s3://sh-openswe-traces/langsmith/` (Parquet). Objects older than 90 days are in Deep Archive — restore before querying. - **Read attribution**: object access is logged to `s3://sh-openswe-traces-logs/s3-access/` (S3 server access logging). - **Cost**: Deep Archive ≈ $1/TB/mo; expect the archive to dominate storage cost. ## Gotchas (from IAM cross-review) - **Writer needs `kms:Decrypt`.** SSE-KMS multipart uploads call `kms:Decrypt` at `CompleteMultipartUpload`; without it every multipart export fails `AccessDenied`. It's granted and safe — the writer has no `s3:GetObject`, so nothing to exfiltrate. - **No ACL headers.** The bucket is `BucketOwnerEnforced`; any PutObject carrying an ACL header is rejected with `AccessControlListNotSupported` (not fixable in policy). boto3/most SDKs send none by default — verify LangSmith's exporter likewise. - **Downstream readers need their own CMK grant.** An Athena/Glue role reading the Parquet needs explicit `kms:Decrypt` (+ `kms:GenerateDataKey`) on `alias/sh-openswe-traces` — see `reference_kms_cmk_grant_migration`. ## Security Traces can contain source code and secrets surfaced in tool I/O. Controls: SSE-KMS at rest (customer-managed CMK, bucket default), versioning (overwrite recovery), Block Public Access, TLS-only bucket policy, a write-only least-privilege writer (bucket-wide `PutObject` + `kms:Decrypt` gated to `kms:ViaService=s3`, no read/delete), the credential secret on a separate managed key, and S3 access logging. CI deploys via a dedicated least-privilege exec role (see `bootstrap.yaml`), not the shared execution role. The IAM surface passed GPT-4.1 cross-review and a `/sh-security-review` fan-out + proof-or-kill verifier (no blocking findings). Deferred, non-blocking: CloudTrail S3 data-events (org trail carries none; access logging covers attribution for now).