sh-openswe-traces

LangSmith Bulk Export destination for the Open SWE deployment. A storage-only stack — no compute. LangSmith runs the export on its own schedule and writes Parquet run/trace data into this bucket; we retain it for periodic auditing and prompt/instruction improvement (query with Athena).

Owned by HCP Terraform in seahaven-prod (sh-openswe-traces-prod). Migrated from mgmt SAM under PLAT-73.

Why this exists

LangSmith retains traces for ~14 days. To audit agent behavior over longer horizons and mine it for prompt improvements, we own the data in S3 rather than paying for extended LangSmith retention. Capture is LangSmith-native Bulk Export; this repo is just the destination and its access controls.

Architecture

LangSmith (Bulk Export, scheduled LangSmith-side)
        │  s3:PutObject  (IAM user access key, least-privilege)
        ▼
  s3://sh-openswe-traces-<account-id>/langsmith/…
        │  SSE-KMS (alias/sh-openswe-traces)
        │  lifecycle: → DEEP_ARCHIVE @ 90d
        ▼
  Athena / manual audit

Resource	Name	Notes
S3 bucket	sh-openswe-traces-<account-id>	BPA all-on, SSE-KMS (default), versioned, TLS-only, prevent_destroy
Log bucket	sh-openswe-traces-logs-<account-id>	S3 server access logs (SSE-S3); logs expire 365d
KMS CMK	alias/sh-openswe-traces	Rotation on; bucket default + writer encrypt through it
IAM user	sh-openswe-langsmith-export	Write-only LangSmith writer; PutObject bucket-wide, no read/delete
Secret	sh-openswe/langsmith-export-s3	Writer's access key (aws/secretsmanager key); populated post-apply

Account-suffixed bucket names avoid global S3 name collision with the retired mgmt buckets during cutover.

Deploy (HCP Terraform)

Workspace: sh-openswe-traces-prod (org seahaven, project seahaven-prod). Working directory: terraform/. Apply method: Manual until sealed. Roles: hcptf-sh-openswe-traces-plan / hcptf-sh-openswe-traces (org-baseline terraform-substrate).

cd terraform
terraform init
terraform plan
# First apply is Manual from the HCP UI (or terraform apply after plan confirm).

If the secret shell was created out-of-band (checklist step 1), import before the first apply that manages it:

terraform import aws_secretsmanager_secret.export_key \
  'arn:aws:secretsmanager:us-east-1:011934824531:secret:sh-openswe/langsmith-export-s3-OQoqqU'

Workspace env (once)

Workspace-level only (never project-scoped variable sets):

• TFC_AWS_PROVIDER_AUTH=true
• TFC_AWS_PLAN_ROLE_ARN=arn:aws:iam::011934824531:role/hcptf-sh-openswe-traces-plan
• TFC_AWS_APPLY_ROLE_ARN=arn:aws:iam::011934824531:role/hcptf-sh-openswe-traces

Post-apply: mint and store the writer access key

The stack creates the IAM user and an empty secret; the access key is minted out-of-band so it never lands in Terraform state. Run this only on a trusted single-user workstation, never in CI — it handles a live credential.

USER=$(terraform -chdir=terraform output -raw export_user_name)

KEY_JSON=$(aws iam create-access-key --user-name "$USER" --profile seahaven-prod \
  | jq '{AccessKeyId: .AccessKey.AccessKeyId, SecretAccessKey: .AccessKey.SecretAccessKey}')

# Pass the secret via stdin, not argv, so it never appears in the process table.
# Strip trailing newlines — a trailing \n breaks HTTP headers at runtime.
printf '%s' "$KEY_JSON" | aws secretsmanager put-secret-value \
  --secret-id sh-openswe/langsmith-export-s3 \
  --secret-string file:///dev/stdin \
  --profile seahaven-prod \
  --region us-east-1

unset KEY_JSON

Configure LangSmith Bulk Export

Driven by the LangSmith API (Plus/Enterprise only). Needs LS_API_KEY (LangSmith API key) and LS_TENANT (workspace id). Run after the writer key is minted/stored — the destination call validates by test-writing to the bucket.

1. Create the destination (creds pulled from Secrets Manager, never pasted):

BUCKET=$(terraform -chdir=terraform output -raw bucket_name)
CREDS=$(aws secretsmanager get-secret-value --secret-id sh-openswe/langsmith-export-s3 \
  --profile seahaven-prod --region us-east-1 \
  --query SecretString --output text)
AKID=$(jq -r .AccessKeyId <<<"$CREDS"); SAK=$(jq -r .SecretAccessKey <<<"$CREDS")

curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports/destinations' \
  -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
  --data @- <<JSON | jq .
{ "destination_type": "s3", "display_name": "sh-openswe-traces us-east-1",
  "config": { "bucket_name": "$BUCKET", "prefix": "langsmith", "region": "us-east-1" },
  "credentials": { "access_key_id": "$AKID", "secret_access_key": "$SAK" } }
JSON

display_name must match ^[a-zA-Z0-9\-_ ']+$ (no parens); omit endpoint_url (S3-native, not GCS/MinIO). Save the returned destination id. LangSmith writes header-less, so bucket-default SSE-KMS encrypts every object under the CMK (confirm via head-object).

2. One scheduled export per tracing project (this workspace has 4). Get project UUIDs from GET /api/v1/sessions, then:

export LS_DEST_ID='<destination id>'
PROJECTS=( '<uuid-1>' '<uuid-2>' '<uuid-3>' '<uuid-4>' )   # all 4, or the subset you audit
for PID in "${PROJECTS[@]}"; do
  curl -sS -X POST 'https://api.smith.langchain.com/api/v1/bulk-exports' \
    -H 'Content-Type: application/json' -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
    --data @- <<JSON | jq '{id, session_id, status}'
{ "bulk_export_destination_id": "$LS_DEST_ID", "session_id": "$PID",
  "start_time": "2026-06-26T00:00:00Z", "interval_hours": 24, "format_version": "v2_beta" }
JSON
done

start_time ~14d back backfills each project's retained window, then it continues daily. Keep inputs/outputs (omit export_fields) — they're the point of the audit. Data lands partitioned per project: langsmith/export_id=…/…/session_id=<id>/….

Monitor exports

# Every export in the workspace: id, project, schedule, status
curl -sS 'https://api.smith.langchain.com/api/v1/bulk-exports' \
  -H "X-API-Key: $LS_API_KEY" -H "X-Tenant-Id: $LS_TENANT" \
  | jq -r '(.bulk_exports // .exports // .)[]
           | [.id,
              (.session_id // "all_experiments"),
              (if .interval_hours then "every \(.interval_hours)h" else "one-off" end),
              .status] | @tsv' | column -t

• A recurring schedule's own status is IntervalScheduled; the daily child exports it spawns have CREATED / RUNNING / COMPLETED / FAILED / CANCELLED / TIMEDOUT (child exports carry source_bulk_export_id).
• One export's detail: GET /api/v1/bulk-exports/<id> — its per-run rows: GET /api/v1/bulk-exports/<id>/runs.
• Stop a schedule: PATCH /api/v1/bulk-exports/<id> with {"status":"Cancelled"}. Already-spawned child exports must be cancelled separately, and a cancelled job can't be restarted — create a new one.

Operations

• Rotate the writer access key quarterly: aws iam create-access-key, update the secret + LangSmith destination (PATCH/recreate), then delete the old key. Key age is monitored account-wide by the Security Hub ACCESS_KEYS_ROTATED Config rule (flags at 90d) — rotation keeps it compliant.
• Audit: point Athena at s3://sh-openswe-traces-<account-id>/langsmith/ (Parquet). Objects older than 90 days are in Deep Archive — restore before querying.
• Read attribution: object access is logged to s3://sh-openswe-traces-logs-<account-id>/s3-access/ (S3 server access logging).
• Cost: Deep Archive ≈ $1/TB/mo; expect the archive to dominate storage cost.

Gotchas (from IAM cross-review)

• Writer needs kms:Decrypt. SSE-KMS multipart uploads call kms:Decrypt at CompleteMultipartUpload; without it every multipart export fails AccessDenied. It's granted and safe — the writer has no s3:GetObject, so nothing to exfiltrate.
• No ACL headers. The bucket is BucketOwnerEnforced; any PutObject carrying an ACL header is rejected with AccessControlListNotSupported (not fixable in policy). boto3/most SDKs send none by default — verify LangSmith's exporter likewise.
• Downstream readers need their own CMK grant. An Athena/Glue role reading the Parquet needs explicit kms:Decrypt (+ kms:GenerateDataKey) on alias/sh-openswe-traces — see reference_kms_cmk_grant_migration.

Security

Traces can contain source code and secrets surfaced in tool I/O. Controls: SSE-KMS at rest (customer-managed CMK, bucket default), versioning (overwrite recovery), Block Public Access, TLS-only bucket policy, a write-only least-privilege writer (bucket-wide PutObject

• kms:Decrypt gated to kms:ViaService=s3, no read/delete), the credential secret on a separate managed key, and S3 access logging.

SAM template.yaml / bootstrap.yaml and the frozen GitHub deploy.yaml are retained only until mgmt cutover completes; do not re-enable SAM CD.

Deferred, non-blocking: CloudTrail S3 data-events (org trail carries none; access logging covers attribution for now).