AWS SAM application that ingests payment CSVs, syncs check data with Bank of America CashPro APIs, processes Gusto payroll confirmation emails into Slack notifications, surfaces an outstanding-payments dashboard in Slack, and routes expense approvals through a Slack reaction-driven workflow.
- **ProcessPayrollEmail** — Lambda triggered by S3 (inbound email) and SQS (batch timer). SES receives Gusto payroll emails at `payroll@int.seahaven.com`, stores them to S3, and this Lambda parses the email body, extracts financial data, and posts a combined Slack notification (employee payroll + contractor payments) after a 10-minute batching window. Runs outside VPC.
- **ProcessPaymentCsv** — Lambda triggered by S3 CSV upload. Parses Stampli payment exports, upserts to DynamoDB, and submits new/cancelled checks to the CashPro Check Management API.
- **FetchBoaTransactions** — Scheduled Lambda. Weekdays 9am ET it calls the CashPro **previous-day** Transaction Inquiry (authoritative sweep, trailing 7 days); weekdays at 16:00/19:00/22:00 UTC (~12/3/6pm ET, fixed-UTC so it drifts an hour in winter) it calls the **current-day** inquiry for same-day visibility (EventBridge `Input: {"endpoint":"current-day"}`, today-only, staleness sweep skipped). Every run archives the exact raw response to the `seahaven-payments-boa-raw-*` bucket (`raw/<endpoint>/<fromDate>_<toDate>/<runAt>.json`, SSE-S3, 730-day lifecycle, PutObject-only grant; Retain-protected — decommission goes through the CFN decommission runbook) and upserts per-date `boa_balance#<asOfDate>#<endpoint>` snapshots (latest-wins on `run_at`, no TTL) from the Summary rows. Classifies each transaction and reconciles onto DynamoDB payment records. Event payload: `{fromDate?, toDate?, endpoint?}` (endpoint allowlisted and validated; unknown fields ignored). Intraday runs are disable-able as a unit via the `IntradaySchedule` rule. See [Bank reconciliation](#bank-reconciliation-fetchboatransactions).
- **SlackAppHome** — Lambda behind API Gateway (`POST /slack/events`). Verifies the Slack signing secret (HMAC-SHA256, 5-minute replay window) before processing, then renders the payments dashboard on the Slack App Home tab with outstanding aging buckets, drill-down modals, and an always-visible "Returned — Needs Action" queue (bank-returned payments awaiting a reissue/void decision, sorted oldest return first). Returned records are excluded from Outstanding totals; terminal voided-and-bounced records appear in neither (audit trail only).
- **ExpenseReceiver** — Lambda behind API Gateway (`POST /slack/expense-events`). Verifies the Slack signing secret (HMAC-SHA256), handles URL verification challenges, and async-invokes ExpenseProcessor. Runs outside VPC.
- **ExpenseProcessor** — Async Lambda invoked by ExpenseReceiver. Processes `:white_check_mark:` reactions to advance expense messages through a four-stage Slack channel pipeline: Submitted → Processed → Authorized → Matched. Runs outside VPC.
ProcessPaymentCsv, FetchBoaTransactions, and SlackAppHome run inside a VPC with a NAT Gateway for a static outbound IP (required by BoA IP whitelisting). ProcessPayrollEmail, ExpenseReceiver, and ExpenseProcessor run outside the VPC.
## Expense Approval Bot
Reaction-driven workflow that routes expense submissions through four Slack channels. A separate Slack app ("Expense Approval Bot") posts to a **Submitted** channel. Users react with :white_check_mark: to advance the message to the next stage.
**Channel pipeline:**
| Stage | Channel ID | Action on :white_check_mark: |
| Submitted | `C0AQ2AWLNEN` | Thread reply on original, copy to Processed |
| Processed | `C0APLSGABAB` | Delete from Processed, post to Authorized |
| Authorized | `C0AQ09CDJH4` | Delete from Authorized, post to Matched |
| Matched | `C0APYUM1JFP` | Terminal stage (no further routing) |
**Architecture:** Two Lambdas — ExpenseReceiver (HTTP endpoint, signature verification, async invoke) and ExpenseProcessor (business logic). This is the same receiver/processor pattern used for Slack's 3-second timeout requirement.
**Secrets (Secrets Manager):**
| Secret | Purpose |
|--------|---------|
| `payments-dashboard/expense-slack-token` | Slack Bot token for the Expense Approval Bot app |
| `payments-dashboard/expense-slack-signing-secret` | Slack signing secret for request verification |
Gusto sends payroll confirmation emails when payroll is run. A Gmail filter on adam@seahavenind.com auto-forwards emails from `automated@gusto.com` and `gustonoreply@gusto.com` to `payroll@int.seahaven.com`.
**Flow:** Gmail forward → SES receipt rule → S3 bucket → Lambda parses email → DynamoDB (pending) → SQS delay queue (10 min) → Lambda batches all pending items for that date → single Slack message → DynamoDB (notified)
**Deduplication:** Each email is deduplicated by DynamoDB key (`PAYROLL_EMAIL#employee#<date>` or `PAYROLL_EMAIL#contractor#<date>#<bank-suffix>`). The batch post is deduplicated by `PAYROLL_BATCH#<date>`. All items have a 90-day TTL.
**Classification.** The API's Detail rows carry the statement line in `detailText` (ACH `DES:PAYMENTS ID:PMT` text, return descriptions) and the posting date in `asOfDate` (ISO); `transactionType: "Summary"` rows are balance/total lines and classify as `summary` (skipped, counted). Only 475 check-paid rows omit `detailText` — their check number rides in `customerReference` (all-zeros references normalize to empty, never check number 0). Classification runs the empirically-enumerated BAI code map first, then the description-text classifiers:
| 266 | Return Item Credit | text decides (`ach_return` on PMT text, `check_return`/`electronic_return` on return text); bare rows fall back to `electronic_return`; unreadable text stays `unknown` (loud, no write) |
| 455 | Preauthorized ACH Debit | `ach_debit` via `DES:PAYMENTS` text; DES-less = third-party autopay → ignored |
A hard code-map event wins over description text; 266/455 carry *fallback* events consulted only when the text yields nothing. Unmapped codes on check-shaped transactions are logged (`console.error`) and counted in the run summary — never silently dropped.
**Matching.** Check events match on check number AND amount, evaluating all candidates (bank postings can drop/collapse digits on long check numbers). A number match with the WRONG amount never auto-resolves — it is the altered-check/collapsed-posting signal and goes to unmatched for human review. An unknown number falls back to an exact-amount match within checks issued in the last 120 days, and only when the posting's digits are a subsequence of the candidate's check number (or vice versa); return credits additionally require a bank-confirmed candidate. Zero or multiple fallback candidates means unmatched, recorded in the run summary with no write. Electronic returns (no check number) match by exact amount among bank-confirmed payments.
**ACH.** ACH is bank-confirmed too (#69): `processPaymentCsv` no longer auto-clears ACH on the send date (rows keep their Stampli status until the bank settles). ACH CCD lines match by the stored `pmt_id` first (reversals reuse the original `PMT <id>`; a `pmt_id` match with the wrong amount is the partial-reversal human case and goes to unmatched), then by the Stampli payment number embedded at the end of `PMT INFO` (internal spaces stripped, amount must agree), then by vendor + exact amount within `send_payment_on`−2..+14 days (candidates whose stored `pmt_id` differs are excluded; vendor prefix matching requires ≥ 10 normalized chars). A settled debit clears the record and persists `pmt_id`. A return credit whose `PMT <id>` attributes to no stored `pmt_id` is an unmatched alert ("unknown PMT id"); credits without a `PMT <id>` may fall back to a unique same-amount match against a bank-confirmed ACH whose `cleared_date` is within the prior 45 days.
**Staleness sweep.** Every run also flags never-bank-confirmed payments: ACH with no `clear_status` sent more than 16 days ago (listed in the summary, capped at 50, plus a total count) and checks issued more than 60 days ago with no `clear_status` (count only). Both alert via `console.error`.
**State transitions.** Every write sets BOTH `status` and `clear_status` (`clear_status` is bank truth; "Cleared without a subsequent return is permanent" keys off it):
| Return credit (even if currently Cleared) | `clear_status=Returned`, `returned_date`; `status` re-written unchanged (the CSV ladder has no Returned rung) |
| Second paid debit on a Returned check | Redeposit: back to Cleared with new dates |
| Return on a Stampli-voided check | Terminal voided-and-bounced (`clear_status=Returned`, cancel status preserved), counted separately |
Each applied event is appended to a `history` list attribute (`{event, date, bankRef, amount}`); identical replayed events are idempotent noops.
**Replay.** The handler accepts an optional payload `{"fromDate": "YYYY-MM-DD", "toDate": "YYYY-MM-DD"}` (strictly validated; `toDate` defaults to `fromDate`) for weekend/outage gap replays and BAI-code enumeration runs. With no payload it queries the trailing 7-day window (today−7 .. today−1) — self-healing across missed runs and holiday gaps; overlapping days are idempotent (event identity `{event, date, amount}`), and multi-day responses were verified pagination-free up to 9-day windows. Feed rows without a valid posting date (`asOfDate`, with `valueDate` as a fixture-era fallback) are never applied with a substituted date — they go to unmatched for review.
**Run summary.** Each run writes an append-only `boa_recon#<fromDate>_<toDate>#<runAt>` item (90-day TTL) with counts per classified event type, matched/applied/redeposit/voided-and-bounced/write-conflict totals, the unmatched check numbers and amounts (list capped at 50; full count kept), unknown BAI codes (capped at 20 distinct keys), and the staleness sweep results. Unmatched and unknown-code transactions also `console.error` (Slack alerting is tracked in #71). Payment writes are conditioned on the read snapshot's `status`/`clear_status` and retried once against a fresh read on conflict.
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `payments-dashboard` stack is represented there as a Mermaid subgraph.
- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)
All BoA and Slack credentials are stored in AWS Secrets Manager (per `engineering-handbook/secrets-and-config.md`). The Slack token is a plaintext secret; the two BoA secrets are JSON grouping each API's credentials:
`boa-account-number` is duplicated into both BoA secrets. Each Lambda is granted `secretsmanager:GetSecretValue` scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (`payments-dashboard/expense-slack-token`, `payments-dashboard/expense-slack-signing-secret`).
The `PaymentsDashboard` DynamoDB table (`AWS::DynamoDB::Table`, `TableName: PaymentsDashboard`, PK `pk` (S), CMK-encrypted) is **owned by this stack**, which is the sole authoritative writer.
**Consumer (read-only):** `seahaven-slack-bot` imports this table via `Table.fromTableName(...)` and reads it read-only (`grantReadData` plus an explicit `kms:Decrypt` grant on the shared CMK) from its `wo-po-lookup` Lambda, which backs the Bedrock agent's payment-lookup action group. The bot depends on:
- **Key schema:** PK `pk` (S) with the item format `payment#<check_number>`. It does `GetItem` by `pk` and a full-table `Scan` filtered `begins_with(pk, "payment#")`.
- **Encryption:** the shared customer-managed CMK (`/seahaven/dynamodb/cmk-arn`). Because the consumer imports the table by name, `grantReadData` does not carry KMS access; a change of CMK requires re-granting on the consumer side or every read fails with `kms:Decrypt AccessDenied` (INFRA-95 / M-3 precedent).
The table is imported by name, so there is no compile-time link between the stacks: any change to the table name, `pk` format, these attribute names, the encryption key, or the table's lifecycle policy will silently break the Bedrock agent at runtime. Coordinate such changes with `seahaven-slack-bot` before shipping (INFRA-138).
All CloudWatch alarms publish to the shared `site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as `notBreaching`. Each alarm evaluates a single 5-minute period.
**SQS dead-letter queues** (messages-present, Maximum > 0):
**DynamoDB** (`PaymentsDashboard` table, `TableName` dimension, Sum > 0): `payments-dashboard-table-read-throttle` (`ReadThrottleEvents`), `payments-dashboard-table-write-throttle` (`WriteThrottleEvents`). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. `SystemErrors` is intentionally not alarmed because it does not emit at the `TableName`-only dimension.
**API Gateway** (implicit HTTP API v2 `ServerlessHttpApi`, `ApiId` dimension): `payments-dashboard-api-5xx` (`5xx` Sum > 0), `payments-dashboard-api-4xx` (`4xx` Sum > 10, client-error noise floor), `payments-dashboard-api-latency-p99` (`Latency` p99 > 3000 ms).
The `BOA_BASE_URL` environment variable in `template.yaml` controls whether Lambdas hit production (`https://api.bofa.com`) or sandbox (`https://api-sb.bofa.com`). All other BoA config is read from Secrets Manager at runtime.
