mirror of
https://github.com/Sea-Haven-Industries/payments-dashboard.git
synced 2026-09-30 07:43:12 +00:00
a6a8132a69
* feat(apphome): surface returned payments as a needs-action queue Bank-returned payments carry status "Cleared" (the CSV ladder has no Returned rung), so the status skip-list hid them from every App Home bucket; five Feb-Apr returns ($5,256.62) surfaced only via a manual statement reconciliation. Route on clear_status ahead of the status skip-list: non-canceled Returned records render in an always-visible action queue (oldest return first) plus a summary tile, and are excluded from Outstanding totals. Terminal voided-and-bounced records stay out of both (audit trail only). Membership keys off clear_status, not returned_date, so redeposited checks drop back out of the queue. Refs: #70 * docs(apphome): restore returned-queue README bullet dropped in rebase
182 lines
18 KiB
Markdown
182 lines
18 KiB
Markdown
# Payments Dashboard
|
||
|
||
|
||
|
||
|
||
|
||
|
||
AWS SAM application that ingests payment CSVs, syncs check data with Bank of America CashPro APIs, processes Gusto payroll confirmation emails into Slack notifications, surfaces an outstanding-payments dashboard in Slack, and routes expense approvals through a Slack reaction-driven workflow.
|
||
|
||
## Architecture
|
||
|
||
- **ProcessPayrollEmail** — Lambda triggered by S3 (inbound email) and SQS (batch timer). SES receives Gusto payroll emails at `payroll@int.seahaven.com`, stores them to S3, and this Lambda parses the email body, extracts financial data, and posts a combined Slack notification (employee payroll + contractor payments) after a 10-minute batching window. Runs outside VPC.
|
||
- **ProcessPaymentCsv** — Lambda triggered by S3 CSV upload. Parses Stampli payment exports, upserts to DynamoDB, and submits new/cancelled checks to the CashPro Check Management API.
|
||
- **FetchBoaTransactions** — Scheduled Lambda. Weekdays 9am ET it calls the CashPro **previous-day** Transaction Inquiry (authoritative sweep, trailing 7 days); weekdays at 16:00/19:00/22:00 UTC (~12/3/6pm ET, fixed-UTC so it drifts an hour in winter) it calls the **current-day** inquiry for same-day visibility (EventBridge `Input: {"endpoint":"current-day"}`, today-only, staleness sweep skipped). Every run archives the exact raw response to the `seahaven-payments-boa-raw-*` bucket (`raw/<endpoint>/<fromDate>_<toDate>/<runAt>.json`, SSE-S3, 730-day lifecycle, PutObject-only grant; Retain-protected — decommission goes through the CFN decommission runbook) and upserts per-date `boa_balance#<asOfDate>#<endpoint>` snapshots (latest-wins on `run_at`, no TTL) from the Summary rows. Classifies each transaction and reconciles onto DynamoDB payment records. Event payload: `{fromDate?, toDate?, endpoint?}` (endpoint allowlisted and validated; unknown fields ignored). Intraday runs are disable-able as a unit via the `IntradaySchedule` rule. See [Bank reconciliation](#bank-reconciliation-fetchboatransactions).
|
||
- **SlackAppHome** — Lambda behind API Gateway (`POST /slack/events`). Verifies the Slack signing secret (HMAC-SHA256, 5-minute replay window) before processing, then renders the payments dashboard on the Slack App Home tab with outstanding aging buckets, drill-down modals, and an always-visible "Returned — Needs Action" queue (bank-returned payments awaiting a reissue/void decision, sorted oldest return first). Returned records are excluded from Outstanding totals; terminal voided-and-bounced records appear in neither (audit trail only).
|
||
|
||
- **ExpenseReceiver** — Lambda behind API Gateway (`POST /slack/expense-events`). Verifies the Slack signing secret (HMAC-SHA256), handles URL verification challenges, and async-invokes ExpenseProcessor. Runs outside VPC.
|
||
- **ExpenseProcessor** — Async Lambda invoked by ExpenseReceiver. Processes `:white_check_mark:` reactions to advance expense messages through a four-stage Slack channel pipeline: Submitted → Processed → Authorized → Matched. Runs outside VPC.
|
||
|
||
ProcessPaymentCsv, FetchBoaTransactions, and SlackAppHome run inside a VPC with a NAT Gateway for a static outbound IP (required by BoA IP whitelisting). ProcessPayrollEmail, ExpenseReceiver, and ExpenseProcessor run outside the VPC.
|
||
|
||
## Expense Approval Bot
|
||
|
||
Reaction-driven workflow that routes expense submissions through four Slack channels. A separate Slack app ("Expense Approval Bot") posts to a **Submitted** channel. Users react with :white_check_mark: to advance the message to the next stage.
|
||
|
||
**Channel pipeline:**
|
||
|
||
| Stage | Channel ID | Action on :white_check_mark: |
|
||
|-------|-----------|------------------------------|
|
||
| Submitted | `C0AQ2AWLNEN` | Thread reply on original, copy to Processed |
|
||
| Processed | `C0APLSGABAB` | Delete from Processed, post to Authorized |
|
||
| Authorized | `C0AQ09CDJH4` | Delete from Authorized, post to Matched |
|
||
| Matched | `C0APYUM1JFP` | Terminal stage (no further routing) |
|
||
|
||
**Architecture:** Two Lambdas — ExpenseReceiver (HTTP endpoint, signature verification, async invoke) and ExpenseProcessor (business logic). This is the same receiver/processor pattern used for Slack's 3-second timeout requirement.
|
||
|
||
**Secrets (Secrets Manager):**
|
||
|
||
| Secret | Purpose |
|
||
|--------|---------|
|
||
| `payments-dashboard/expense-slack-token` | Slack Bot token for the Expense Approval Bot app |
|
||
| `payments-dashboard/expense-slack-signing-secret` | Slack signing secret for request verification |
|
||
|
||
## Payroll Email Pipeline
|
||
|
||
Gusto sends payroll confirmation emails when payroll is run. A Gmail filter on adam@seahavenind.com auto-forwards emails from `automated@gusto.com` and `gustonoreply@gusto.com` to `payroll@int.seahaven.com`.
|
||
|
||
**Flow:** Gmail forward → SES receipt rule → S3 bucket → Lambda parses email → DynamoDB (pending) → SQS delay queue (10 min) → Lambda batches all pending items for that date → single Slack message → DynamoDB (notified)
|
||
|
||
**Deduplication:** Each email is deduplicated by DynamoDB key (`PAYROLL_EMAIL#employee#<date>` or `PAYROLL_EMAIL#contractor#<date>#<bank-suffix>`). The batch post is deduplicated by `PAYROLL_BATCH#<date>`. All items have a 90-day TTL.
|
||
|
||
## BoA CashPro API Integration
|
||
|
||
Two separate CashPro APIs are used, each with its own OAuth credentials:
|
||
|
||
| API | Purpose | Endpoint |
|
||
|-----|---------|----------|
|
||
| Check Management | Issue and cancel checks | `/cashpro/checkmanagement/v1/check-issues` |
|
||
| Reporting (Transaction Inquiry) | Fetch previous-day transactions | `/cashpro/reporting/v1/transaction-inquiries/previous-day` |
|
||
|
||
**Authentication flow:**
|
||
1. POST to `/authn/v1/client-authentication` with `applicationID`, `client_id`, and `client_secret`
|
||
2. Receive a Bearer `access_token` (valid 1 hour)
|
||
3. Pass the token in the `Authorization` header for subsequent API calls
|
||
|
||
**Base URLs:**
|
||
- Production: `https://api.bofa.com`
|
||
- Sandbox: `https://api-sb.bofa.com`
|
||
|
||
## Bank reconciliation (fetchBoaTransactions)
|
||
|
||
The scheduled Lambda reconciles the Previous Day feed onto `payment#` records (pure logic lives in `src/boaRecon.js`, covered by `npm test`).
|
||
|
||
**Classification.** The API's Detail rows carry the statement line in `detailText` (ACH `DES:PAYMENTS ID:PMT` text, return descriptions) and the posting date in `asOfDate` (ISO); `transactionType: "Summary"` rows are balance/total lines and classify as `summary` (skipped, counted). Only 475 check-paid rows omit `detailText` — their check number rides in `customerReference` (all-zeros references normalize to empty, never check number 0). Classification runs the empirically-enumerated BAI code map first, then the description-text classifiers:
|
||
|
||
| BAI code | Feed label | Event |
|
||
|---|---|---|
|
||
| 475 | Check Paid | `check_paid` (number in `customerReference`) |
|
||
| 255 | Check Posted and Returned CR | `check_return` (number in `customerReference`) |
|
||
| 252 | Debit Reversal Credit | `check_return` (second return-credit code) |
|
||
| 266 | Return Item Credit | text decides (`ach_return` on PMT text, `check_return`/`electronic_return` on return text); bare rows fall back to `electronic_return`; unreadable text stays `unknown` (loud, no write) |
|
||
| 455 | Preauthorized ACH Debit | `ach_debit` via `DES:PAYMENTS` text; DES-less = third-party autopay → ignored |
|
||
| 170/201/470/481 | totals, transfers, loan payments | ignored |
|
||
|
||
A hard code-map event wins over description text; 266/455 carry *fallback* events consulted only when the text yields nothing. Unmapped codes on check-shaped transactions are logged (`console.error`) and counted in the run summary — never silently dropped.
|
||
|
||
**Matching.** Check events match on check number AND amount, evaluating all candidates (bank postings can drop/collapse digits on long check numbers). A number match with the WRONG amount never auto-resolves — it is the altered-check/collapsed-posting signal and goes to unmatched for human review. An unknown number falls back to an exact-amount match within checks issued in the last 120 days, and only when the posting's digits are a subsequence of the candidate's check number (or vice versa); return credits additionally require a bank-confirmed candidate. Zero or multiple fallback candidates means unmatched, recorded in the run summary with no write. Electronic returns (no check number) match by exact amount among bank-confirmed payments.
|
||
|
||
**ACH.** ACH is bank-confirmed too (#69): `processPaymentCsv` no longer auto-clears ACH on the send date (rows keep their Stampli status until the bank settles). ACH CCD lines match by the stored `pmt_id` first (reversals reuse the original `PMT <id>`; a `pmt_id` match with the wrong amount is the partial-reversal human case and goes to unmatched), then by the Stampli payment number embedded at the end of `PMT INFO` (internal spaces stripped, amount must agree), then by vendor + exact amount within `send_payment_on` −2..+14 days (candidates whose stored `pmt_id` differs are excluded; vendor prefix matching requires ≥ 10 normalized chars). A settled debit clears the record and persists `pmt_id`. A return credit whose `PMT <id>` attributes to no stored `pmt_id` is an unmatched alert ("unknown PMT id"); credits without a `PMT <id>` may fall back to a unique same-amount match against a bank-confirmed ACH whose `cleared_date` is within the prior 45 days.
|
||
|
||
**Staleness sweep.** Every run also flags never-bank-confirmed payments: ACH with no `clear_status` sent more than 16 days ago (listed in the summary, capped at 50, plus a total count) and checks issued more than 60 days ago with no `clear_status` (count only). Both alert via `console.error`.
|
||
|
||
**State transitions.** Every write sets BOTH `status` and `clear_status` (`clear_status` is bank truth; "Cleared without a subsequent return is permanent" keys off it):
|
||
|
||
| Bank event | Result |
|
||
|---|---|
|
||
| Paid debit | `status=Cleared`, `clear_status=Cleared`, `paid_date`, `cleared_date`, `bank_reference` |
|
||
| Return credit (even if currently Cleared) | `clear_status=Returned`, `returned_date`; `status` re-written unchanged (the CSV ladder has no Returned rung) |
|
||
| Second paid debit on a Returned check | Redeposit: back to Cleared with new dates |
|
||
| Return on a Stampli-voided check | Terminal voided-and-bounced (`clear_status=Returned`, cancel status preserved), counted separately |
|
||
|
||
Each applied event is appended to a `history` list attribute (`{event, date, bankRef, amount}`); identical replayed events are idempotent noops.
|
||
|
||
**Replay.** The handler accepts an optional payload `{"fromDate": "YYYY-MM-DD", "toDate": "YYYY-MM-DD"}` (strictly validated; `toDate` defaults to `fromDate`) for weekend/outage gap replays and BAI-code enumeration runs. With no payload it queries the trailing 7-day window (today−7 .. today−1) — self-healing across missed runs and holiday gaps; overlapping days are idempotent (event identity `{event, date, amount}`), and multi-day responses were verified pagination-free up to 9-day windows. Feed rows without a valid posting date (`asOfDate`, with `valueDate` as a fixture-era fallback) are never applied with a substituted date — they go to unmatched for review.
|
||
|
||
**Run summary.** Each run writes an append-only `boa_recon#<fromDate>_<toDate>#<runAt>` item (90-day TTL) with counts per classified event type, matched/applied/redeposit/voided-and-bounced/write-conflict totals, the unmatched check numbers and amounts (list capped at 50; full count kept), unknown BAI codes (capped at 20 distinct keys), and the staleness sweep results. Unmatched and unknown-code transactions also `console.error` (Slack alerting is tracked in #71). Payment writes are conditioned on the read snapshot's `status`/`clear_status` and retried once against a fresh read on conflict.
|
||
|
||
## Documentation
|
||
|
||
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `payments-dashboard` stack is represented there as a Mermaid subgraph.
|
||
|
||
- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)
|
||
|
||
## Secrets
|
||
|
||
All BoA and Slack credentials are stored in AWS Secrets Manager (per `engineering-handbook/secrets-and-config.md`). The Slack token is a plaintext secret; the two BoA secrets are JSON grouping each API's credentials:
|
||
|
||
| Secret | Type | Contents |
|
||
|--------|------|----------|
|
||
| `payments-dashboard/slack-bot-token` | plaintext | Slack Bot OAuth token (used by `processPayrollEmail`, `slackAppHome`) |
|
||
| `payments-dashboard/slack-signing-secret` | plaintext | Slack signing secret for `slackAppHome` request verification |
|
||
| `payments-dashboard/boa-check-mgmt` | JSON | `appId`, `clientId`, `token`, `accountNumber`, `companyId` — Check Management API (`processPaymentCsv`) |
|
||
| `payments-dashboard/boa-reporting` | JSON | `appId`, `clientId`, `token`, `accountNumber`, `bankId` — Reporting API (`fetchBoaTransactions`) |
|
||
|
||
`boa-account-number` is duplicated into both BoA secrets. Each Lambda is granted `secretsmanager:GetSecretValue` scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (`payments-dashboard/expense-slack-token`, `payments-dashboard/expense-slack-signing-secret`).
|
||
|
||
## Consumers / data contract
|
||
|
||
The `PaymentsDashboard` DynamoDB table (`AWS::DynamoDB::Table`, `TableName: PaymentsDashboard`, PK `pk` (S), CMK-encrypted) is **owned by this stack**, which is the sole authoritative writer.
|
||
|
||
**Consumer (read-only):** `seahaven-slack-bot` imports this table via `Table.fromTableName(...)` and reads it read-only (`grantReadData` plus an explicit `kms:Decrypt` grant on the shared CMK) from its `wo-po-lookup` Lambda, which backs the Bedrock agent's payment-lookup action group. The bot depends on:
|
||
|
||
- **Key schema:** PK `pk` (S) with the item format `payment#<check_number>`. It does `GetItem` by `pk` and a full-table `Scan` filtered `begins_with(pk, "payment#")`.
|
||
- **Attributes:** `check_number`, `payee`, `amount_usd`, `method`, `status`, `send_payment_on`, `clear_status`, `cleared_date`, `invoice_numbers`, `company_subsidiary`, `bank_reference`.
|
||
- **Encryption:** the shared customer-managed CMK (`/seahaven/dynamodb/cmk-arn`). Because the consumer imports the table by name, `grantReadData` does not carry KMS access; a change of CMK requires re-granting on the consumer side or every read fails with `kms:Decrypt AccessDenied` (INFRA-95 / M-3 precedent).
|
||
|
||
The table is imported by name, so there is no compile-time link between the stacks: any change to the table name, `pk` format, these attribute names, the encryption key, or the table's lifecycle policy will silently break the Bedrock agent at runtime. Coordinate such changes with `seahaven-slack-bot` before shipping (INFRA-138).
|
||
|
||
**Key prefixes in this table** (all owned by this stack): `payment#<check_number>` (payment records), `metadata` (ingest metadata), `boa_txn#<ts>#<action>` (BoA submission journal, 90d TTL), `boa_recon#<from>_<to>#<runAt>` (reconciliation run summaries, 90d TTL), `boa_balance#<asOfDate>#<endpoint>` (daily balance snapshots, latest-wins, no TTL), `PAYROLL_*` (payroll pipeline, 90d TTL). New prefixes are invisible to `seahaven-slack-bot`'s `begins_with(pk, "payment#")` scan — no consumer coordination needed when adding one.
|
||
|
||
## Monitoring & Alarms
|
||
|
||
All CloudWatch alarms publish to the shared `site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as `notBreaching`. Each alarm evaluates a single 5-minute period.
|
||
|
||
**SQS dead-letter queues** (messages-present, Maximum > 0):
|
||
|
||
| Alarm | Source |
|
||
|-------|--------|
|
||
| `payments-payroll-batch-dlq-messages` | `payments-payroll-batch-dlq` |
|
||
| `payments-processPaymentCsv-async-dlq-messages` | async-invoke OnFailure DLQ |
|
||
| `payments-processPayrollEmail-async-dlq-messages` | async-invoke OnFailure DLQ |
|
||
|
||
**Lambda** (per function — `payments-<fn>-...`):
|
||
|
||
| Type | Metric / Statistic | Threshold |
|
||
|------|--------------------|-----------|
|
||
| `-errors` (all 6) | `Errors` / Sum | > 0 |
|
||
| `-throttles` (all 6) | `Throttles` / Sum | > 0 |
|
||
| `-duration` (all 6) | `Duration` / Maximum | ~80% of each function's timeout |
|
||
|
||
Duration thresholds (ms): processPaymentCsv 96000, processPayrollEmail 48000, fetchBoaTransactions 48000, slackAppHome 24000, expenseProcessor 12000, expenseReceiver 4000.
|
||
|
||
**DynamoDB** (`PaymentsDashboard` table, `TableName` dimension, Sum > 0): `payments-dashboard-table-read-throttle` (`ReadThrottleEvents`), `payments-dashboard-table-write-throttle` (`WriteThrottleEvents`). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. `SystemErrors` is intentionally not alarmed because it does not emit at the `TableName`-only dimension.
|
||
|
||
**API Gateway** (implicit HTTP API v2 `ServerlessHttpApi`, `ApiId` dimension): `payments-dashboard-api-5xx` (`5xx` Sum > 0), `payments-dashboard-api-4xx` (`4xx` Sum > 10, client-error noise floor), `payments-dashboard-api-latency-p99` (`Latency` p99 > 3000 ms).
|
||
|
||
## Scripts
|
||
|
||
| Script | Purpose |
|
||
|--------|---------|
|
||
| `scripts/test-boa-sandbox.js` | One-off sandbox connectivity test for both CashPro APIs |
|
||
| `scripts/seed-from-csv.js` | Seed DynamoDB from a local CSV file |
|
||
| `scripts/seed-bank-status.js` | Seed bank clear status data into DynamoDB |
|
||
|
||
## Deployment
|
||
|
||
```bash
|
||
sam build
|
||
sam deploy --guided
|
||
```
|
||
|
||
The `BOA_BASE_URL` environment variable in `template.yaml` controls whether Lambdas hit production (`https://api.bofa.com`) or sandbox (`https://api-sb.bofa.com`). All other BoA config is read from Secrets Manager at runtime.
|