mirror of
https://github.com/Sea-Haven-Industries/payments-dashboard.git
synced 2026-10-03 05:43:28 +00:00
f7adb6e8b8
Some checks are pending
Deploy / deploy (push) Waiting to run
* feat(fetchboa): v2 return/redeposit-aware reconciliation (#66) The two-code 475/255 filter missed every return on the Jan-Jul statement (29 ARP refer-to-maker credits, 24 electronic return credits, and the redeposit cycles), leaving 5 paid-then-returned checks stuck at Cleared ($5,256.62, vendors unpaid). Rewrite fetchBoaTransactions around a pure reconciliation module (src/boaRecon.js) covered by node:test: - BAI transaction-code event map (only 475 = check paid is confirmed; remaining codes pend the enumeration replay) with a description-text fallback classifier for ARP refer-to-maker, return-of-posted-check (check-numbered and electronic) and ACH DES:PAYMENTS formats. Unknown check-shaped transactions are logged and counted, never dropped. - Matching on check number AND amount over all candidates; wrong-amount or collapsed-number postings fall back to a unique exact-amount match within checks issued in the last 120 days; ambiguity means unmatched with no write. - Transitions always set BOTH status and clear_status (the missed returns slipped through the divergence between them). clear_status is bank truth: returns apply even to Cleared records, a second paid debit on a Returned check is a redeposit back to Cleared, and a return on a voided check is terminal voided-and-bounced. Every applied event is appended to a history list; identical replayed events are noops. - Optional {fromDate, toDate} replay payload (strictly validated) for gap replays and the BAI-code enumeration runs; default stays yesterday. - Each run writes a boa_recon#<runDate> summary item (90-day TTL) with per-event counts, unmatched check numbers/amounts, and unknown codes; unmatched/unknown also console.error (Slack alerting is #71). ACH transactions are classified but not yet acted on; bank-confirming ACH lands with #69. * feat(ach): bank-confirm ACH, drop CSV-date auto-clear (#69) processPaymentCsv auto-cleared ACH the moment send_payment_on passed, but the bank disagrees often enough to matter: settlement lags the send date by up to 8 days, settled ACH can bounce and re-debit (Uline $19,281.12, Heights $10,000.00 on the 5/26 overdrawn week), and voided ACH that settled anyway returned via electronic credits invisible to a Check-only feed. Move ACH to bank confirmation: - processPaymentCsv: ACH rows keep their Stampli status; the send-date auto-clear is removed. The bankConfirmed protection is unchanged, so bank-cleared records still cannot be moved backward by the CSV. - fetchBoaTransactions scans all payments (method filter dropped) and processes ACH CCD lines: match by stored pmt_id first (reversals reuse the original PMT id), then by the Stampli payment number embedded in PMT INFO (internal spaces stripped, amount must agree), then vendor + exact amount within send_payment_on -2..+14 days. Settled debit -> Cleared/Cleared with dates, pmt_id persisted on the record; credit reusing the pmt_id (or a unique same-amount return credit against a bank-confirmed ACH) -> clear_status=Returned + returned_date + history event. Ambiguity is unmatched, no write. Existing DDB ACH records already Cleared by the old auto-clear are unaffected: bank confirmation is additive and the CSV path never moves a record backward. Handler event contract extended (optional fromDate/toDate); cross-family review required before merge. * fix(fetchboa): close review findings — coverage gap, matcher corroboration, replay identity (#66) Security-review gate (detector fan-out + verifier + GPT-4.1 cross-family) blocked on F6 and confirmed six lower findings; all are closed here. - F6 (HIGH): the default run now queries a trailing 3-day window (today-3..today-1) so the Monday run covers Friday-Sunday postings the previous-day cadence silently skipped. A per-run staleness sweep flags never-bank-confirmed payments (ACH > 16 days, checks > 60 days) in the summary and via console.error. - F2: replay identity is {event, date, amount} — bankRef excluded (feeds omit/reformat it between runs); a paid event dated on/before the latest return in history is a replayed original, never a redeposit; rows without a valid valueDate are routed to unmatched instead of being applied under a substituted date; within a day, debits sort before credits. - F1: a check-number match with the wrong amount no longer falls through to the amount fallback (altered-check/collapsed-posting is a human-review case); the amount fallback requires digit-subsequence corroboration between posting and candidate numbers; check_return fallback requires a bank-confirmed candidate. - F4: the ach_return amount fallback requires cleared_date within 45 days before the credit; an unattributable PMT id is an unmatched alert, never an amount guess. - F5: the pmt_id rung requires amount equality; mismatch is the partial-reversal human case. - F3: vendor prefix matching requires >= 10 normalized chars (short names must match exactly); candidates with a conflicting stored pmt_id are excluded; vendor+amount matches are audit-logged. - F7: payment writes are conditioned on the snapshot's status/clear_status and retried once against a fresh read; residual conflicts are counted, not silently interleaved. - F9/summary bounds: run summary pk is append-only (boa_recon#<from>_<to>#<runAt>); unmatched list capped at 50, unknown codes at 20 keys/16 chars, stale list at 50 (full counts kept). - ReDoS: description bounded to 500 chars before classification; CHECK/embedded-number regexes use bounded quantifiers. - FBOA2-1: both BoA res.json() parses are wrapped so a malformed 200 throws a status-only error and cannot leak a body snippet. Handler event contract extended (optional fromDate/toDate); cross-family review required before merge. * fix(boaRecon): add method=Check filter to matchElectronicReturn Electronic returns only apply to checks, but the matcher was not filtering by method, so an electronic return could incorrectly match against a same-amount, bank-confirmed ACH record.
169 lines
15 KiB
Markdown
169 lines
15 KiB
Markdown
# Payments Dashboard
|
||
|
||
|
||
|
||
|
||
|
||
|
||
AWS SAM application that ingests payment CSVs, syncs check data with Bank of America CashPro APIs, processes Gusto payroll confirmation emails into Slack notifications, surfaces an outstanding-payments dashboard in Slack, and routes expense approvals through a Slack reaction-driven workflow.
|
||
|
||
## Architecture
|
||
|
||
- **ProcessPayrollEmail** — Lambda triggered by S3 (inbound email) and SQS (batch timer). SES receives Gusto payroll emails at `payroll@int.seahaven.com`, stores them to S3, and this Lambda parses the email body, extracts financial data, and posts a combined Slack notification (employee payroll + contractor payments) after a 10-minute batching window. Runs outside VPC.
|
||
- **ProcessPaymentCsv** — Lambda triggered by S3 CSV upload. Parses Stampli payment exports, upserts to DynamoDB, and submits new/cancelled checks to the CashPro Check Management API.
|
||
- **FetchBoaTransactions** — Scheduled Lambda (weekdays 9am ET). Calls the CashPro Previous Day Transaction Inquiry API, classifies each transaction (paid checks, ARP refer-to-maker and return-of-posted-check credits, electronic returns), and reconciles them onto DynamoDB payment records. See [Bank reconciliation](#bank-reconciliation-fetchboatransactions).
|
||
- **SlackAppHome** — Lambda behind API Gateway (`POST /slack/events`). Verifies the Slack signing secret (HMAC-SHA256, 5-minute replay window) before processing, then renders the payments dashboard on the Slack App Home tab with outstanding aging buckets and drill-down modals.
|
||
|
||
- **ExpenseReceiver** — Lambda behind API Gateway (`POST /slack/expense-events`). Verifies the Slack signing secret (HMAC-SHA256), handles URL verification challenges, and async-invokes ExpenseProcessor. Runs outside VPC.
|
||
- **ExpenseProcessor** — Async Lambda invoked by ExpenseReceiver. Processes `:white_check_mark:` reactions to advance expense messages through a four-stage Slack channel pipeline: Submitted → Processed → Authorized → Matched. Runs outside VPC.
|
||
|
||
ProcessPaymentCsv, FetchBoaTransactions, and SlackAppHome run inside a VPC with a NAT Gateway for a static outbound IP (required by BoA IP whitelisting). ProcessPayrollEmail, ExpenseReceiver, and ExpenseProcessor run outside the VPC.
|
||
|
||
## Expense Approval Bot
|
||
|
||
Reaction-driven workflow that routes expense submissions through four Slack channels. A separate Slack app ("Expense Approval Bot") posts to a **Submitted** channel. Users react with :white_check_mark: to advance the message to the next stage.
|
||
|
||
**Channel pipeline:**
|
||
|
||
| Stage | Channel ID | Action on :white_check_mark: |
|
||
|-------|-----------|------------------------------|
|
||
| Submitted | `C0AQ2AWLNEN` | Thread reply on original, copy to Processed |
|
||
| Processed | `C0APLSGABAB` | Delete from Processed, post to Authorized |
|
||
| Authorized | `C0AQ09CDJH4` | Delete from Authorized, post to Matched |
|
||
| Matched | `C0APYUM1JFP` | Terminal stage (no further routing) |
|
||
|
||
**Architecture:** Two Lambdas — ExpenseReceiver (HTTP endpoint, signature verification, async invoke) and ExpenseProcessor (business logic). This is the same receiver/processor pattern used for Slack's 3-second timeout requirement.
|
||
|
||
**Secrets (Secrets Manager):**
|
||
|
||
| Secret | Purpose |
|
||
|--------|---------|
|
||
| `payments-dashboard/expense-slack-token` | Slack Bot token for the Expense Approval Bot app |
|
||
| `payments-dashboard/expense-slack-signing-secret` | Slack signing secret for request verification |
|
||
|
||
## Payroll Email Pipeline
|
||
|
||
Gusto sends payroll confirmation emails when payroll is run. A Gmail filter on adam@seahavenind.com auto-forwards emails from `automated@gusto.com` and `gustonoreply@gusto.com` to `payroll@int.seahaven.com`.
|
||
|
||
**Flow:** Gmail forward → SES receipt rule → S3 bucket → Lambda parses email → DynamoDB (pending) → SQS delay queue (10 min) → Lambda batches all pending items for that date → single Slack message → DynamoDB (notified)
|
||
|
||
**Deduplication:** Each email is deduplicated by DynamoDB key (`PAYROLL_EMAIL#employee#<date>` or `PAYROLL_EMAIL#contractor#<date>#<bank-suffix>`). The batch post is deduplicated by `PAYROLL_BATCH#<date>`. All items have a 90-day TTL.
|
||
|
||
## BoA CashPro API Integration
|
||
|
||
Two separate CashPro APIs are used, each with its own OAuth credentials:
|
||
|
||
| API | Purpose | Endpoint |
|
||
|-----|---------|----------|
|
||
| Check Management | Issue and cancel checks | `/cashpro/checkmanagement/v1/check-issues` |
|
||
| Reporting (Transaction Inquiry) | Fetch previous-day transactions | `/cashpro/reporting/v1/transaction-inquiries/previous-day` |
|
||
|
||
**Authentication flow:**
|
||
1. POST to `/authn/v1/client-authentication` with `applicationID`, `client_id`, and `client_secret`
|
||
2. Receive a Bearer `access_token` (valid 1 hour)
|
||
3. Pass the token in the `Authorization` header for subsequent API calls
|
||
|
||
**Base URLs:**
|
||
- Production: `https://api.bofa.com`
|
||
- Sandbox: `https://api-sb.bofa.com`
|
||
|
||
## Bank reconciliation (fetchBoaTransactions)
|
||
|
||
The scheduled Lambda reconciles the Previous Day feed onto `payment#` records (pure logic lives in `src/boaRecon.js`, covered by `npm test`).
|
||
|
||
**Classification.** Transactions are classified via a BAI transaction-code map (`BAI_CODE_EVENTS`; only 475 = check paid debit is empirically confirmed so far — the remaining codes are pending an enumeration replay over known event dates) with a fallback classifier over the statement description formats (ARP refer-to-maker, return-of-posted-check check-numbered and electronic variants, ACH CCD `DES:PAYMENTS ID:PMT` lines). Unmapped codes on check-shaped transactions are logged (`console.error`) and counted in the run summary — never silently dropped.
|
||
|
||
**Matching.** Check events match on check number AND amount, evaluating all candidates (bank postings can drop/collapse digits on long check numbers). A number match with the WRONG amount never auto-resolves — it is the altered-check/collapsed-posting signal and goes to unmatched for human review. An unknown number falls back to an exact-amount match within checks issued in the last 120 days, and only when the posting's digits are a subsequence of the candidate's check number (or vice versa); return credits additionally require a bank-confirmed candidate. Zero or multiple fallback candidates means unmatched, recorded in the run summary with no write. Electronic returns (no check number) match by exact amount among bank-confirmed payments.
|
||
|
||
**ACH.** ACH is bank-confirmed too (#69): `processPaymentCsv` no longer auto-clears ACH on the send date (rows keep their Stampli status until the bank settles). ACH CCD lines match by the stored `pmt_id` first (reversals reuse the original `PMT <id>`; a `pmt_id` match with the wrong amount is the partial-reversal human case and goes to unmatched), then by the Stampli payment number embedded at the end of `PMT INFO` (internal spaces stripped, amount must agree), then by vendor + exact amount within `send_payment_on` −2..+14 days (candidates whose stored `pmt_id` differs are excluded; vendor prefix matching requires ≥ 10 normalized chars). A settled debit clears the record and persists `pmt_id`. A return credit whose `PMT <id>` attributes to no stored `pmt_id` is an unmatched alert ("unknown PMT id"); credits without a `PMT <id>` may fall back to a unique same-amount match against a bank-confirmed ACH whose `cleared_date` is within the prior 45 days.
|
||
|
||
**Staleness sweep.** Every run also flags never-bank-confirmed payments: ACH with no `clear_status` sent more than 16 days ago (listed in the summary, capped at 50, plus a total count) and checks issued more than 60 days ago with no `clear_status` (count only). Both alert via `console.error`.
|
||
|
||
**State transitions.** Every write sets BOTH `status` and `clear_status` (`clear_status` is bank truth; "Cleared without a subsequent return is permanent" keys off it):
|
||
|
||
| Bank event | Result |
|
||
|---|---|
|
||
| Paid debit | `status=Cleared`, `clear_status=Cleared`, `paid_date`, `cleared_date`, `bank_reference` |
|
||
| Return credit (even if currently Cleared) | `clear_status=Returned`, `returned_date`; `status` re-written unchanged (the CSV ladder has no Returned rung) |
|
||
| Second paid debit on a Returned check | Redeposit: back to Cleared with new dates |
|
||
| Return on a Stampli-voided check | Terminal voided-and-bounced (`clear_status=Returned`, cancel status preserved), counted separately |
|
||
|
||
Each applied event is appended to a `history` list attribute (`{event, date, bankRef, amount}`); identical replayed events are idempotent noops.
|
||
|
||
**Replay.** The handler accepts an optional payload `{"fromDate": "YYYY-MM-DD", "toDate": "YYYY-MM-DD"}` (strictly validated; `toDate` defaults to `fromDate`) for weekend/outage gap replays and BAI-code enumeration runs. With no payload it queries the trailing 3-day window (today−3 .. today−1), so the Monday run covers Friday–Sunday; overlapping days are idempotent (event identity `{event, date, amount}`). Feed rows without a valid `valueDate` are never applied with a substituted date — they go to unmatched for review.
|
||
|
||
**Run summary.** Each run writes an append-only `boa_recon#<fromDate>_<toDate>#<runAt>` item (90-day TTL) with counts per classified event type, matched/applied/redeposit/voided-and-bounced/write-conflict totals, the unmatched check numbers and amounts (list capped at 50; full count kept), unknown BAI codes (capped at 20 distinct keys), and the staleness sweep results. Unmatched and unknown-code transactions also `console.error` (Slack alerting is tracked in #71). Payment writes are conditioned on the read snapshot's `status`/`clear_status` and retried once against a fresh read on conflict.
|
||
|
||
## Documentation
|
||
|
||
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `payments-dashboard` stack is represented there as a Mermaid subgraph.
|
||
|
||
- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)
|
||
|
||
## Secrets
|
||
|
||
All BoA and Slack credentials are stored in AWS Secrets Manager (per `engineering-handbook/secrets-and-config.md`). The Slack token is a plaintext secret; the two BoA secrets are JSON grouping each API's credentials:
|
||
|
||
| Secret | Type | Contents |
|
||
|--------|------|----------|
|
||
| `payments-dashboard/slack-bot-token` | plaintext | Slack Bot OAuth token (used by `processPayrollEmail`, `slackAppHome`) |
|
||
| `payments-dashboard/slack-signing-secret` | plaintext | Slack signing secret for `slackAppHome` request verification |
|
||
| `payments-dashboard/boa-check-mgmt` | JSON | `appId`, `clientId`, `token`, `accountNumber`, `companyId` — Check Management API (`processPaymentCsv`) |
|
||
| `payments-dashboard/boa-reporting` | JSON | `appId`, `clientId`, `token`, `accountNumber`, `bankId` — Reporting API (`fetchBoaTransactions`) |
|
||
|
||
`boa-account-number` is duplicated into both BoA secrets. Each Lambda is granted `secretsmanager:GetSecretValue` scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (`payments-dashboard/expense-slack-token`, `payments-dashboard/expense-slack-signing-secret`).
|
||
|
||
## Consumers / data contract
|
||
|
||
The `PaymentsDashboard` DynamoDB table (`AWS::DynamoDB::Table`, `TableName: PaymentsDashboard`, PK `pk` (S), CMK-encrypted) is **owned by this stack**, which is the sole authoritative writer.
|
||
|
||
**Consumer (read-only):** `seahaven-slack-bot` imports this table via `Table.fromTableName(...)` and reads it read-only (`grantReadData` plus an explicit `kms:Decrypt` grant on the shared CMK) from its `wo-po-lookup` Lambda, which backs the Bedrock agent's payment-lookup action group. The bot depends on:
|
||
|
||
- **Key schema:** PK `pk` (S) with the item format `payment#<check_number>`. It does `GetItem` by `pk` and a full-table `Scan` filtered `begins_with(pk, "payment#")`.
|
||
- **Attributes:** `check_number`, `payee`, `amount_usd`, `method`, `status`, `send_payment_on`, `clear_status`, `cleared_date`, `invoice_numbers`, `company_subsidiary`, `bank_reference`.
|
||
- **Encryption:** the shared customer-managed CMK (`/seahaven/dynamodb/cmk-arn`). Because the consumer imports the table by name, `grantReadData` does not carry KMS access; a change of CMK requires re-granting on the consumer side or every read fails with `kms:Decrypt AccessDenied` (INFRA-95 / M-3 precedent).
|
||
|
||
The table is imported by name, so there is no compile-time link between the stacks: any change to the table name, `pk` format, these attribute names, the encryption key, or the table's lifecycle policy will silently break the Bedrock agent at runtime. Coordinate such changes with `seahaven-slack-bot` before shipping (INFRA-138).
|
||
|
||
## Monitoring & Alarms
|
||
|
||
All CloudWatch alarms publish to the shared `site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as `notBreaching`. Each alarm evaluates a single 5-minute period.
|
||
|
||
**SQS dead-letter queues** (messages-present, Maximum > 0):
|
||
|
||
| Alarm | Source |
|
||
|-------|--------|
|
||
| `payments-payroll-batch-dlq-messages` | `payments-payroll-batch-dlq` |
|
||
| `payments-processPaymentCsv-async-dlq-messages` | async-invoke OnFailure DLQ |
|
||
| `payments-processPayrollEmail-async-dlq-messages` | async-invoke OnFailure DLQ |
|
||
|
||
**Lambda** (per function — `payments-<fn>-...`):
|
||
|
||
| Type | Metric / Statistic | Threshold |
|
||
|------|--------------------|-----------|
|
||
| `-errors` (all 6) | `Errors` / Sum | > 0 |
|
||
| `-throttles` (all 6) | `Throttles` / Sum | > 0 |
|
||
| `-duration` (all 6) | `Duration` / Maximum | ~80% of each function's timeout |
|
||
|
||
Duration thresholds (ms): processPaymentCsv 96000, processPayrollEmail 48000, fetchBoaTransactions 48000, slackAppHome 24000, expenseProcessor 12000, expenseReceiver 4000.
|
||
|
||
**DynamoDB** (`PaymentsDashboard` table, `TableName` dimension, Sum > 0): `payments-dashboard-table-read-throttle` (`ReadThrottleEvents`), `payments-dashboard-table-write-throttle` (`WriteThrottleEvents`). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. `SystemErrors` is intentionally not alarmed because it does not emit at the `TableName`-only dimension.
|
||
|
||
**API Gateway** (implicit HTTP API v2 `ServerlessHttpApi`, `ApiId` dimension): `payments-dashboard-api-5xx` (`5xx` Sum > 0), `payments-dashboard-api-4xx` (`4xx` Sum > 10, client-error noise floor), `payments-dashboard-api-latency-p99` (`Latency` p99 > 3000 ms).
|
||
|
||
## Scripts
|
||
|
||
| Script | Purpose |
|
||
|--------|---------|
|
||
| `scripts/test-boa-sandbox.js` | One-off sandbox connectivity test for both CashPro APIs |
|
||
| `scripts/seed-from-csv.js` | Seed DynamoDB from a local CSV file |
|
||
| `scripts/seed-bank-status.js` | Seed bank clear status data into DynamoDB |
|
||
|
||
## Deployment
|
||
|
||
```bash
|
||
sam build
|
||
sam deploy --guided
|
||
```
|
||
|
||
The `BOA_BASE_URL` environment variable in `template.yaml` controls whether Lambdas hit production (`https://api.bofa.com`) or sandbox (`https://api-sb.bofa.com`). All other BoA config is read from Secrets Manager at runtime.
|