mirror of
https://github.com/Sea-Haven-Industries/payments-dashboard.git
synced 2026-10-03 03:23:26 +00:00
Adam Moussa
a49b91a25f
processPaymentCsv auto-cleared ACH the moment send_payment_on passed, but the bank disagrees often enough to matter: settlement lags the send date by up to 8 days, settled ACH can bounce and re-debit (Uline $19,281.12, Heights $10,000.00 on the 5/26 overdrawn week), and voided ACH that settled anyway returned via electronic credits invisible to a Check-only feed. Move ACH to bank confirmation: - processPaymentCsv: ACH rows keep their Stampli status; the send-date auto-clear is removed. The bankConfirmed protection is unchanged, so bank-cleared records still cannot be moved backward by the CSV. - fetchBoaTransactions scans all payments (method filter dropped) and processes ACH CCD lines: match by stored pmt_id first (reversals reuse the original PMT id), then by the Stampli payment number embedded in PMT INFO (internal spaces stripped, amount must agree), then vendor + exact amount within send_payment_on -2..+14 days. Settled debit -> Cleared/Cleared with dates, pmt_id persisted on the record; credit reusing the pmt_id (or a unique same-amount return credit against a bank-confirmed ACH) -> clear_status=Returned + returned_date + history event. Ambiguity is unmatched, no write. Existing DDB ACH records already Cleared by the old auto-clear are unaffected: bank confirmation is additive and the CSV path never moves a record backward. Handler event contract extended (optional fromDate/toDate); cross-family review required before merge.
167 lines
14 KiB
Markdown
167 lines
14 KiB
Markdown
# Payments Dashboard
|
||
|
||
|
||
|
||
|
||
|
||
|
||
AWS SAM application that ingests payment CSVs, syncs check data with Bank of America CashPro APIs, processes Gusto payroll confirmation emails into Slack notifications, surfaces an outstanding-payments dashboard in Slack, and routes expense approvals through a Slack reaction-driven workflow.
|
||
|
||
## Architecture
|
||
|
||
- **ProcessPayrollEmail** — Lambda triggered by S3 (inbound email) and SQS (batch timer). SES receives Gusto payroll emails at `payroll@int.seahaven.com`, stores them to S3, and this Lambda parses the email body, extracts financial data, and posts a combined Slack notification (employee payroll + contractor payments) after a 10-minute batching window. Runs outside VPC.
|
||
- **ProcessPaymentCsv** — Lambda triggered by S3 CSV upload. Parses Stampli payment exports, upserts to DynamoDB, and submits new/cancelled checks to the CashPro Check Management API.
|
||
- **FetchBoaTransactions** — Scheduled Lambda (weekdays 9am ET). Calls the CashPro Previous Day Transaction Inquiry API, classifies each transaction (paid checks, ARP refer-to-maker and return-of-posted-check credits, electronic returns), and reconciles them onto DynamoDB payment records. See [Bank reconciliation](#bank-reconciliation-fetchboatransactions).
|
||
- **SlackAppHome** — Lambda behind API Gateway (`POST /slack/events`). Verifies the Slack signing secret (HMAC-SHA256, 5-minute replay window) before processing, then renders the payments dashboard on the Slack App Home tab with outstanding aging buckets and drill-down modals.
|
||
|
||
- **ExpenseReceiver** — Lambda behind API Gateway (`POST /slack/expense-events`). Verifies the Slack signing secret (HMAC-SHA256), handles URL verification challenges, and async-invokes ExpenseProcessor. Runs outside VPC.
|
||
- **ExpenseProcessor** — Async Lambda invoked by ExpenseReceiver. Processes `:white_check_mark:` reactions to advance expense messages through a four-stage Slack channel pipeline: Submitted → Processed → Authorized → Matched. Runs outside VPC.
|
||
|
||
ProcessPaymentCsv, FetchBoaTransactions, and SlackAppHome run inside a VPC with a NAT Gateway for a static outbound IP (required by BoA IP whitelisting). ProcessPayrollEmail, ExpenseReceiver, and ExpenseProcessor run outside the VPC.
|
||
|
||
## Expense Approval Bot
|
||
|
||
Reaction-driven workflow that routes expense submissions through four Slack channels. A separate Slack app ("Expense Approval Bot") posts to a **Submitted** channel. Users react with :white_check_mark: to advance the message to the next stage.
|
||
|
||
**Channel pipeline:**
|
||
|
||
| Stage | Channel ID | Action on :white_check_mark: |
|
||
|-------|-----------|------------------------------|
|
||
| Submitted | `C0AQ2AWLNEN` | Thread reply on original, copy to Processed |
|
||
| Processed | `C0APLSGABAB` | Delete from Processed, post to Authorized |
|
||
| Authorized | `C0AQ09CDJH4` | Delete from Authorized, post to Matched |
|
||
| Matched | `C0APYUM1JFP` | Terminal stage (no further routing) |
|
||
|
||
**Architecture:** Two Lambdas — ExpenseReceiver (HTTP endpoint, signature verification, async invoke) and ExpenseProcessor (business logic). This is the same receiver/processor pattern used for Slack's 3-second timeout requirement.
|
||
|
||
**Secrets (Secrets Manager):**
|
||
|
||
| Secret | Purpose |
|
||
|--------|---------|
|
||
| `payments-dashboard/expense-slack-token` | Slack Bot token for the Expense Approval Bot app |
|
||
| `payments-dashboard/expense-slack-signing-secret` | Slack signing secret for request verification |
|
||
|
||
## Payroll Email Pipeline
|
||
|
||
Gusto sends payroll confirmation emails when payroll is run. A Gmail filter on adam@seahavenind.com auto-forwards emails from `automated@gusto.com` and `gustonoreply@gusto.com` to `payroll@int.seahaven.com`.
|
||
|
||
**Flow:** Gmail forward → SES receipt rule → S3 bucket → Lambda parses email → DynamoDB (pending) → SQS delay queue (10 min) → Lambda batches all pending items for that date → single Slack message → DynamoDB (notified)
|
||
|
||
**Deduplication:** Each email is deduplicated by DynamoDB key (`PAYROLL_EMAIL#employee#<date>` or `PAYROLL_EMAIL#contractor#<date>#<bank-suffix>`). The batch post is deduplicated by `PAYROLL_BATCH#<date>`. All items have a 90-day TTL.
|
||
|
||
## BoA CashPro API Integration
|
||
|
||
Two separate CashPro APIs are used, each with its own OAuth credentials:
|
||
|
||
| API | Purpose | Endpoint |
|
||
|-----|---------|----------|
|
||
| Check Management | Issue and cancel checks | `/cashpro/checkmanagement/v1/check-issues` |
|
||
| Reporting (Transaction Inquiry) | Fetch previous-day transactions | `/cashpro/reporting/v1/transaction-inquiries/previous-day` |
|
||
|
||
**Authentication flow:**
|
||
1. POST to `/authn/v1/client-authentication` with `applicationID`, `client_id`, and `client_secret`
|
||
2. Receive a Bearer `access_token` (valid 1 hour)
|
||
3. Pass the token in the `Authorization` header for subsequent API calls
|
||
|
||
**Base URLs:**
|
||
- Production: `https://api.bofa.com`
|
||
- Sandbox: `https://api-sb.bofa.com`
|
||
|
||
## Bank reconciliation (fetchBoaTransactions)
|
||
|
||
The scheduled Lambda reconciles the Previous Day feed onto `payment#` records (pure logic lives in `src/boaRecon.js`, covered by `npm test`).
|
||
|
||
**Classification.** Transactions are classified via a BAI transaction-code map (`BAI_CODE_EVENTS`; only 475 = check paid debit is empirically confirmed so far — the remaining codes are pending an enumeration replay over known event dates) with a fallback classifier over the statement description formats (ARP refer-to-maker, return-of-posted-check check-numbered and electronic variants, ACH CCD `DES:PAYMENTS ID:PMT` lines). Unmapped codes on check-shaped transactions are logged (`console.error`) and counted in the run summary — never silently dropped.
|
||
|
||
**Matching.** Check events match on check number AND amount, evaluating all candidates (bank postings can drop/collapse digits on long check numbers). A number match with the wrong amount — or an unknown number — falls back to an exact-amount match within checks issued in the last 120 days; zero or multiple fallback candidates means unmatched, and unmatched transactions are recorded in the run summary with no write. Electronic returns (no check number) match by exact amount among bank-confirmed payments.
|
||
|
||
**ACH.** ACH is bank-confirmed too (#69): `processPaymentCsv` no longer auto-clears ACH on the send date (rows keep their Stampli status until the bank settles). ACH CCD lines match by the stored `pmt_id` first (reversals reuse the original `PMT <id>`), then by the Stampli payment number embedded at the end of `PMT INFO` (internal spaces stripped, amount must agree), then by vendor + exact amount within `send_payment_on` −2..+14 days. A settled debit clears the record and persists `pmt_id`; a credit reusing the `pmt_id` (or a unique same-amount return credit against a bank-confirmed ACH) marks it Returned.
|
||
|
||
**State transitions.** Every write sets BOTH `status` and `clear_status` (`clear_status` is bank truth; "Cleared without a subsequent return is permanent" keys off it):
|
||
|
||
| Bank event | Result |
|
||
|---|---|
|
||
| Paid debit | `status=Cleared`, `clear_status=Cleared`, `paid_date`, `cleared_date`, `bank_reference` |
|
||
| Return credit (even if currently Cleared) | `clear_status=Returned`, `returned_date`; `status` re-written unchanged (the CSV ladder has no Returned rung) |
|
||
| Second paid debit on a Returned check | Redeposit: back to Cleared with new dates |
|
||
| Return on a Stampli-voided check | Terminal voided-and-bounced (`clear_status=Returned`, cancel status preserved), counted separately |
|
||
|
||
Each applied event is appended to a `history` list attribute (`{event, date, bankRef, amount}`); identical replayed events are idempotent noops.
|
||
|
||
**Replay.** The handler accepts an optional payload `{"fromDate": "YYYY-MM-DD", "toDate": "YYYY-MM-DD"}` (strictly validated; `toDate` defaults to `fromDate`; no payload = yesterday) for weekend/outage gap replays and BAI-code enumeration runs.
|
||
|
||
**Run summary.** Each run writes a `boa_recon#<runDate>` item (90-day TTL) with counts per classified event type, matched/applied/redeposit/voided-and-bounced totals, the unmatched check numbers and amounts, and any unknown BAI codes encountered. Unmatched and unknown-code transactions also `console.error` (Slack alerting is tracked in #71).
|
||
|
||
## Documentation
|
||
|
||
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `payments-dashboard` stack is represented there as a Mermaid subgraph.
|
||
|
||
- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)
|
||
|
||
## Secrets
|
||
|
||
All BoA and Slack credentials are stored in AWS Secrets Manager (per `engineering-handbook/secrets-and-config.md`). The Slack token is a plaintext secret; the two BoA secrets are JSON grouping each API's credentials:
|
||
|
||
| Secret | Type | Contents |
|
||
|--------|------|----------|
|
||
| `payments-dashboard/slack-bot-token` | plaintext | Slack Bot OAuth token (used by `processPayrollEmail`, `slackAppHome`) |
|
||
| `payments-dashboard/slack-signing-secret` | plaintext | Slack signing secret for `slackAppHome` request verification |
|
||
| `payments-dashboard/boa-check-mgmt` | JSON | `appId`, `clientId`, `token`, `accountNumber`, `companyId` — Check Management API (`processPaymentCsv`) |
|
||
| `payments-dashboard/boa-reporting` | JSON | `appId`, `clientId`, `token`, `accountNumber`, `bankId` — Reporting API (`fetchBoaTransactions`) |
|
||
|
||
`boa-account-number` is duplicated into both BoA secrets. Each Lambda is granted `secretsmanager:GetSecretValue` scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (`payments-dashboard/expense-slack-token`, `payments-dashboard/expense-slack-signing-secret`).
|
||
|
||
## Consumers / data contract
|
||
|
||
The `PaymentsDashboard` DynamoDB table (`AWS::DynamoDB::Table`, `TableName: PaymentsDashboard`, PK `pk` (S), CMK-encrypted) is **owned by this stack**, which is the sole authoritative writer.
|
||
|
||
**Consumer (read-only):** `seahaven-slack-bot` imports this table via `Table.fromTableName(...)` and reads it read-only (`grantReadData` plus an explicit `kms:Decrypt` grant on the shared CMK) from its `wo-po-lookup` Lambda, which backs the Bedrock agent's payment-lookup action group. The bot depends on:
|
||
|
||
- **Key schema:** PK `pk` (S) with the item format `payment#<check_number>`. It does `GetItem` by `pk` and a full-table `Scan` filtered `begins_with(pk, "payment#")`.
|
||
- **Attributes:** `check_number`, `payee`, `amount_usd`, `method`, `status`, `send_payment_on`, `clear_status`, `cleared_date`, `invoice_numbers`, `company_subsidiary`, `bank_reference`.
|
||
- **Encryption:** the shared customer-managed CMK (`/seahaven/dynamodb/cmk-arn`). Because the consumer imports the table by name, `grantReadData` does not carry KMS access; a change of CMK requires re-granting on the consumer side or every read fails with `kms:Decrypt AccessDenied` (INFRA-95 / M-3 precedent).
|
||
|
||
The table is imported by name, so there is no compile-time link between the stacks: any change to the table name, `pk` format, these attribute names, the encryption key, or the table's lifecycle policy will silently break the Bedrock agent at runtime. Coordinate such changes with `seahaven-slack-bot` before shipping (INFRA-138).
|
||
|
||
## Monitoring & Alarms
|
||
|
||
All CloudWatch alarms publish to the shared `site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as `notBreaching`. Each alarm evaluates a single 5-minute period.
|
||
|
||
**SQS dead-letter queues** (messages-present, Maximum > 0):
|
||
|
||
| Alarm | Source |
|
||
|-------|--------|
|
||
| `payments-payroll-batch-dlq-messages` | `payments-payroll-batch-dlq` |
|
||
| `payments-processPaymentCsv-async-dlq-messages` | async-invoke OnFailure DLQ |
|
||
| `payments-processPayrollEmail-async-dlq-messages` | async-invoke OnFailure DLQ |
|
||
|
||
**Lambda** (per function — `payments-<fn>-...`):
|
||
|
||
| Type | Metric / Statistic | Threshold |
|
||
|------|--------------------|-----------|
|
||
| `-errors` (all 6) | `Errors` / Sum | > 0 |
|
||
| `-throttles` (all 6) | `Throttles` / Sum | > 0 |
|
||
| `-duration` (all 6) | `Duration` / Maximum | ~80% of each function's timeout |
|
||
|
||
Duration thresholds (ms): processPaymentCsv 96000, processPayrollEmail 48000, fetchBoaTransactions 48000, slackAppHome 24000, expenseProcessor 12000, expenseReceiver 4000.
|
||
|
||
**DynamoDB** (`PaymentsDashboard` table, `TableName` dimension, Sum > 0): `payments-dashboard-table-read-throttle` (`ReadThrottleEvents`), `payments-dashboard-table-write-throttle` (`WriteThrottleEvents`). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. `SystemErrors` is intentionally not alarmed because it does not emit at the `TableName`-only dimension.
|
||
|
||
**API Gateway** (implicit HTTP API v2 `ServerlessHttpApi`, `ApiId` dimension): `payments-dashboard-api-5xx` (`5xx` Sum > 0), `payments-dashboard-api-4xx` (`4xx` Sum > 10, client-error noise floor), `payments-dashboard-api-latency-p99` (`Latency` p99 > 3000 ms).
|
||
|
||
## Scripts
|
||
|
||
| Script | Purpose |
|
||
|--------|---------|
|
||
| `scripts/test-boa-sandbox.js` | One-off sandbox connectivity test for both CashPro APIs |
|
||
| `scripts/seed-from-csv.js` | Seed DynamoDB from a local CSV file |
|
||
| `scripts/seed-bank-status.js` | Seed bank clear status data into DynamoDB |
|
||
|
||
## Deployment
|
||
|
||
```bash
|
||
sam build
|
||
sam deploy --guided
|
||
```
|
||
|
||
The `BOA_BASE_URL` environment variable in `template.yaml` controls whether Lambdas hit production (`https://api.bofa.com`) or sandbox (`https://api-sb.bofa.com`). All other BoA config is read from Secrets Manager at runtime.
|