f7adb6e8b8
Some checks are pending
Deploy / deploy (push) Waiting to run
* feat(fetchboa): v2 return/redeposit-aware reconciliation (#66) The two-code 475/255 filter missed every return on the Jan-Jul statement (29 ARP refer-to-maker credits, 24 electronic return credits, and the redeposit cycles), leaving 5 paid-then-returned checks stuck at Cleared ($5,256.62, vendors unpaid). Rewrite fetchBoaTransactions around a pure reconciliation module (src/boaRecon.js) covered by node:test: - BAI transaction-code event map (only 475 = check paid is confirmed; remaining codes pend the enumeration replay) with a description-text fallback classifier for ARP refer-to-maker, return-of-posted-check (check-numbered and electronic) and ACH DES:PAYMENTS formats. Unknown check-shaped transactions are logged and counted, never dropped. - Matching on check number AND amount over all candidates; wrong-amount or collapsed-number postings fall back to a unique exact-amount match within checks issued in the last 120 days; ambiguity means unmatched with no write. - Transitions always set BOTH status and clear_status (the missed returns slipped through the divergence between them). clear_status is bank truth: returns apply even to Cleared records, a second paid debit on a Returned check is a redeposit back to Cleared, and a return on a voided check is terminal voided-and-bounced. Every applied event is appended to a history list; identical replayed events are noops. - Optional {fromDate, toDate} replay payload (strictly validated) for gap replays and the BAI-code enumeration runs; default stays yesterday. - Each run writes a boa_recon#<runDate> summary item (90-day TTL) with per-event counts, unmatched check numbers/amounts, and unknown codes; unmatched/unknown also console.error (Slack alerting is #71). ACH transactions are classified but not yet acted on; bank-confirming ACH lands with #69. * feat(ach): bank-confirm ACH, drop CSV-date auto-clear (#69) processPaymentCsv auto-cleared ACH the moment send_payment_on passed, but the bank disagrees often enough to matter: settlement lags the send date by up to 8 days, settled ACH can bounce and re-debit (Uline $19,281.12, Heights $10,000.00 on the 5/26 overdrawn week), and voided ACH that settled anyway returned via electronic credits invisible to a Check-only feed. Move ACH to bank confirmation: - processPaymentCsv: ACH rows keep their Stampli status; the send-date auto-clear is removed. The bankConfirmed protection is unchanged, so bank-cleared records still cannot be moved backward by the CSV. - fetchBoaTransactions scans all payments (method filter dropped) and processes ACH CCD lines: match by stored pmt_id first (reversals reuse the original PMT id), then by the Stampli payment number embedded in PMT INFO (internal spaces stripped, amount must agree), then vendor + exact amount within send_payment_on -2..+14 days. Settled debit -> Cleared/Cleared with dates, pmt_id persisted on the record; credit reusing the pmt_id (or a unique same-amount return credit against a bank-confirmed ACH) -> clear_status=Returned + returned_date + history event. Ambiguity is unmatched, no write. Existing DDB ACH records already Cleared by the old auto-clear are unaffected: bank confirmation is additive and the CSV path never moves a record backward. Handler event contract extended (optional fromDate/toDate); cross-family review required before merge. * fix(fetchboa): close review findings — coverage gap, matcher corroboration, replay identity (#66) Security-review gate (detector fan-out + verifier + GPT-4.1 cross-family) blocked on F6 and confirmed six lower findings; all are closed here. - F6 (HIGH): the default run now queries a trailing 3-day window (today-3..today-1) so the Monday run covers Friday-Sunday postings the previous-day cadence silently skipped. A per-run staleness sweep flags never-bank-confirmed payments (ACH > 16 days, checks > 60 days) in the summary and via console.error. - F2: replay identity is {event, date, amount} — bankRef excluded (feeds omit/reformat it between runs); a paid event dated on/before the latest return in history is a replayed original, never a redeposit; rows without a valid valueDate are routed to unmatched instead of being applied under a substituted date; within a day, debits sort before credits. - F1: a check-number match with the wrong amount no longer falls through to the amount fallback (altered-check/collapsed-posting is a human-review case); the amount fallback requires digit-subsequence corroboration between posting and candidate numbers; check_return fallback requires a bank-confirmed candidate. - F4: the ach_return amount fallback requires cleared_date within 45 days before the credit; an unattributable PMT id is an unmatched alert, never an amount guess. - F5: the pmt_id rung requires amount equality; mismatch is the partial-reversal human case. - F3: vendor prefix matching requires >= 10 normalized chars (short names must match exactly); candidates with a conflicting stored pmt_id are excluded; vendor+amount matches are audit-logged. - F7: payment writes are conditioned on the snapshot's status/clear_status and retried once against a fresh read; residual conflicts are counted, not silently interleaved. - F9/summary bounds: run summary pk is append-only (boa_recon#<from>_<to>#<runAt>); unmatched list capped at 50, unknown codes at 20 keys/16 chars, stale list at 50 (full counts kept). - ReDoS: description bounded to 500 chars before classification; CHECK/embedded-number regexes use bounded quantifiers. - FBOA2-1: both BoA res.json() parses are wrapped so a malformed 200 throws a status-only error and cannot leak a body snippet. Handler event contract extended (optional fromDate/toDate); cross-family review required before merge. * fix(boaRecon): add method=Check filter to matchElectronicReturn Electronic returns only apply to checks, but the matcher was not filtering by method, so an electronic return could incorrectly match against a same-amount, bank-confirmed ACH record. |
||
|---|---|---|
| .claude | ||
| .github | ||
| scripts | ||
| src | ||
| tests | ||
| .gitignore | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| samconfig.toml.example | ||
| template.yaml | ||
Payments Dashboard
AWS SAM application that ingests payment CSVs, syncs check data with Bank of America CashPro APIs, processes Gusto payroll confirmation emails into Slack notifications, surfaces an outstanding-payments dashboard in Slack, and routes expense approvals through a Slack reaction-driven workflow.
Architecture
-
ProcessPayrollEmail — Lambda triggered by S3 (inbound email) and SQS (batch timer). SES receives Gusto payroll emails at
payroll@int.seahaven.com, stores them to S3, and this Lambda parses the email body, extracts financial data, and posts a combined Slack notification (employee payroll + contractor payments) after a 10-minute batching window. Runs outside VPC. -
ProcessPaymentCsv — Lambda triggered by S3 CSV upload. Parses Stampli payment exports, upserts to DynamoDB, and submits new/cancelled checks to the CashPro Check Management API.
-
FetchBoaTransactions — Scheduled Lambda (weekdays 9am ET). Calls the CashPro Previous Day Transaction Inquiry API, classifies each transaction (paid checks, ARP refer-to-maker and return-of-posted-check credits, electronic returns), and reconciles them onto DynamoDB payment records. See Bank reconciliation.
-
SlackAppHome — Lambda behind API Gateway (
POST /slack/events). Verifies the Slack signing secret (HMAC-SHA256, 5-minute replay window) before processing, then renders the payments dashboard on the Slack App Home tab with outstanding aging buckets and drill-down modals. -
ExpenseReceiver — Lambda behind API Gateway (
POST /slack/expense-events). Verifies the Slack signing secret (HMAC-SHA256), handles URL verification challenges, and async-invokes ExpenseProcessor. Runs outside VPC. -
ExpenseProcessor — Async Lambda invoked by ExpenseReceiver. Processes
:white_check_mark:reactions to advance expense messages through a four-stage Slack channel pipeline: Submitted → Processed → Authorized → Matched. Runs outside VPC.
ProcessPaymentCsv, FetchBoaTransactions, and SlackAppHome run inside a VPC with a NAT Gateway for a static outbound IP (required by BoA IP whitelisting). ProcessPayrollEmail, ExpenseReceiver, and ExpenseProcessor run outside the VPC.
Expense Approval Bot
Reaction-driven workflow that routes expense submissions through four Slack channels. A separate Slack app ("Expense Approval Bot") posts to a Submitted channel. Users react with ✅ to advance the message to the next stage.
Channel pipeline:
| Stage | Channel ID | Action on ✅ |
|---|---|---|
| Submitted | C0AQ2AWLNEN |
Thread reply on original, copy to Processed |
| Processed | C0APLSGABAB |
Delete from Processed, post to Authorized |
| Authorized | C0AQ09CDJH4 |
Delete from Authorized, post to Matched |
| Matched | C0APYUM1JFP |
Terminal stage (no further routing) |
Architecture: Two Lambdas — ExpenseReceiver (HTTP endpoint, signature verification, async invoke) and ExpenseProcessor (business logic). This is the same receiver/processor pattern used for Slack's 3-second timeout requirement.
Secrets (Secrets Manager):
| Secret | Purpose |
|---|---|
payments-dashboard/expense-slack-token |
Slack Bot token for the Expense Approval Bot app |
payments-dashboard/expense-slack-signing-secret |
Slack signing secret for request verification |
Payroll Email Pipeline
Gusto sends payroll confirmation emails when payroll is run. A Gmail filter on adam@seahavenind.com auto-forwards emails from automated@gusto.com and gustonoreply@gusto.com to payroll@int.seahaven.com.
Flow: Gmail forward → SES receipt rule → S3 bucket → Lambda parses email → DynamoDB (pending) → SQS delay queue (10 min) → Lambda batches all pending items for that date → single Slack message → DynamoDB (notified)
Deduplication: Each email is deduplicated by DynamoDB key (PAYROLL_EMAIL#employee#<date> or PAYROLL_EMAIL#contractor#<date>#<bank-suffix>). The batch post is deduplicated by PAYROLL_BATCH#<date>. All items have a 90-day TTL.
BoA CashPro API Integration
Two separate CashPro APIs are used, each with its own OAuth credentials:
| API | Purpose | Endpoint |
|---|---|---|
| Check Management | Issue and cancel checks | /cashpro/checkmanagement/v1/check-issues |
| Reporting (Transaction Inquiry) | Fetch previous-day transactions | /cashpro/reporting/v1/transaction-inquiries/previous-day |
Authentication flow:
- POST to
/authn/v1/client-authenticationwithapplicationID,client_id, andclient_secret - Receive a Bearer
access_token(valid 1 hour) - Pass the token in the
Authorizationheader for subsequent API calls
Base URLs:
- Production:
https://api.bofa.com - Sandbox:
https://api-sb.bofa.com
Bank reconciliation (fetchBoaTransactions)
The scheduled Lambda reconciles the Previous Day feed onto payment# records (pure logic lives in src/boaRecon.js, covered by npm test).
Classification. Transactions are classified via a BAI transaction-code map (BAI_CODE_EVENTS; only 475 = check paid debit is empirically confirmed so far — the remaining codes are pending an enumeration replay over known event dates) with a fallback classifier over the statement description formats (ARP refer-to-maker, return-of-posted-check check-numbered and electronic variants, ACH CCD DES:PAYMENTS ID:PMT lines). Unmapped codes on check-shaped transactions are logged (console.error) and counted in the run summary — never silently dropped.
Matching. Check events match on check number AND amount, evaluating all candidates (bank postings can drop/collapse digits on long check numbers). A number match with the WRONG amount never auto-resolves — it is the altered-check/collapsed-posting signal and goes to unmatched for human review. An unknown number falls back to an exact-amount match within checks issued in the last 120 days, and only when the posting's digits are a subsequence of the candidate's check number (or vice versa); return credits additionally require a bank-confirmed candidate. Zero or multiple fallback candidates means unmatched, recorded in the run summary with no write. Electronic returns (no check number) match by exact amount among bank-confirmed payments.
ACH. ACH is bank-confirmed too (#69): processPaymentCsv no longer auto-clears ACH on the send date (rows keep their Stampli status until the bank settles). ACH CCD lines match by the stored pmt_id first (reversals reuse the original PMT <id>; a pmt_id match with the wrong amount is the partial-reversal human case and goes to unmatched), then by the Stampli payment number embedded at the end of PMT INFO (internal spaces stripped, amount must agree), then by vendor + exact amount within send_payment_on −2..+14 days (candidates whose stored pmt_id differs are excluded; vendor prefix matching requires ≥ 10 normalized chars). A settled debit clears the record and persists pmt_id. A return credit whose PMT <id> attributes to no stored pmt_id is an unmatched alert ("unknown PMT id"); credits without a PMT <id> may fall back to a unique same-amount match against a bank-confirmed ACH whose cleared_date is within the prior 45 days.
Staleness sweep. Every run also flags never-bank-confirmed payments: ACH with no clear_status sent more than 16 days ago (listed in the summary, capped at 50, plus a total count) and checks issued more than 60 days ago with no clear_status (count only). Both alert via console.error.
State transitions. Every write sets BOTH status and clear_status (clear_status is bank truth; "Cleared without a subsequent return is permanent" keys off it):
| Bank event | Result |
|---|---|
| Paid debit | status=Cleared, clear_status=Cleared, paid_date, cleared_date, bank_reference |
| Return credit (even if currently Cleared) | clear_status=Returned, returned_date; status re-written unchanged (the CSV ladder has no Returned rung) |
| Second paid debit on a Returned check | Redeposit: back to Cleared with new dates |
| Return on a Stampli-voided check | Terminal voided-and-bounced (clear_status=Returned, cancel status preserved), counted separately |
Each applied event is appended to a history list attribute ({event, date, bankRef, amount}); identical replayed events are idempotent noops.
Replay. The handler accepts an optional payload {"fromDate": "YYYY-MM-DD", "toDate": "YYYY-MM-DD"} (strictly validated; toDate defaults to fromDate) for weekend/outage gap replays and BAI-code enumeration runs. With no payload it queries the trailing 3-day window (today−3 .. today−1), so the Monday run covers Friday–Sunday; overlapping days are idempotent (event identity {event, date, amount}). Feed rows without a valid valueDate are never applied with a substituted date — they go to unmatched for review.
Run summary. Each run writes an append-only boa_recon#<fromDate>_<toDate>#<runAt> item (90-day TTL) with counts per classified event type, matched/applied/redeposit/voided-and-bounced/write-conflict totals, the unmatched check numbers and amounts (list capped at 50; full count kept), unknown BAI codes (capped at 20 distinct keys), and the staleness sweep results. Unmatched and unknown-code transactions also console.error (Slack alerting is tracked in #71). Payment writes are conditioned on the read snapshot's status/clear_status and retried once against a fresh read on conflict.
Documentation
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's payments-dashboard stack is represented there as a Mermaid subgraph.
- AWS Architecture Map (Confluence, IT space, page 1540098)
Secrets
All BoA and Slack credentials are stored in AWS Secrets Manager (per engineering-handbook/secrets-and-config.md). The Slack token is a plaintext secret; the two BoA secrets are JSON grouping each API's credentials:
| Secret | Type | Contents |
|---|---|---|
payments-dashboard/slack-bot-token |
plaintext | Slack Bot OAuth token (used by processPayrollEmail, slackAppHome) |
payments-dashboard/slack-signing-secret |
plaintext | Slack signing secret for slackAppHome request verification |
payments-dashboard/boa-check-mgmt |
JSON | appId, clientId, token, accountNumber, companyId — Check Management API (processPaymentCsv) |
payments-dashboard/boa-reporting |
JSON | appId, clientId, token, accountNumber, bankId — Reporting API (fetchBoaTransactions) |
boa-account-number is duplicated into both BoA secrets. Each Lambda is granted secretsmanager:GetSecretValue scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (payments-dashboard/expense-slack-token, payments-dashboard/expense-slack-signing-secret).
Consumers / data contract
The PaymentsDashboard DynamoDB table (AWS::DynamoDB::Table, TableName: PaymentsDashboard, PK pk (S), CMK-encrypted) is owned by this stack, which is the sole authoritative writer.
Consumer (read-only): seahaven-slack-bot imports this table via Table.fromTableName(...) and reads it read-only (grantReadData plus an explicit kms:Decrypt grant on the shared CMK) from its wo-po-lookup Lambda, which backs the Bedrock agent's payment-lookup action group. The bot depends on:
- Key schema: PK
pk(S) with the item formatpayment#<check_number>. It doesGetItembypkand a full-tableScanfilteredbegins_with(pk, "payment#"). - Attributes:
check_number,payee,amount_usd,method,status,send_payment_on,clear_status,cleared_date,invoice_numbers,company_subsidiary,bank_reference. - Encryption: the shared customer-managed CMK (
/seahaven/dynamodb/cmk-arn). Because the consumer imports the table by name,grantReadDatadoes not carry KMS access; a change of CMK requires re-granting on the consumer side or every read fails withkms:Decrypt AccessDenied(INFRA-95 / M-3 precedent).
The table is imported by name, so there is no compile-time link between the stacks: any change to the table name, pk format, these attribute names, the encryption key, or the table's lifecycle policy will silently break the Bedrock agent at runtime. Coordinate such changes with seahaven-slack-bot before shipping (INFRA-138).
Monitoring & Alarms
All CloudWatch alarms publish to the shared site-alerts SNS topic (arn:aws:sns:us-east-1:328440206208:site-alerts). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as notBreaching. Each alarm evaluates a single 5-minute period.
SQS dead-letter queues (messages-present, Maximum > 0):
| Alarm | Source |
|---|---|
payments-payroll-batch-dlq-messages |
payments-payroll-batch-dlq |
payments-processPaymentCsv-async-dlq-messages |
async-invoke OnFailure DLQ |
payments-processPayrollEmail-async-dlq-messages |
async-invoke OnFailure DLQ |
Lambda (per function — payments-<fn>-...):
| Type | Metric / Statistic | Threshold |
|---|---|---|
-errors (all 6) |
Errors / Sum |
> 0 |
-throttles (all 6) |
Throttles / Sum |
> 0 |
-duration (all 6) |
Duration / Maximum |
~80% of each function's timeout |
Duration thresholds (ms): processPaymentCsv 96000, processPayrollEmail 48000, fetchBoaTransactions 48000, slackAppHome 24000, expenseProcessor 12000, expenseReceiver 4000.
DynamoDB (PaymentsDashboard table, TableName dimension, Sum > 0): payments-dashboard-table-read-throttle (ReadThrottleEvents), payments-dashboard-table-write-throttle (WriteThrottleEvents). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. SystemErrors is intentionally not alarmed because it does not emit at the TableName-only dimension.
API Gateway (implicit HTTP API v2 ServerlessHttpApi, ApiId dimension): payments-dashboard-api-5xx (5xx Sum > 0), payments-dashboard-api-4xx (4xx Sum > 10, client-error noise floor), payments-dashboard-api-latency-p99 (Latency p99 > 3000 ms).
Scripts
| Script | Purpose |
|---|---|
scripts/test-boa-sandbox.js |
One-off sandbox connectivity test for both CashPro APIs |
scripts/seed-from-csv.js |
Seed DynamoDB from a local CSV file |
scripts/seed-bank-status.js |
Seed bank clear status data into DynamoDB |
Deployment
sam build
sam deploy --guided
The BOA_BASE_URL environment variable in template.yaml controls whether Lambdas hit production (https://api.bofa.com) or sandbox (https://api-sb.bofa.com). All other BoA config is read from Secrets Manager at runtime.