fix(csv): M/D/YY date parsing with loud rejects; stop canceled rows zeroing stored fields (#72)
* fix(csv): parse M/D/YY dates with loud rejects; stop canceled rows zeroing stored fields (#65, #68)
Stampli exports switched from MM/DD/YYYY to M/D/YY, which toISODate
mangled into garbage ISO strings and slackAppHome's parseMDYLocal turned
into year-1926 dates. Canceled rows also blank 'Amount in USD' and can
blank dates, which the unconditional upsert wrote over previously stored
real values (116 records at amount_usd=0 as of the 2026-07-21
reconciliation).
- src/dates.js: shared strict parser for M/D/YY + MM/DD/YYYY (real-date
validation, 2-digit years 2000-based, plausibility window helper)
- processPaymentCsv: canonicalize send_payment_on to MM/DD/YYYY; reject
rows with unparseable/implausible dates and fail the invocation after
valid rows are processed (surfaces via errors alarm + async DLQ;
retry-safe since upserts are idempotent and existing checks are not
re-offered to BoA); only overwrite amount_usd/send_payment_on with
real values, falling back to the 'Amount' column on cancels
- BoA add_Issue/cancel_Issue now use stored record values when the CSV
row is blank (cancel_Issue previously sent amount 0.00 for voids) and
skip registration on missing date/amount instead of sending garbage;
same guard on the backfill path
- slackAppHome: use the shared parser (fixes 1926 aging)
* fix(csv): validate BoA registration data before writes; harden logging (security review)
Hardening from the /sh-security-review pass on this branch:
- BoA eligibility (resolvable amount + issue date) is now decided and
validated BEFORE the DDB upsert: a cancel-transition row we cannot act
on is rejected with the record untouched, so the transition gate stays
open for a later clean file instead of silently losing the cancel
forever (the skip guard previously ran after the status write)
- First-seen rows that are already canceled are stored but never
offered to add_Issue — registering a voided check as an active issue
created a live issue with no cancel to follow (worsened by the Amount
fallback making the previously-failing call succeed)
- isPlausibleSendYear window is now relative (year-7..year+2) instead
of hardcoded 2020-2035
- Untrusted CSV values are JSON-encoded and truncated before log
interpolation (quoted CSV cells carry newlines -> CloudWatch log-line
forgery, CWE-117)
- OAuth token-exchange failures no longer throw the raw BoA authn
response body (aligns with the PR #63 log-scrub posture)
Verified: unit smoke on the date module; full replay of the real
2026-07-21 export (1,197 rows) against live table state produces 1,197
upserts, 0 rejects, 0 spurious BoA submissions.
* fix(csv): comma-tolerant amount parsing in backfill (PR #72 review)
Number() on a hand-inserted comma-formatted string amount would NaN and
skip the check during backfill; hoist the CSV path's parseAmount to
module scope and share it. No live records are affected (all amount_usd
values are DDB Number type) — defensive consistency.
2026-07-21 20:28:13 -04:00
|
|
|
// Shared date parsing for Stampli M/D/YY and MM/DD/YYYY formats.
|
|
|
|
|
// Stampli exports switched from MM/DD/YYYY to M/D/YY in 2026-06/07; both
|
|
|
|
|
// formats must parse, and anything else must be rejected (null), never
|
|
|
|
|
// passed through as a malformed string (payments-dashboard#65).
|
|
|
|
|
|
|
|
|
|
// Parse "M/D/YY" or "MM/DD/YYYY" into { year, month, day }, or null.
|
|
|
|
|
// Two-digit years are 2000-based. Validates the date is real (no 2/30).
|
|
|
|
|
export function parseMDY(value) {
|
|
|
|
|
const match = /^(\d{1,2})\/(\d{1,2})\/(\d{2}|\d{4})$/.exec(String(value ?? "").trim());
|
|
|
|
|
if (!match) return null;
|
|
|
|
|
const month = parseInt(match[1], 10);
|
|
|
|
|
const day = parseInt(match[2], 10);
|
|
|
|
|
let year = parseInt(match[3], 10);
|
|
|
|
|
if (match[3].length === 2) year += 2000;
|
|
|
|
|
const dt = new Date(year, month - 1, day);
|
|
|
|
|
if (dt.getFullYear() !== year || dt.getMonth() !== month - 1 || dt.getDate() !== day) {
|
|
|
|
|
return null;
|
|
|
|
|
}
|
|
|
|
|
return { year, month, day };
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// "YYYY-MM-DD" or null.
|
|
|
|
|
export function toISODate(value) {
|
|
|
|
|
const p = parseMDY(value);
|
|
|
|
|
if (!p) return null;
|
|
|
|
|
return `${p.year}-${String(p.month).padStart(2, "0")}-${String(p.day).padStart(2, "0")}`;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Canonical "MM/DD/YYYY" or null. Stored form for send_payment_on.
|
|
|
|
|
export function toCanonicalMDY(value) {
|
|
|
|
|
const p = parseMDY(value);
|
|
|
|
|
if (!p) return null;
|
|
|
|
|
return `${String(p.month).padStart(2, "0")}/${String(p.day).padStart(2, "0")}/${p.year}`;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Local-midnight Date or null. Replaces slackAppHome's parseMDYLocal, which
|
|
|
|
|
// built year-1926 dates from 2-digit years via new Date(26, ...).
|
|
|
|
|
export function parseMDYLocal(value) {
|
|
|
|
|
const p = parseMDY(value);
|
|
|
|
|
if (!p) return null;
|
|
|
|
|
return new Date(p.year, p.month - 1, p.day);
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-22 16:38:56 -04:00
|
|
|
// Local-midnight Date from a stored ISO "YYYY-MM-DD" (returned_date,
|
|
|
|
|
// paid_date), or null. new Date("YYYY-MM-DD") parses as UTC midnight and
|
|
|
|
|
// renders a day early in US-local display; this stays local like
|
|
|
|
|
// parseMDYLocal. Validates the date is real (no 2026-02-30).
|
|
|
|
|
export function parseISOLocal(value) {
|
|
|
|
|
const match = /^(\d{4})-(\d{2})-(\d{2})$/.exec(String(value ?? "").trim());
|
|
|
|
|
if (!match) return null;
|
|
|
|
|
const year = parseInt(match[1], 10);
|
|
|
|
|
const month = parseInt(match[2], 10);
|
|
|
|
|
const day = parseInt(match[3], 10);
|
|
|
|
|
const dt = new Date(year, month - 1, day);
|
|
|
|
|
if (dt.getFullYear() !== year || dt.getMonth() !== month - 1 || dt.getDate() !== day) {
|
|
|
|
|
return null;
|
|
|
|
|
}
|
|
|
|
|
return dt;
|
|
|
|
|
}
|
|
|
|
|
|
fix(csv): M/D/YY date parsing with loud rejects; stop canceled rows zeroing stored fields (#72)
* fix(csv): parse M/D/YY dates with loud rejects; stop canceled rows zeroing stored fields (#65, #68)
Stampli exports switched from MM/DD/YYYY to M/D/YY, which toISODate
mangled into garbage ISO strings and slackAppHome's parseMDYLocal turned
into year-1926 dates. Canceled rows also blank 'Amount in USD' and can
blank dates, which the unconditional upsert wrote over previously stored
real values (116 records at amount_usd=0 as of the 2026-07-21
reconciliation).
- src/dates.js: shared strict parser for M/D/YY + MM/DD/YYYY (real-date
validation, 2-digit years 2000-based, plausibility window helper)
- processPaymentCsv: canonicalize send_payment_on to MM/DD/YYYY; reject
rows with unparseable/implausible dates and fail the invocation after
valid rows are processed (surfaces via errors alarm + async DLQ;
retry-safe since upserts are idempotent and existing checks are not
re-offered to BoA); only overwrite amount_usd/send_payment_on with
real values, falling back to the 'Amount' column on cancels
- BoA add_Issue/cancel_Issue now use stored record values when the CSV
row is blank (cancel_Issue previously sent amount 0.00 for voids) and
skip registration on missing date/amount instead of sending garbage;
same guard on the backfill path
- slackAppHome: use the shared parser (fixes 1926 aging)
* fix(csv): validate BoA registration data before writes; harden logging (security review)
Hardening from the /sh-security-review pass on this branch:
- BoA eligibility (resolvable amount + issue date) is now decided and
validated BEFORE the DDB upsert: a cancel-transition row we cannot act
on is rejected with the record untouched, so the transition gate stays
open for a later clean file instead of silently losing the cancel
forever (the skip guard previously ran after the status write)
- First-seen rows that are already canceled are stored but never
offered to add_Issue — registering a voided check as an active issue
created a live issue with no cancel to follow (worsened by the Amount
fallback making the previously-failing call succeed)
- isPlausibleSendYear window is now relative (year-7..year+2) instead
of hardcoded 2020-2035
- Untrusted CSV values are JSON-encoded and truncated before log
interpolation (quoted CSV cells carry newlines -> CloudWatch log-line
forgery, CWE-117)
- OAuth token-exchange failures no longer throw the raw BoA authn
response body (aligns with the PR #63 log-scrub posture)
Verified: unit smoke on the date module; full replay of the real
2026-07-21 export (1,197 rows) against live table state produces 1,197
upserts, 0 rejects, 0 spurious BoA submissions.
* fix(csv): comma-tolerant amount parsing in backfill (PR #72 review)
Number() on a hand-inserted comma-formatted string amount would NaN and
skip the check during backfill; hoist the CSV path's parseAmount to
module scope and share it. No live records are affected (all amount_usd
values are DDB Number type) — defensive consistency.
2026-07-21 20:28:13 -04:00
|
|
|
// Plausibility window for ingested send dates. Anything outside is treated
|
|
|
|
|
// as parser garbage and the row is rejected loudly rather than stored.
|
|
|
|
|
// Relative to the current year so it never expires (7 years back covers
|
|
|
|
|
// historical re-ingests; 2 years forward covers future-scheduled checks).
|
|
|
|
|
export function isPlausibleSendYear(value) {
|
|
|
|
|
const p = parseMDY(value);
|
|
|
|
|
if (p === null) return false;
|
|
|
|
|
const now = new Date().getFullYear();
|
|
|
|
|
return p.year >= now - 7 && p.year <= now + 2;
|
|
|
|
|
}
|