procurement-ingest/.claude/workflows/phase-6-site-extractor.js

export const meta = {
  name: 'phase-6-site-extractor',
  description: 'Phase 6 of the procurement-ingest refactor (docs/refactor-evaluation.md): site_extractor reconciliation — end the three-way site_code contradiction. extract_site_code (lambdas/po/site_extractor/handler.py:34) already prefers record["site_code"] but validates it with the digit-requiring, prefix-anchored SITE_CODE_PATTERN (line 23) which rejects all-letter codes like KLAL (permanent pending-review rows) and accepts overlong junk like DLI6X/SNY55. Fix: validate the direct field with the canonical derived_fields shape + skip-list semantics (fullmatch), import derive_site_code for the fallback path, ship derived_fields.py into the site_extractor asset via the Phase 2/3 bundling mechanism (Phase 3 left site_extractor from_asset UNTOUCHED on purpose), delete the dead [A-Z]{4,5} regex ladder (line 62). derived_fields.py itself stays byte-untouched. Characterization tests pin current behavior first, then flip to fixed in the same PR. Gated on the shadow bake concluding. Validated field can be ai_fallback-sourced, so push is gated on /sh-security-review. Committed locally, never pushed.',
  phases: [
    { title: 'Setup', detail: 'HARD-GATE on shadow-bake conclusion (args {bakeConcluded:true}) + Phase 3 on base; branch feature/phase-6-site-extractor', model: 'haiku' },
    { title: 'Recon', detail: '3 mappers: site_extractor handler + the digit-pattern defect, derived_fields canonical shape/skip-list/derive_site_code, post-Phase-3 bundling sites + bundle-consistency test + backfill script (read-only)' },
    { title: 'Spec', detail: 'serial fable spec: pin the handler fix (canonical fullmatch + skip-list), the site_extractor bundling change, backfill repoint-or-delete, the characterization-first-then-flip test plan, bundle/parity rules' },
    { title: 'Implement', detail: 'opus: site_extractor handler + new characterization tests; sonnet: cdk site_extractor from_asset + test_bundle_consistency; opus: backfill_sites.py + README — disjoint files', model: 'opus' },
    { title: 'Verify', detail: 'mechanical gates (suite green, ruff, synth, asset CONTAINS derived_fields.py, tests flipped to fixed, zero derived_fields diff) + 3 fable lenses (shape-parity, bundling, regression)' },
    { title: 'Fix', detail: 'opus fixer, full re-verify, max 3 rounds', model: 'opus' },
    { title: 'Package', detail: 'single commit via -F (no push)', model: 'sonnet' },
  ],
}

// ---------------------------------------------------------------- constants

const REPO = '/Users/adammoussa/Documents/repositories/seahaven/procurement-ingest'
const BRANCH = 'feature/phase-6-site-extractor'
let _args = args
if (typeof _args === 'string') {
  try { _args = JSON.parse(_args) } catch (e) { _args = null }
}
const BASE = (_args && _args.base) || 'main'

const CONSTRAINTS = `
PINNED BEHAVIORAL CONSTRAINTS (docs/refactor-evaluation.md Phase 6 + §4 item 6 — violating any is a build failure):
1. THE DEFECT is in lambdas/po/site_extractor/handler.py: extract_site_code
   (line 34) already prefers record["site_code"] (line 36), but validates it
   with SITE_CODE_PATTERN = re.compile(r"[A-Z]{2,4}\\d{1,2}") (line 23) applied
   via .match (prefix-anchored, digit-REQUIRING). That pattern REJECTS
   all-letter codes like KLAL — so a KLAL-class site can NEVER self-register
   and becomes a permanent pending-review row — and ACCEPTS overlong junk like
   DLI6X / SNY55 (prefix match, no end anchor). THE FIX: validate the direct
   record["site_code"] field with the CANONICAL derived_fields shape + skip-list
   semantics using FULLMATCH (derived_fields.py: _STRICT_CODE_RE at line 50 =
   re.compile(r"[A-Z][A-Z0-9]{2,4}") + _is_valid_code at line 97 = "not in _SKIP
   and >=2 alphabetic chars", _SKIP at line 69). Import derive_site_code
   (derived_fields.py:259) for the FALLBACK path (ship_to / line_items). DELETE
   the bespoke regex ladder — the line 62 alternation
   r"^([A-Z]{1,4}[0-9]{1,2}|[A-Z]{4,5})\\b" whose [A-Z]{4,5} arm is dead code
   (it can never survive the digit-requiring SITE_CODE_PATTERN.match on the very
   next line) — and every other SITE_CODE_PATTERN reference. NET EXPECTED
   BEHAVIOR: KLAL is ACCEPTED, DLI6X and SNY55 are REJECTED.
2. SHIP derived_fields.py INTO the site_extractor Lambda asset. Add the cp/root
   bundling change for site_extractor's from_asset in cdk/po_stack.py (currently
   lines 684-686: Code.from_asset("../lambdas/po/site_extractor",
   exclude=["**/__pycache__/**"]) — Phase 3 left this UNTOUCHED on purpose).
   THIS phase makes lambdas/po/email_processor/derived_fields.py reachable there,
   MIRRORING the Phase 2/3 mechanism the two email processors use (widened
   asset root + a scoped cp so the module lands FLAT in /asset-output so the
   bare-name import 'from derived_fields import derive_site_code' resolves).
   KEEP the __pycache__ / tests excludes intact; do NOT ship tests/ or a stale
   package/ into the zip. The spec agent pins the exact root + cp form (Docker
   bundling vs a no-Docker local staging — ONE mechanism, exact code).
3. Keep shape/skip validation ON the record["site_code"] field even though it
   is ai_fallback-SOURCED (the LLM value is authoritative during/after the bake)
   — this is validated authority, NOT blind trust and NOT a regex bypass. Keep
   parse_address (line 69) AND the pending-review write flow (write_pending_review
   line 223 + the handler else-branch lines 283-290) BYTE-UNCHANGED.
4. Repoint scripts/backfill_sites.py at derive_site_code (it currently does
   'from handler import extract_site_code, parse_address, upsert_site' via a
   sys.path hack) — OR DELETE it entirely (one-time backfill script; deletion is
   acceptable and PREFERRED if simpler). Pick one and state why.
5. CHARACTERIZATION TESTS FIRST. site_extractor is 0% covered today (§2). Write
   tests that PIN CURRENT behavior first — INCLUDING the KLAL rejection
   divergence and the overlong-junk (DLI6X / SNY55) acceptance, parse_address
   variants, and the pending-review fallback — then FLIP the assertions to the
   FIXED behavior in the SAME PR, so the change is provably intentional. The
   final committed suite asserts FIXED behavior (KLAL accepted, DLI6X/SNY55
   rejected); the characterization-of-old-behavior stage is a build step, not a
   red suite left behind.
6. derived_fields.py itself stays UNTOUCHED — import it, do NOT edit it.
   git diff ${BASE}...HEAD -- lambdas/po/email_processor/derived_fields.py MUST
   be empty. The shadow-bake authority model must not shift in this phase.
7. AWS access is READ-ONLY (get-function, downloading the deployed site_extractor
   zip via presigned URL to confirm the asset contents). NEVER cdk deploy, never
   invoke, never mutate. No push, no PR, no cdk deploy in this workflow.
`

const PREAMBLE = `
You are one of several agents building refactor Phase 6 in the git repo at
${REPO} on branch ${BRANCH} (already checked out — do NOT switch branches,
do NOT create branches, do NOT commit, NEVER push, do NOT run cdk deploy or
touch AWS resources beyond read-only calls).
Authoritative spec: docs/refactor-evaluation.md, section "Phase 6 —
site_extractor reconciliation" and section "§4" item 6 (site_extractor
characterization). The doc wins on any conflict.
Work ONLY in the files you are told you own; other agents are concurrently
editing other files in this same working tree.
${CONSTRAINTS}
Your final message is consumed by an orchestrator script, not a human —
return only the structured data requested.
`

// ------------------------------------------------------------------ schemas

const RECON = {
  type: 'object',
  required: ['summary', 'facts'],
  properties: {
    summary: { type: 'string' },
    facts: { type: 'array', items: { type: 'string' } },
    blockers: { type: 'array', items: { type: 'string' } },
  },
}

const SPEC = {
  type: 'object',
  required: ['handlerFix', 'bundlingEdit', 'backfillDecision', 'characterizationTests', 'parityRules', 'notes'],
  properties: {
    handlerFix: { type: 'string', description: 'the exact edit to lambdas/po/site_extractor/handler.py: how the direct record["site_code"] field is validated with the canonical derived_fields fullmatch shape + skip-list, the derive_site_code import + fallback wiring, every line deleted (SITE_CODE_PATTERN def, the line-62 alternation, all SITE_CODE_PATTERN references), and an explicit statement that parse_address + write_pending_review + the pending-review else-branch are byte-unchanged. Prove KLAL accepted, DLI6X/SNY55 rejected under the new logic.' },
    bundlingEdit: { type: 'string', description: 'the complete replacement from_asset block for site_extractor in cdk/po_stack.py: widened asset root, the scoped cp form staging both the site_extractor *.py AND lambdas/po/email_processor/derived_fields.py flat, the __pycache__/tests/package excludes, and whether it uses Docker bundling or a no-Docker local staging — ONE mechanism, exact code, mirroring the Phase 2/3 email-processor form.' },
    backfillDecision: { type: 'string', description: 'repoint scripts/backfill_sites.py at derive_site_code OR delete it — which, the exact resulting file (or git rm), and why.' },
    characterizationTests: { type: 'string', description: 'the new test file(s) under lambdas/po/site_extractor/tests/: how they load the handler (loader idiom), the CURRENT-behavior pins (KLAL rejected, DLI6X/SNY55 accepted, parse_address variants, pending-review fallback) and the FIXED-behavior assertions they flip to, plus any test_bundle_consistency edit if it now pins the site_extractor asset.' },
    parityRules: { type: 'string', description: 'how Verify judges the change: the staged site_extractor asset must CONTAIN derived_fields.py (+ the handler and its own siblings, flat) and nothing stray (no tests/, no __pycache__, no package/), asset hash deterministic, and the derived_fields.py diff must be empty vs base.' },
    notes: { type: 'string' },
  },
}

const IMPL = {
  type: 'object',
  required: ['filesChanged', 'summary', 'checksRun'],
  properties: {
    filesChanged: { type: 'array', items: { type: 'string' } },
    summary: { type: 'string' },
    checksRun: { type: 'string' },
    blockers: { type: 'array', items: { type: 'string' } },
  },
}

const CHECKS = {
  type: 'object',
  required: ['passed', 'details'],
  properties: {
    passed: { type: 'boolean' },
    details: { type: 'string' },
    scopeViolations: { type: 'array', items: { type: 'string' } },
  },
}

const FINDINGS = {
  type: 'object',
  required: ['findings'],
  properties: {
    findings: {
      type: 'array',
      items: {
        type: 'object',
        required: ['title', 'severity', 'confirmed', 'evidence', 'fix'],
        properties: {
          title: { type: 'string' },
          severity: { enum: ['critical', 'high', 'medium', 'low'] },
          confirmed: { type: 'boolean' },
          evidence: { type: 'string' },
          fix: { type: 'string' },
        },
      },
    },
  },
}

// ------------------------------------------------------------------- setup

phase('Setup')

// HARD-GATE (a): the shadow bake must have concluded. This workflow cannot
// auto-verify the bake ended, so require an explicit opt-in BEFORE any git work.
if (!_args || _args.bakeConcluded !== true) {
  return {
    aborted: 'Phase 6 gated on shadow-bake conclusion',
    blockers: [
      'Phase 6 gated on shadow-bake conclusion — re-run with args {bakeConcluded:true} once the DerivedFieldAgreement bake is signed off and the switchover PR (Python authoritative on ai_fallback) has landed.',
    ],
  }
}

const setup = await agent(`
In ${REPO}:
1. SEQUENCING GATE — Phase 3 (lambdas/shared/ extraction with the widened
   ../lambdas asset roots) must be on ${BASE}: derived_fields.py must be
   shippable into the site_extractor asset via the Phase 2/3 bundling
   mechanism, and Phase 3 DELIBERATELY left site_extractor's from_asset
   untouched — THIS phase adds the bundling change for site_extractor.
   git fetch origin, then pick the base ref: origin/${BASE} if that remote ref
   exists, otherwise the local branch ${BASE} (a stacked local-only base is
   expected and fine). Verify on the base ref:
   (a) Phase 2/3 mechanism present: cdk/po_stack.py on the base ref contains
       Code.from_asset("../lambdas") for the PO email processor
       (git show <baseref>:cdk/po_stack.py | grep -n '\\.\\./lambdas');
   (b) Phase 3 landed: lambdas/shared/ exists on the base ref with a
       cp shared/*.py in the email-processor bundling command
       (git show <baseref>:cdk/po_stack.py | grep -n 'shared'), and
       git show <baseref>:lambdas/shared/ses_auth.py | head -3 succeeds;
   (c) site_extractor's from_asset is STILL the untouched narrow form on the
       base ref (git show <baseref>:cdk/po_stack.py | grep -n
       '../lambdas/po/site_extractor') — this phase is the one that widens it.
   If (a) or (b) is missing, STOP with a blocker naming the unmet phase and do
   nothing else. If (c) is already widened, note it (someone beat us to it) but
   do not treat as a hard blocker.
2. Verify clean working tree (untracked .coverage / .claude/ / the local 44 MB
   lambdas/po/email_processor/package/ dir are fine; any OTHER dirt = blocker,
   never stash or discard).
3. git checkout ${BASE}; then git pull --ff-only ONLY if the branch has an
   upstream (a local-only base skips the pull — not a blocker); then
   git checkout -b ${BRANCH}
4. gh pr list --state open --json number,title,headRefName (overlap check).
Return facts: HEAD sha, per-gate evidence, open PRs, blockers.
`, { label: 'setup:branch', model: 'haiku', schema: RECON })

if (!setup || (setup.blockers && setup.blockers.length)) {
  return { aborted: 'setup blockers', blockers: setup ? setup.blockers : ['setup agent died'], facts: setup ? setup.facts : [] }
}
log(`Branch ${BRANCH} ready off ${BASE}. ${setup.summary}`)

// ------------------------------------------------------------------- recon

phase('Recon')
const recon = await parallel([
  () => agent(`${PREAMBLE}
Read-only recon of the site_extractor DEFECT surface
(lambdas/po/site_extractor/handler.py):
1. Quote SITE_CODE_PATTERN verbatim with file:line and EVERY place it is used
   (.match / .search). Note that .match is prefix-anchored and the pattern
   requires 1-2 trailing digits.
2. Quote extract_site_code in full with line numbers: the direct-field branch
   (record["site_code"]), the ship_to name branches, the line_items branch and
   its r"^([A-Z]{1,4}[0-9]{1,2}|[A-Z]{4,5})\\b" alternation — prove the
   [A-Z]{4,5} arm is DEAD (it can never survive the very-next-line
   SITE_CODE_PATTERN.match, which requires a digit).
3. Concretely trace KLAL (all letters), DLI6, DLI6X, SNY55 through the CURRENT
   code — which return, which fall through — to pin the exact current behavior
   the characterization tests must first reproduce.
4. Quote the EXACT boundaries (file:line) of parse_address, write_pending_review
   and the handler else-branch that calls write_pending_review — these must stay
   byte-unchanged; note everything that must NOT move.
5. Confirm there is NO tests/ dir under lambdas/po/site_extractor today and note
   how the email-processor test loaders resolve a bare 'import handler' (the
   idiom the new characterization tests will reuse).
20-30 precise facts.`,
    { label: 'recon:site-extractor', model: 'sonnet', phase: 'Recon', schema: RECON }),

  () => agent(`${PREAMBLE}
Read-only recon of the CANONICAL shape in
lambdas/po/email_processor/derived_fields.py that the fix must adopt:
1. Quote _STRICT_CODE_RE (line ~50), _CODE_TOKEN_RE, _SKIP (line ~69) and
   _is_valid_code (line ~97) verbatim with file:line. State exactly what shape
   counts as a valid site code (3-5 chars, letter-first, all-letters allowed,
   >=2 alphabetic chars, not skip-listed) using FULLMATCH.
2. Quote derive_site_code (line ~259) signature + docstring + return contract
   (str | None, NEVER raises) and describe what INPUT it takes (the parsed PO
   dict) and the priority-ordered shapes it tries — so the fallback wiring in
   the handler passes the right shape of dict.
3. Determine precisely: under the canonical fullmatch shape + _is_valid_code,
   is KLAL valid? is DLI6X valid? is SNY55 valid? Show the reasoning token by
   token. If the canonical shape does NOT by itself reject DLI6X/SNY55, say so
   explicitly and identify what additional check (standalone-token lookaround,
   length, _is_valid_code, or applying derive-style resolution to the field)
   is required to hit the doc's pinned outcome (KLAL accepted; DLI6X/SNY55
   rejected) — this is load-bearing for the spec.
4. List every import derived_fields.py needs at module load (must be
   stdlib-only so it can be dropped flat into the site_extractor asset with no
   new dependency).
15-25 facts. Flag as a blocker any way the pinned KLAL/DLI6X/SNY55 outcome
cannot be met with derived_fields semantics alone.`,
    { label: 'recon:derived-fields-shape', model: 'sonnet', phase: 'Recon', schema: RECON }),

  () => agent(`${PREAMBLE}
Read-only recon of the bundling + test + script surface this phase touches:
1. cdk/po_stack.py — quote the site_extractor Function block verbatim with
   file:line (currently ~677-694): function_name, runtime, architecture,
   handler property, memory/timeout, env, and the from_asset call
   (root + exclude) that this phase must widen. Also quote the PO
   email-processor from_asset (root ../lambdas + the cp glob + exclude list)
   as the Phase 2/3 template to mirror. Confirm no requirements.txt exists for
   site_extractor. List every function property that must NOT change.
2. tests/test_bundle_consistency.py — quote PO_EXPECTED_TOP_LEVEL_MODULES and
   every test; determine whether ANY assertion currently pins the
   site_extractor asset (if none does, note that a new site_extractor pin —
   asserting derived_fields.py now ships and a commented-out cp fails — is the
   clean way to keep teeth; if one does, quote it).
3. scripts/backfill_sites.py — quote the sys.path hack + the
   'from handler import extract_site_code, parse_address, upsert_site' line and
   how it uses extract_site_code, so the spec can choose repoint vs delete.
4. Read-only AWS (us-east-1): aws lambda get-function for
   po-ingest-site-extractor; download its Code.Location zip to a scratch dir and
   produce the COMPLETE unzip -l file list (the baseline the new bundled asset
   must cover PLUS derived_fields.py); record CodeSha256.
15-25 facts.`,
    { label: 'recon:bundling-tests-script', model: 'sonnet', phase: 'Recon', schema: RECON }),
])

const reconOk = recon.filter(Boolean)
const pack = reconOk.map(r => `## ${r.summary}\n${r.facts.join('\n')}`).join('\n\n')
const reconBlockers = reconOk.flatMap(r => r.blockers || [])
log(`Recon complete: ${reconOk.length}/3 mappers, ${reconBlockers.length} blockers`)
if (reconBlockers.length) {
  return { aborted: 'recon blockers (likely a shape-parity gap — resolve before implementing)', blockers: reconBlockers, reconPack: pack }
}

// -------------------------------------------------------------------- spec

phase('Spec')
const spec = await agent(`${PREAMBLE}
You are the SPEC agent — the single authority that pins every contested
decision BEFORE parallel implementation (parallel leaves cannot see each
other's choices). Using the recon pack below plus your own reads of the actual
files, produce the binding implementation spec:
- handlerFix: the exact edit to lambdas/po/site_extractor/handler.py.
  Validate record["site_code"] with the CANONICAL derived_fields fullmatch
  shape + skip-list (constraint 1) so KLAL is ACCEPTED and DLI6X/SNY55 are
  REJECTED — if recon found the bare fullmatch shape does not reject
  DLI6X/SNY55, pin the exact additional check that does (do NOT reintroduce a
  digit requirement). Import derive_site_code from derived_fields for the
  fallback path; wire it with the correct input shape. DELETE the
  SITE_CODE_PATTERN def, the line-62 alternation, and every SITE_CODE_PATTERN
  reference. State EXPLICITLY that parse_address, write_pending_review and the
  pending-review else-branch are byte-unchanged.
- bundlingEdit: the complete replacement from_asset block for site_extractor —
  widened root + scoped cp staging both site_extractor *.py and
  lambdas/po/email_processor/derived_fields.py FLAT into /asset-output, with
  the __pycache__/tests/package excludes, mirroring the Phase 2/3
  email-processor form. Pick ONE mechanism (Docker bundling vs no-Docker local
  staging) and give exact code. Mind that the module must land flat so
  'from derived_fields import derive_site_code' resolves at runtime.
- backfillDecision: repoint scripts/backfill_sites.py at derive_site_code OR
  git rm it — pick one, give the exact resulting file or the rm, and say why.
- characterizationTests: the new tests under lambdas/po/site_extractor/tests/ —
  loader idiom, the CURRENT-behavior pins (KLAL rejected, DLI6X/SNY55 accepted,
  parse_address variants, pending-review fallback) that get FLIPPED to
  FIXED-behavior assertions in the same change, and any test_bundle_consistency
  edit that now pins the site_extractor asset (derived_fields ships;
  commented-out cp fails).
- parityRules: exactly how Verify judges the staged asset vs the deployed
  baseline — staged site_extractor asset CONTAINS derived_fields.py + handler +
  own siblings (flat), nothing stray, asset hash deterministic, and
  derived_fields.py diff empty vs ${BASE}.
Recon pack:\n${pack}`,
  { label: 'spec:pin-site-extractor', phase: 'Spec', schema: SPEC })

if (!spec) return { aborted: 'spec agent died — rerun workflow', reconBlockers }
const specBlock = `BINDING SPEC (from the spec agent — implement EXACTLY this):\n${JSON.stringify(spec, null, 2)}`
log('Spec pinned: handler fix, site_extractor bundling, backfill decision, characterization tests, parity rules')

// --------------------------------------------------------------- implement

phase('Implement')
const impl = await parallel([
  () => agent(`${PREAMBLE}
YOU OWN: lambdas/po/site_extractor/ ONLY (the handler AND a new tests/
subdir). Do not touch cdk/, scripts/, tests/ at repo root, or README.
Task: execute spec.handlerFix + spec.characterizationTests.
1. Edit handler.py per spec.handlerFix: validate record["site_code"] with the
   canonical derived_fields fullmatch shape + skip-list, import
   derive_site_code for the fallback path, DELETE SITE_CODE_PATTERN and the
   line-62 [A-Z]{4,5} alternation and every SITE_CODE_PATTERN reference. Keep
   parse_address, write_pending_review and the pending-review else-branch
   BYTE-UNCHANGED (constraint 3). Do NOT edit derived_fields.py (constraint 6).
2. Write the characterization tests under lambdas/po/site_extractor/tests/:
   FIRST pin CURRENT behavior (KLAL rejected, DLI6X/SNY55 accepted,
   parse_address variants, pending-review fallback), THEN flip the assertions
   to the FIXED behavior (KLAL accepted, DLI6X/SNY55 rejected) in this same
   change — the COMMITTED suite asserts FIXED behavior (constraint 5). The
   tests must import derived_fields via the same flat/loader idiom the asset
   uses so they exercise the real derive_site_code.
Run before returning: ruff check lambdas/po/site_extractor && ruff format
lambdas/po/site_extractor --check, and pytest -q --no-cov on your new suite
(with derived_fields.py resolvable on sys.path the way the asset ships it —
stub AWS/boto env as needed). git diff ${BASE}...HEAD --
lambdas/po/email_processor/derived_fields.py MUST be empty.
${specBlock}`,
    { label: 'impl:site-extractor', model: 'opus', phase: 'Implement', schema: IMPL }),

  () => agent(`${PREAMBLE}
YOU OWN: cdk/po_stack.py (the site_extractor from_asset region ONLY — alarms,
IAM, tables, env, and the email-processor/web_ui assets are all off-limits) and
tests/test_bundle_consistency.py.
Task: apply spec.bundlingEdit (widen the site_extractor asset root + scoped cp
staging derived_fields.py flat, mirroring the Phase 2/3 email-processor form,
keeping the __pycache__/tests/package excludes) and, per
spec.characterizationTests / spec.parityRules, add or update the
site_extractor pin in test_bundle_consistency.py so a commented-out or narrowed
derived_fields cp FAILS the test (keep existing tests green, don't lose teeth).
Touch nothing else; every other function property and every other asset stays
byte-identical.
Run before returning: ruff check cdk tests/test_bundle_consistency.py && cd cdk
&& npx cdk synth po-ingest -q -o /tmp/phase6-synth (artifact-id selector, NOT
stack_name). Confirm the staged site_extractor asset CONTAINS derived_fields.py
flat alongside handler.py and NO tests/__pycache__/package; note the asset hash.
If the handler edits have not landed yet the synth still works (asset is just
files) — but if the new tests/ dir would be staged, prove the exclude drops it.
${specBlock}`,
    { label: 'impl:cdk-bundle-test', model: 'sonnet', phase: 'Implement', schema: IMPL }),

  () => agent(`${PREAMBLE}
YOU OWN: scripts/backfill_sites.py and README.md ONLY.
Task A: apply spec.backfillDecision — either repoint scripts/backfill_sites.py
at derive_site_code (importing from derived_fields, not the handler's deleted
extractor) OR git rm it. Do exactly what the spec pinned.
Task B: README — update the site_code / site_extractor section to reflect the
single reconciled definition (derived_fields canonical shape is now
authoritative for site_extractor too; KLAL-class all-letter codes self-register;
derived_fields.py ships into the site_extractor asset via the same bundling cp).
Fix any stale "three inconsistent site_code definitions" wording (finding X2).
Match existing README style.
Run before returning: ruff check on any script you kept, and pytest -q --no-cov
at repo root (must be green against the OTHER agents' edits — they land in
parallel; poll by re-running up to ~10 min before reporting a blocker).
${specBlock}`,
    { label: 'impl:backfill-readme', model: 'opus', phase: 'Implement', schema: IMPL }),
])

const implOk = impl.filter(Boolean)
const implBlockers = implOk.flatMap(r => r.blockers || [])
log(`Implement complete: ${implOk.length}/3 agents, blockers: ${implBlockers.length}`)

// ---------------------------------------------------- verify + fix loop

const EXPECTED_SCOPE = [
  'lambdas/po/site_extractor/',
  'cdk/po_stack.py',
  'scripts/backfill_sites.py',
  'tests/test_bundle_consistency.py',
  'README.md',
]

const mechanicalPrompt = `${PREAMBLE}
Independent re-verification — trust nothing self-reported. Run ALL gates,
quoting failures verbatim:
1. pytest -q --no-cov (repo root, all roots green — including the NEW
   site_extractor suite). Confirm the new suite actually exists and is
   collected (it did not exist before this phase).
2. ruff check . && ruff format --check .
3. cd cdk && npx cdk synth po-ingest -q (artifact-id selector, NOT stack_name).
4. ASSET CONTENTS: locate the staged site_extractor asset under the synth -o
   dir and prove it CONTAINS derived_fields.py flat alongside handler.py and
   the site_extractor's own siblings, and contains NO tests/, NO __pycache__,
   NO package/ (constraint 2). List the asset's top-level files.
5. CHARACTERIZATION: confirm the committed site_extractor tests now assert the
   FIXED behavior — KLAL ACCEPTED, DLI6X and SNY55 REJECTED — and that a
   current-behavior-of-old-code pin was not left red (constraint 5).
6. UNTOUCHABLE: git diff ${BASE}...HEAD --
   lambdas/po/email_processor/derived_fields.py MUST output NOTHING
   (constraint 6). Also confirm parse_address + write_pending_review + the
   pending-review else-branch in site_extractor/handler.py are byte-unchanged
   vs ${BASE} (git diff the file and check only extract_site_code /
   SITE_CODE_PATTERN / imports changed).
7. DELETION: SITE_CODE_PATTERN is gone from site_extractor/handler.py (grep
   returns nothing) and no path still uses the deleted digit-requiring regex or
   the line-62 [A-Z]{4,5} alternation.
8. git status --porcelain scope check: every modified/added/deleted path under
   ${EXPECTED_SCOPE.join(', ')} (untracked .coverage/.claude/package/
   tolerated). If backfill_sites.py was deleted, the deletion is staged/tracked.
passed=true only if all green. YOU MAY NOT edit files.`

const lenses = [
  { key: 'shape-parity', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — shape-parity lens. Prove extract_site_code now matches
derive_site_code's canonical shape + skip-list. (1) Byte-read the fixed
extract_site_code and derived_fields' _STRICT_CODE_RE / _is_valid_code / _SKIP;
construct the token set {KLAL, DLI6, DLI6X, SNY55, RME (skip-listed), B187
(<2 alpha)} and trace each through the fixed direct-field validation AND the
derive_site_code fallback — KLAL MUST be accepted, DLI6X and SNY55 MUST be
rejected, skip-listed and sub-2-alpha tokens rejected. (2) Prove NO path still
uses the deleted digit-requiring SITE_CODE_PATTERN or the [A-Z]{4,5} dead arm
(grep the whole handler). (3) Attack the fallback wiring: does derive_site_code
receive the dict shape it expects, and does it NEVER raise (a stream record with
missing/empty fields must not throw)? confirmed=true only with a concrete
token-trace or file:line proof.` },
  { key: 'bundling', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — bundling lens. Prove derived_fields.py actually ships in
the site_extractor zip and the excludes stay intact. (1) cd cdk && npx cdk
synth po-ingest -q -o /tmp/phase6-bundle-a, then again into
/tmp/phase6-bundle-b; locate the staged site_extractor asset in each and prove
derived_fields.py is present flat next to handler.py, that the runtime import
'from derived_fields import derive_site_code' resolves with that dir alone on
sys.path (python3 -c with env stubs), and that the two synths produce IDENTICAL
asset hashes (determinism). (2) Drop a throwaway __pycache__/junk.pyc and a
tests/scratch_test.py under the widened root region (delete them afterwards),
re-synth, and confirm the excludes keep the asset hash unchanged and neither
file appears. (3) Confirm NO other asset (email processors, web_ui) changed
hash as a side effect of the root widening. confirmed=true only with evidence.` },
  { key: 'regression', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — regression lens. The fix must be surgical. (1) git diff
${BASE}...HEAD -- lambdas/po/site_extractor/handler.py — prove parse_address,
write_pending_review, upsert_site, load_address_cache, lookup_by_address,
deserialize_image and the handler() event loop / pending-review else-branch are
BYTE-UNCHANGED; the ONLY deltas are inside extract_site_code + its imports +
the deleted SITE_CODE_PATTERN. (2) Prove derived_fields.py is byte-unchanged
vs ${BASE} (constraint 6) — the shadow-bake authority model did not shift.
(3) Confirm the ai_fallback-sourced record["site_code"] is still VALIDATED
(shape + skip-list), not blindly trusted and not regex-bypassed (constraint 3).
(4) If backfill_sites.py was kept, prove it no longer imports the deleted
extractor; if deleted, prove nothing else imports it. confirmed=true only with
file:line or diff evidence.` },
]

let round = 0
let checks = null
let confirmed = []
while (round < 3) {
  phase('Verify')
  const results = await parallel([
    () => agent(mechanicalPrompt, { label: `verify:mechanical-r${round}`, model: 'sonnet', phase: 'Verify', schema: CHECKS }),
    ...lenses.map(l => () =>
      agent(l.prompt, { label: `verify:${l.key}-r${round}`, phase: 'Verify', schema: FINDINGS })),
  ])
  checks = results[0]
  confirmed = results.slice(1).filter(Boolean)
    .flatMap(r => r.findings || [])
    .filter(f => f.confirmed && f.severity !== 'low')
  const green = checks && checks.passed
  log(`Verify round ${round}: mechanical ${checks && checks.passed ? 'GREEN' : 'RED'}, confirmed findings: ${confirmed.length}`)
  if (green && confirmed.length === 0) break

  round += 1
  if (round >= 3) break
  phase('Fix')
  await agent(`${PREAMBLE}
You are the fix agent — you may edit files under: ${EXPECTED_SCOPE.join(', ')}.
Fix EVERY item below minimally; the binding spec and 7 pinned constraints still
hold (a finding that conflicts with a constraint is reported, not "fixed" — the
constraint wins, esp. constraint 6's derived_fields untouchability and
constraint 3's validated-not-blind-trust rule). Re-run the specific failing
gate/test per fix.
MECHANICAL:\n${checks ? checks.details : '(agent died — rerun all gates)'}
CONFIRMED FINDINGS:\n${JSON.stringify(confirmed, null, 2)}
${specBlock}`,
    { label: `fix:round-${round}`, model: 'opus', phase: 'Fix', schema: IMPL })
}

const verifyClean = checks && checks.passed && confirmed.length === 0
if (!verifyClean) {
  return {
    status: 'NEEDS ATTENTION — verify not clean after 3 rounds; branch left uncommitted',
    branch: BRANCH,
    mechanical: checks,
    unresolvedFindings: confirmed,
    implBlockers,
    reconBlockers,
    spec,
  }
}

// ----------------------------------------------------------------- package

phase('Package')
const commit = await agent(`${PREAMBLE.replace('do NOT commit, ', '')}
YOU are the commit agent:
1. Read ~/Documents/repositories/seahaven/engineering-handbook/commit-messages.md
   and follow it exactly.
2. git add only paths under: ${EXPECTED_SCOPE.join(', ')} and
   .claude/workflows/phase-6-site-extractor.js. NOT .coverage, NOT package/.
   Verify the staged set with git status — if backfill_sites.py was deleted,
   its deletion MUST be staged too.
3. ONE commit; write the message to /tmp/phase6-commit-msg.txt and use
   git commit -F /tmp/phase6-commit-msg.txt (backticks in -m get eaten by zsh).
   Suggested subject:
   "feat: reconcile site_extractor site_code with derived_fields canonical shape (refactor phase 6)"
   Body: the digit-pattern defect (KLAL never self-registered, DLI6X/SNY55
   false-accepted), the canonical fullmatch + skip-list fix + derive_site_code
   fallback, the derived_fields.py bundling cp into the site_extractor asset
   (Phase 2/3 mechanism), the characterization-first-then-flip test approach,
   the backfill repoint/delete decision, and that derived_fields.py is
   byte-unchanged. NO AI attribution / Co-Authored-By lines.
4. Do NOT push. Return commit sha + shortstat in summary.`,
  { label: 'package:commit', model: 'sonnet', phase: 'Package', schema: IMPL })

return {
  status: 'BUILT — committed locally, NOT pushed',
  branch: BRANCH,
  base: BASE,
  commit: commit ? commit.summary : 'commit agent died — commit manually',
  spec: { handlerFix: spec.handlerFix, bundlingEdit: spec.bundlingEdit, backfillDecision: spec.backfillDecision, notes: spec.notes },
  implementation: implOk.map(r => r.summary),
  filesChanged: implOk.flatMap(r => r.filesChanged),
  verifyRounds: round + 1,
  blockers: implBlockers.concat(reconBlockers),
  outstandingGates: [
    '/sh-security-review RECOMMENDED before push — the validated record["site_code"] field can be ai_fallback-sourced (attacker-influenced via a DKIM-passing injected body), so the new shape/skip validation is untrusted-input handling; run it on the committed diff and flag it.',
    'cross-family cross_review.py NOT required (no IAM/policy change, no Lambda handler signature/event/return-contract change — internal validation logic + a bundling cp only).',
    'push + PR + gh pr checks green.',
    'deploy-then-merge: deploy from branch, smoke green, and confirm the LIVE success signal = the pending-site-review write rate DROPS (KLAL-class all-letter codes now self-register instead of falling to pending review), plus one real DynamoDB-stream event processed clean, THEN merge.',
    'GOTCHA: npx cdk synth po-ingest uses the ARTIFACT-ID selector (po-ingest), not the stack_name — using the wrong selector silently synths nothing or the wrong stack.',
  ],
}