export const meta = { name: 'phase-7-ops-recovery', description: 'Phase 7 of the procurement-ingest refactor (docs/refactor-evaluation.md): ops/recovery + dependency hygiene. Generalize scripts/reprocess.py (--pipeline po|wo, --key/--prefix/--since; TARGETED replay primary, full-prefix DEMOTED behind --all with documented caveats) + a synthetic-event-shape contract test pinning the raw-key/no-URL-decode S3 event. New docs/runbook-dlq-recovery.md for the no-redrive async-destination DLQ (receive->key->re-invoke->verify->purge; 14d DLQ / 90d S3 windows; sender-auth + ai_fallback_rejected drops never reach the DLQ), linked from README alarms. Dependency hygiene: drop vendored boto3 from both email-processor requirements (handbook empty-with-comment form; runtime copy suffices per lambda-template.md), simplify bundling to cp-only where the vendored dep is gone, exact-pin moto==, pin-then-add dependabot entries for /tests + po/web_ui + po/site_extractor, reduce wo/web_ui dead manifest to empty-with-comment. Optional local-only rm of the untracked 44 MB package/ dir. Gates on Phase 2 merged (../lambdas asset root). Ops tooling + dep hygiene, no auth/untrusted-input/IAM change. Committed locally, never pushed.', phases: [ { title: 'Setup', detail: 'verify Phase 2 on base (Code.from_asset("../lambdas")), branch feature/phase-7-ops-recovery', model: 'haiku' }, { title: 'Recon', detail: '4 mappers: reprocess.py + event shape, dependency/manifest/dependabot state, post-Phase-2 bundling + package/ dir, DLQ/alarm/S3-lifecycle names (read-only AWS)' }, { title: 'Spec', detail: 'serial fable spec: reprocess CLI + caveats, runbook contents, requirements/moto/dependabot edits, cp-only bundling form, package/ cleanup, contract test, verify rules' }, { title: 'Implement', detail: 'opus: scripts/ + reprocess contract test; sonnet: requirements + dependabot + cdk bundling; opus: runbook + README — disjoint files', model: 'opus' }, { title: 'Verify', detail: 'mechanical gates + 3 fable lenses (bundling-integrity, reprocess-safety, dependency)', model: 'sonnet' }, { title: 'Fix', detail: 'opus fixer, full re-verify, max 3 rounds', model: 'opus' }, { title: 'Package', detail: 'single commit via -F (no push); optional local rm of package/ reported, not committed', model: 'sonnet' }, ], } // ---------------------------------------------------------------- constants const REPO = '/Users/adammoussa/Documents/repositories/seahaven/procurement-ingest' const BRANCH = 'feature/phase-7-ops-recovery' let _args = args if (typeof _args === 'string') { try { _args = JSON.parse(_args) } catch (e) { _args = null } } const BASE = (_args && _args.base) || 'main' const CONSTRAINTS = ` PINNED BEHAVIORAL CONSTRAINTS (docs/refactor-evaluation.md Phase 7 — violating any is a build failure): 1. GENERALIZE scripts/reprocess.py: add --pipeline po|wo (selects the function name + bucket per pipeline — po-email-processor / po-ingest-emails-, workorder-email-processor / its bucket), plus --key (single object), --prefix, and --since (time filter on LastModified). TARGETED replay (--key / --prefix / --since) is the PRIMARY, default mode. FULL-PREFIX replay is DEMOTED behind an explicit --all flag. The --all caveats MUST be documented BOTH in --help text and in code comments: Event-type (async) invocation is CONCURRENT so sorting does NOT serialize; switch to RequestResponse if order matters; metrics get double-counted on replay; Bedrock is re-billed; out-of-order replay can REGRESS already-merged fields. Keep the existing dry-run-by-default / --execute safety (targeted replay stays dry-run unless --execute). 2. SYNTHETIC-EVENT-SHAPE CONTRACT TEST (§4.7): pin the exact S3 event shape reprocess emits — Records[0].s3.bucket.name + Records[0].s3.object.key — and assert the key is the RAW object key with NO URL-decoding (a real S3 notification URL-encodes the key; reprocess builds from the raw list_objects_v2 key, and the handler is what decodes — replaying a pre-decoded key would double-decode). Test lives under scripts' tests or tests/ and runs in the standard pytest collection. 3. NEW docs/runbook-dlq-recovery.md: the async on-failure destination DLQ has NO console redrive-to-source. Documented procedure: receive-message -> extract the S3 key from the event body -> targeted re-invoke (via the generalized reprocess.py --key) -> verify the write -> purge the message. State the recovery WINDOWS explicitly: 14-day DLQ breadcrumb retention, 90-day raw-email S3 retention (the inbound/ lifecycle rule OVERRIDES the table RETAIN policy — S3 is the real replay floor). Document that sender-auth rejections AND ai_fallback_rejected drops INTENTIONALLY never reach the DLQ (they are fail-closed SKIPS, not errors — no retry, no DLQ message). Link the runbook from the README alarms section. 4. DEPENDENCY HYGIENE — DROP vendored boto3 from BOTH email-processor requirements.txt: replace 'boto3>=1.43.47' with the handbook empty-with-comment form (a comment explaining the Lambda runtime provides boto3 per lambda-template.md — only-boto3 functions correctly use the runtime copy; no pinned third-party dep remains). Same empty-with-comment reduction for wo/web_ui/requirements.txt (a dead manifest today). 5. SIMPLIFY BUNDLING TO CP-ONLY where the vendored dep is gone: once the email-processor requirements are empty-with-comment, 'pip install -r ... ' installs nothing, so the pip step is removable and the command becomes cp-only. This is ONLY safe BECAUSE there are no third-party binary deps left. Do NOT regress: the Phase 2 exclude lists (['**/__pycache__/**','**/tests/**','**/package/**']) STAY; the widened '../lambdas' asset root STAYS; if a Phase 3 'cp shared/*.py' line is present on the base it STAYS. The '--platform manylinux2014_aarch64 --only-binary=:all:' pin is only meaningful while a pip install runs — its accidental removal caused the PR #34 outage, so removing the WHOLE pip step (deliberately, because nothing is installed) is the only acceptable way it disappears; you may NOT keep a pip install while dropping the pin. If recon finds ANY third-party dep still required by an email processor, cp-only is a blocker — keep the pinned pip step. 6. EXACT-PIN moto: change tests/requirements.txt 'moto>=5.0.0' to 'moto==' (resolve the actually-installed/current 5.x version; floor pins make the dependabot entry a no-op). 7. DEPENDABOT: add pip entries for '/tests', '/lambdas/po/web_ui', and '/lambdas/po/site_extractor' — but PIN THOSE MANIFESTS FIRST. po/web_ui and po/site_extractor have NO requirements.txt today, so a dependabot entry pointed at them is a no-op until a PINNED manifest exists; create each manifest (exact-pinned boto3, or empty-with-comment ONLY if the dependabot entry would then be pointless — a dependabot entry needs at least one pinned dep to act on). Match the existing dependabot.yml entry shape (weekly, minor-and-patch group). tests/ gets its entry once moto is exact-pinned (constraint 6). 8. DELETE the untracked 44 MB lambdas/po/email_processor/package/ dir as an OPTIONAL local-cleanup step: it is UNTRACKED / local-only, so this is a filesystem 'rm -rf', NOT a 'git rm' (there is nothing tracked to commit). With Phase 2's '**/package/**' exclude already deployed, the deletion is asset-hash-NEUTRAL. The Package agent PERFORMS and REPORTS it; it is NOT part of the commit. 9. UNTOUCHABLE: no handler / lambda runtime code change under lambdas/*/ email_processor/*.py (this is dep + ops + bundling only). No alarm, IAM, table, env, runtime, memory, timeout, or logical-ID delta on either stack — the ONLY cdk delta permitted is the bundling command string on the two email processors (+ any resulting asset-hash / CDK metadata change). The dropped-vendored-boto3 functions MUST still 'import boto3' at runtime from the Lambda runtime copy. 10. AWS access is READ-ONLY (get-function, get-function-event-invoke-config, SQS get-queue-attributes, S3 get-bucket-lifecycle-configuration, describe-alarms). NEVER cdk deploy, never invoke, never mutate, never purge a real queue. ` const PREAMBLE = ` You are one of several agents building refactor Phase 7 in the git repo at ${REPO} on branch ${BRANCH} (already checked out — do NOT switch branches, do NOT create branches, do NOT commit, NEVER push, do NOT run cdk deploy or touch AWS resources beyond read-only calls). Authoritative spec: docs/refactor-evaluation.md, section "Phase 7 — Ops/recovery + dependency hygiene". Work ONLY in the files you are told you own; other agents are concurrently editing other files in this same working tree. ${CONSTRAINTS} Your final message is consumed by an orchestrator script, not a human — return only the structured data requested. ` // ------------------------------------------------------------------ schemas const RECON = { type: 'object', required: ['summary', 'facts'], properties: { summary: { type: 'string' }, facts: { type: 'array', items: { type: 'string' } }, blockers: { type: 'array', items: { type: 'string' } }, }, } const SPEC = { type: 'object', required: ['reprocessCli', 'contractTest', 'runbook', 'dependencyEdits', 'bundlingEdits', 'packageCleanup', 'verifyRules', 'notes'], properties: { reprocessCli: { type: 'string', description: 'the complete generalized scripts/reprocess.py design: argparse surface (--pipeline po|wo, --key/--prefix/--since, --all, --execute), per-pipeline function-name/bucket resolution, which modes are dry-run-by-default, and the EXACT --help + code-comment caveat text for --all (concurrency/no-serialize, RequestResponse-if-order-matters, double-count, Bedrock re-bill, out-of-order regression)' }, contractTest: { type: 'string', description: 'the synthetic-event-shape contract test: file path, what it asserts about Records[0].s3.bucket.name / object.key, and the exact raw-key / no-URL-decoding assertion (encoded vs raw key example)' }, runbook: { type: 'string', description: 'the docs/runbook-dlq-recovery.md outline: the no-redrive procedure steps (receive->key->targeted re-invoke via reprocess --key->verify->purge), the 14d-DLQ / 90d-S3 windows with the lifecycle-overrides-RETAIN note, the intentional sender-auth + ai_fallback_rejected non-DLQ statement, and the exact README alarms-section link edit' }, dependencyEdits: { type: 'string', description: 'per requirements.txt: exact new contents (both email-processor empty-with-comment, wo/web_ui empty-with-comment, po/web_ui + po/site_extractor new pinned manifests, tests/ moto==); the resolved moto version; the exact dependabot.yml additions matching the existing entry shape' }, bundlingEdits: { type: 'string', description: 'both email-processor bundling command strings BEFORE and AFTER, verbatim: the cp-only result, proof that the pip step is safely removable (no third-party dep remains), and confirmation the exclude lists + widened root + any Phase-3 shared cp line are preserved' }, packageCleanup: { type: 'string', description: 'the package/ dir cleanup: the exact rm command, why it is a filesystem rm not git rm (untracked), and why it is asset-hash-neutral given the Phase 2 exclude — this is a REPORTED local step, not a commit' }, verifyRules: { type: 'string', description: 'how Verify judges success: which pytest tests must pass (incl. the new contract test), ruff scope now including scripts/, both cdk synth expectations, the dependabot yaml-parse check, requirements resolve/import check, and the runtime import-boto3 proof for the cp-only assets' }, notes: { type: 'string' }, }, } const IMPL = { type: 'object', required: ['filesChanged', 'summary', 'checksRun'], properties: { filesChanged: { type: 'array', items: { type: 'string' } }, summary: { type: 'string' }, checksRun: { type: 'string' }, blockers: { type: 'array', items: { type: 'string' } }, }, } const CHECKS = { type: 'object', required: ['passed', 'details'], properties: { passed: { type: 'boolean' }, details: { type: 'string' }, scopeViolations: { type: 'array', items: { type: 'string' } }, }, } const FINDINGS = { type: 'object', required: ['findings'], properties: { findings: { type: 'array', items: { type: 'object', required: ['title', 'severity', 'confirmed', 'evidence', 'fix'], properties: { title: { type: 'string' }, severity: { enum: ['critical', 'high', 'medium', 'low'] }, confirmed: { type: 'boolean' }, evidence: { type: 'string' }, fix: { type: 'string' }, }, }, }, }, } // ------------------------------------------------------------------- setup phase('Setup') const setup = await agent(` In ${REPO}: 1. SEQUENCING GATE — Phase 2 must be on ${BASE}. Phase 7 "can run parallel to 4-6" (largely independent), BUT the cp-only bundling simplification and the package/ cleanup touch the same cdk bundling that Phases 2/3 established, so this phase must NOT race Phase 2. git fetch origin, then pick the base ref: origin/${BASE} if that remote ref exists, otherwise the local branch ${BASE} (a stacked local-only base is expected and fine). Verify on the base ref that BOTH cdk/po_stack.py and cdk/wo_stack.py contain Code.from_asset("../lambdas") for the email processors (git show :cdk/po_stack.py | grep -n '\\.\\./lambdas', same for wo_stack.py). If either is missing, STOP with a blocker naming Phase 2 as unmet and do nothing else. NOTE (report, do not block): Phase 7 is otherwise independent of Phases 4-6. If Phase 4 (cdk/common.py dedup) lands on ${BASE} FIRST, the stack-touching bundling edits here must be rebased onto the moved bundling code — flag that in your facts so the implement agents know whether the bundling lives inline or in a common helper. 2. Verify clean working tree (untracked .coverage / .claude/ / the local 44 MB lambdas/po/email_processor/package/ dir are fine; any OTHER dirt = blocker, never stash or discard). 3. git checkout ${BASE}; then git pull --ff-only ONLY if the branch has an upstream (a local-only base skips the pull — not a blocker); then git checkout -b ${BRANCH} 4. gh pr list --state open --json number,title,headRefName (overlap check — especially any in-flight Phase 2/3/4 stack PR). Return facts: HEAD sha, Phase 2 gate evidence, whether Phase 4 has landed (bundling inline vs helper), open PRs, blockers. `, { label: 'setup:branch', model: 'haiku', schema: RECON }) if (!setup || (setup.blockers && setup.blockers.length)) { return { aborted: 'setup blockers', blockers: setup ? setup.blockers : ['setup agent died'], facts: setup ? setup.facts : [] } } log(`Branch ${BRANCH} ready off ${BASE}. ${setup.summary}`) // ------------------------------------------------------------------- recon phase('Recon') const recon = await parallel([ () => agent(`${PREAMBLE} Read-only recon of scripts/reprocess.py and the event shape it must pin: 1. Quote scripts/reprocess.py verbatim with file:line — the argparse surface today (--execute only), FUNCTION_NAME, get_bucket_name (po-only, po-ingest-emails-), list_inbound_keys (paginator, Prefix='inbound/'), build_s3_event (the exact Records/s3/bucket.name/object.key shape), the Event (async) InvocationType. 2. Find the WO analogues so --pipeline wo can resolve: the workorder email-processor function name and its email bucket naming (grep the cdk stacks / function_name / bucket_name). State them exactly. 3. Determine how the email-processor handler reads the key from an S3 event: does it URL-decode Records[].s3.object.key (urllib.parse.unquote_plus)? Quote the handler line. This decides the contract test's no-double-decode assertion (reprocess must emit the RAW key so the handler decode path is exercised exactly once — a pre-decoded key double-decodes '+'/'%20'). 4. Is there ANY existing test that touches scripts/reprocess.py? (grep tests/ and scripts/). Where should the new synthetic-event contract test live so it is collected by the standard pytest run (repo-root pytest collects tests/ and both email_processor tests/ roots)? Note whether scripts/ is on sys.path for import. 15-25 precise facts.`, { label: 'recon:reprocess', model: 'sonnet', phase: 'Recon', schema: RECON }), () => agent(`${PREAMBLE} Read-only recon of the dependency / manifest / dependabot state: 1. Quote every requirements.txt under lambdas/ and tests/ verbatim with path: lambdas/po/email_processor (boto3>=1.43.47), lambdas/wo/email_processor (boto3>=1.43.47), lambdas/wo/web_ui (boto3>=1.43.47 — dead manifest), tests/ (moto>=5.0.0), cdk/. Confirm po/web_ui and po/site_extractor have NO requirements.txt (they must be CREATED + pinned before a dependabot entry is not a no-op). 2. Resolve the moto version to exact-pin: what 5.x is actually installed / currently resolves (pip show moto or the lockfile / venv)? State the exact version string to pin. 3. Quote .github/dependabot.yml verbatim: the existing pip entries (/cdk, /lambdas/po/email_processor, /lambdas/wo/email_processor, /lambdas/wo/web_ui) and the github-actions entry — the exact shape (weekly interval, minor-and-patch group) the three NEW entries must match. 4. Confirm via lambda-template.md (handbook) the exact empty-with-comment form for only-boto3 functions (the Lambda runtime provides boto3) — quote the canonical comment wording so the edits match the handbook. 5. Does anything under lambdas/*/email_processor actually import a THIRD-PARTY package other than boto3 (grep imports; anthropic/requests/etc.)? If yes, cp-only is unsafe — that is a blocker for constraint 5. 15-25 facts.`, { label: 'recon:dependencies', model: 'sonnet', phase: 'Recon', schema: RECON }), () => agent(`${PREAMBLE} Read-only recon of the post-Phase-2 CDK bundling + the package/ dir: 1. Quote BOTH email-processor Code.from_asset blocks verbatim with current file:line (cdk/po_stack.py ~241, cdk/wo_stack.py ~240): asset root, the FULL exclude list, the entire bundling command list — the pip install '--platform manylinux2014_aarch64 --only-binary=:all: -r /email_processor/requirements.txt -t /asset-output' step, the '&&', the 'cp /email_processor/*.py /asset-output/' step, and whether a Phase-3 'cp shared/*.py /asset-output/' line is already present on this branch's base. Note the explanatory NOTE comment in po_stack. 2. State the exact cp-only rewrite for each (drop the pip step entirely, keep every cp line + the widened root + the exclude list). Confirm the pip step is the ONLY thing removed. 3. The three plain from_asset calls (po/web_ui, po/site_extractor, wo/web_ui) — quote them; they are NOT bundling-simplified here (no pip step), just confirm they are untouched by this phase. 4. lambdas/po/email_processor/package/ — du -sh it, confirm it is UNTRACKED (git status / git ls-files must not list it), and confirm the Phase 2 '**/package/**' exclude already keeps it out of the asset hash (so its removal is asset-hash-neutral). 5. If Phase 4 has landed (bundling moved into a cdk/common helper) note the new location. 12-20 facts.`, { label: 'recon:cdk-bundling', model: 'haiku', phase: 'Recon', schema: RECON }), () => agent(`${PREAMBLE} Read-only AWS + docs recon for the runbook (region us-east-1, READ-ONLY): 1. Both email-processor DLQs: exact SQS queue names/ARNs and the MessageRetentionPeriod (confirm the ~14-day breadcrumb window). Find them via the stacks (dead_letter_queue / make_processor_dlq) and aws sqs get-queue-attributes. Also the async event-invoke config / on-failure destination if present (get-function-event-invoke-config) — confirm it is an async on-failure destination DLQ with NO console redrive-to-source. 2. Both raw-email S3 buckets: exact names and the inbound/ prefix lifecycle rule (get-bucket-lifecycle-configuration) — confirm the ~90-day expiry that overrides the table RETAIN policy (S3 is the replay floor). 3. The relevant CloudWatch alarm names the runbook references: the sender-auth-rejected alarm(s) and the ai_fallback_rejected alarm(s) per pipeline (describe-alarms / grep the stacks) — so the runbook states that those two drop classes are fail-closed SKIPS that never reach the DLQ. 4. Quote the README alarms section (its heading + surrounding lines) so the Implement agent can add the runbook link in the right place, matching style. Return names/ARNs/windows as facts; flag any name you could not resolve so the runbook can be validated against real infra later.`, { label: 'recon:dlq-alarms', model: 'sonnet', phase: 'Recon', schema: RECON }), ]) const reconOk = recon.filter(Boolean) const pack = reconOk.map(r => `## ${r.summary}\n${r.facts.join('\n')}`).join('\n\n') const reconBlockers = reconOk.flatMap(r => r.blockers || []) .filter(b => b && !/^\s*(none|n\/a)\b/i.test(b)) log(`Recon complete: ${reconOk.length}/4 mappers, ${reconBlockers.length} blockers`) if (reconBlockers.length) { return { aborted: 'recon blockers (likely a surviving third-party dep blocking cp-only, or an unresolvable pipeline name)', blockers: reconBlockers, reconPack: pack } } // -------------------------------------------------------------------- spec phase('Spec') const spec = await agent(`${PREAMBLE} You are the SPEC agent — the single authority that pins every contested decision BEFORE parallel implementation (parallel leaves cannot see each other's choices). Using the recon pack below plus your own reads of the actual files, produce the binding implementation spec: - reprocessCli: the complete generalized scripts/reprocess.py — argparse surface (--pipeline po|wo required; --key / --prefix / --since targeted modes; --all for full-prefix; --execute preserving dry-run-by-default), the per-pipeline function-name + bucket resolution table, and the EXACT --help + in-code caveat text for --all (all five caveats from constraint 1 verbatim). TARGETED is the default path; --all is the ONLY way to sweep the whole prefix. - contractTest: the synthetic-event-shape test — file path (collected by the standard pytest run), the assertions on Records[0].s3.bucket.name and object.key, and the raw-key / no-URL-decoding assertion with a concrete encoded-vs-raw example (e.g. a key with a space or '+'). - runbook: the full docs/runbook-dlq-recovery.md outline with the real DLQ/queue/alarm/bucket names from recon, the no-redrive procedure steps (receive -> extract S3 key from event body -> targeted re-invoke via reprocess.py --key -> verify -> purge), the 14d-DLQ / 90d-S3 windows + lifecycle-overrides-RETAIN note, the intentional sender-auth + ai_fallback_rejected non-DLQ statement, and the exact README alarms-section link edit. - dependencyEdits: exact new contents of every manifest — both email-processor requirements.txt (handbook empty-with-comment), wo/web_ui (empty-with-comment), po/web_ui + po/site_extractor (NEW; decide pinned dep vs whether the dependabot entry is worthwhile — a dependabot entry needs at least one pinned dep to act on, so pin boto3 exactly if you want the entry live), tests/ (moto== from recon) — plus the exact dependabot.yml additions (three entries) matching the existing shape. - bundlingEdits: both email-processor bundling command strings BEFORE and AFTER, verbatim — the cp-only result, an explicit proof line that no third-party dep remains (so removing the whole pip step is safe, NOT a regression of the manylinux pin), and confirmation the exclude lists + widened '../lambdas' root + any Phase-3 'cp shared/*.py' line survive untouched. If Phase 4 moved bundling into a helper, target that location. - packageCleanup: the exact rm -rf command for lambdas/po/email_processor/package/, the note that it is a filesystem rm (UNTRACKED — never git rm) performed and REPORTED by the Package agent, not committed, and why it is asset-hash-neutral (Phase 2 exclude). - verifyRules: exactly how Verify judges success — the pytest set incl. the new contract test, ruff now INCLUDING scripts/, both cdk synth, dependabot.yml yaml-parse, requirements resolve/import, and the runtime 'import boto3' proof for a cp-only staged asset. Recon pack:\n${pack}`, { label: 'spec:pin-ops-recovery', phase: 'Spec', schema: SPEC }) if (!spec) return { aborted: 'spec agent died — rerun workflow', reconBlockers } const specBlock = `BINDING SPEC (from the spec agent — implement EXACTLY this):\n${JSON.stringify(spec, null, 2)}` log('Spec pinned: reprocess CLI + caveats, contract test, runbook, dependency edits, cp-only bundling, package cleanup') // --------------------------------------------------------------- implement phase('Implement') const impl = await parallel([ () => agent(`${PREAMBLE} YOU OWN: scripts/reprocess.py and the new synthetic-event contract test file ONLY (per spec.contractTest's path — do not touch any other test). Do not touch cdk/, requirements, dependabot, docs, or README. Task: rewrite scripts/reprocess.py per spec.reprocessCli — add --pipeline po|wo, --key/--prefix/--since (TARGETED, the default), --all (DEMOTED full-prefix, with all five caveats in BOTH --help and code comments), preserving dry-run-by-default / --execute. Keep build_s3_event emitting the RAW key (no URL-decoding). Then add the contract test per spec.contractTest pinning the event shape + the raw-key / no-double-decode assertion. Run before returning: ruff check scripts && ruff format scripts --check, python3 scripts/reprocess.py --help (must render the caveats), and pytest -q --no-cov -k (tests land in parallel — if collection of other roots is red from another agent's in-flight edit, run your test file directly; poll up to ~10 min before reporting a blocker). ${specBlock}`, { label: 'impl:reprocess', model: 'opus', phase: 'Implement', schema: IMPL }), () => agent(`${PREAMBLE} YOU OWN: every requirements.txt (lambdas/po/email_processor, lambdas/wo/email_processor, lambdas/wo/web_ui, tests/, and the two NEW manifests lambdas/po/web_ui/requirements.txt + lambdas/po/site_extractor/requirements.txt), .github/dependabot.yml, and the bundling regions of cdk/po_stack.py + cdk/wo_stack.py ONLY (only the email-processor from_asset bundling command — alarms, IAM, tables, env, memory, timeout, the three plain from_asset calls are all off-limits). Task: apply spec.dependencyEdits (drop vendored boto3 -> empty-with-comment in both email-processor + wo/web_ui; create the two pinned po manifests; exact-pin moto==) + the three dependabot entries + spec.bundlingEdits (cp-only: remove ONLY the pip step from both email-processor commands, preserving the exclude lists, widened root, and any Phase-3 shared cp line). Do NOT delete package/ (that is the Package agent's reported step). Run before returning: ruff check cdk, python3 -c 'import yaml,pathlib; yaml.safe_load(pathlib.Path(".github/dependabot.yml").read_text())' (parses), cd cdk && npx cdk synth po-ingest -q -o /tmp/phase7-synth && npx cdk synth workorder-ingest -q -o /tmp/phase7-synth (artifact-id selectors). Docker bundling runs — confirm each staged email-processor asset still contains handler.py + its siblings (and shared/*.py if Phase 3 landed) and that 'import boto3' resolves at runtime from the Lambda runtime copy (the staged asset need not vendor it). Note asset hashes. If the other agents' edits have not landed the synth is unaffected (bundling is yours) — but poll ~10 min if Docker is slow before a blocker. ${specBlock}`, { label: 'impl:deps-bundling', model: 'sonnet', phase: 'Implement', schema: IMPL }), () => agent(`${PREAMBLE} YOU OWN: docs/runbook-dlq-recovery.md (NEW) and README.md ONLY. Task A: write docs/runbook-dlq-recovery.md per spec.runbook — the no-redrive async-destination DLQ procedure (receive-message -> extract the S3 key from the event body -> targeted re-invoke via 'python scripts/reprocess.py --pipeline --key --execute' -> verify the write -> purge the message), the real queue/bucket/alarm names from the spec, the recovery windows (14-day DLQ breadcrumb, 90-day raw-email S3 with the lifecycle-overrides-RETAIN note), and the explicit statement that sender-auth rejections AND ai_fallback_rejected drops are fail-closed SKIPS that never reach the DLQ (no retry, no DLQ message). Task B: README — link the new runbook from the alarms section per spec.runbook's exact edit, and note reprocess.py is now pipeline-general (--pipeline / --key / --prefix / --since targeted; --all demoted). Match existing README style. Run before returning: confirm the runbook links resolve and the README edit is in the alarms section. (No pytest owned by you; if you want a doc-lint, ruff does not lint .md.) ${specBlock}`, { label: 'impl:runbook-readme', model: 'opus', phase: 'Implement', schema: IMPL }), ]) const implOk = impl.filter(Boolean) const implBlockers = implOk.flatMap(r => r.blockers || []) log(`Implement complete: ${implOk.length}/3 agents, blockers: ${implBlockers.length}`) // ---------------------------------------------------- verify + fix loop const EXPECTED_SCOPE = [ 'scripts/reprocess.py', 'tests/', 'docs/runbook-dlq-recovery.md', 'lambdas/po/email_processor/requirements.txt', 'lambdas/wo/email_processor/requirements.txt', 'lambdas/wo/web_ui/requirements.txt', 'lambdas/po/web_ui/requirements.txt', 'lambdas/po/site_extractor/requirements.txt', '.github/dependabot.yml', 'cdk/po_stack.py', 'cdk/wo_stack.py', 'README.md', ] const mechanicalPrompt = `${PREAMBLE} Independent re-verification — trust nothing self-reported. Run ALL gates, quoting failures verbatim: 1. pytest -q --no-cov (repo root, all roots green) — INCLUDING the new synthetic-event contract test (name it and quote its pass line). 2. ruff check . && ruff format --check . — scripts/ is now IN scope (it was previously omitted); confirm scripts/reprocess.py lints and formats clean. 3. cd cdk && npx cdk synth po-ingest -q && npx cdk synth workorder-ingest -q (Docker bundling runs; both must succeed). 4. cd cdk && npx cdk diff po-ingest ; npx cdk diff workorder-ingest — constraint 9: the ONLY resource delta allowed is the two email processors' Code/S3Key (asset) + CDK metadata. Any alarm/IAM/env/runtime/ memory/timeout/handler-prop/logical-ID delta = FAIL. Paste the diff summaries. 5. DEPENDABOT: python3 -c 'import yaml,pathlib; d=yaml.safe_load(pathlib.Path(".github/dependabot.yml").read_text()); print([u["directory"] for u in d["updates"]])' — the three new dirs (/tests, /lambdas/po/web_ui, /lambdas/po/site_extractor) present, shape matches existing entries. 6. MANIFESTS: every email-processor + wo/web_ui requirements.txt is the empty-with-comment form (no pinned third-party dep); tests/requirements.txt is moto==; po/web_ui + po/site_extractor manifests EXIST and are PINNED (a dependabot entry pointed at a floor/absent manifest is a no-op — FAIL if unpinned). pip install --dry-run (or pip-compile check) resolves each without error. 7. RUNTIME IMPORT: no lambda handler .py under lambdas/*/email_processor changed (git diff ${BASE}...HEAD -- must be empty for those); 'import boto3' still resolves — prove the Lambda runtime copy suffices (the cp-only staged asset need not vendor boto3). 8. RUNBOOK: docs/runbook-dlq-recovery.md exists, states the 14d/90d windows, the no-redrive procedure, and the sender-auth + ai_fallback_rejected non-DLQ note; README alarms section links it. 9. git status --porcelain scope check: every modified/added path under ${EXPECTED_SCOPE.join(', ')} (untracked .coverage/.claude/package/ tolerated; package/ must NOT be staged). passed=true only if all green. YOU MAY NOT edit files.` const lenses = [ { key: 'bundling-integrity', prompt: `${PREAMBLE} ADVERSARIAL REVIEW — bundling-integrity lens. The bundling was simplified to cp-only and vendored boto3 was dropped; try to prove a runtime ImportError was introduced (the PR #34 / PR #105 failure class). (1) cd cdk && synth both stacks; inside each staged email-processor asset dir list the files and run python3 -c "import handler" with that dir alone on sys.path (env-var stubs as needed) — does every runtime-imported module still ship (handler siblings + shared/*.py if Phase 3 landed)? (2) Does 'import boto3' succeed against the Lambda runtime copy — i.e. is boto3 genuinely NOT needed in the zip, or does some code path need a newer boto3 than the runtime ships? (3) Are the Phase 2/3 excludes (['**/__pycache__/**','**/tests/**','**/package/**']) STILL present and is the widened '../lambdas' root intact? (4) Was the manylinux pin removed ONLY as part of removing the whole pip step (because nothing is installed), NOT while keeping a pip install? A cp-only command that still tries to pip-install without the pin, or a surviving third-party dep, is a critical finding. (5) DETERMINISM: synth po-ingest twice into fresh -o dirs — identical asset hashes. confirmed=true only with a concrete ImportError sketch or file:line proof.` }, { key: 'reprocess-safety', prompt: `${PREAMBLE} ADVERSARIAL REVIEW — reprocess-safety lens. Attack the generalized reprocess.py. (1) Is the DEFAULT behavior TARGETED (—key/—prefix/—since), and is it IMPOSSIBLE to sweep the whole inbound/ prefix without the explicit --all flag? Trace the arg parsing — can a missing/empty --prefix silently fall through to a full-prefix replay? That would be a critical finding (accidental mass re-invoke = mass Bedrock re-bill + merged-field regression). (2) Are ALL FIVE --all caveats present in BOTH --help and code comments (concurrency/no-serialize, RequestResponse-if-order-matters, double-count, Bedrock re-bill, out-of-order regression)? (3) Does build_s3_event emit the RAW key with NO URL-decoding, and does the contract test actually FAIL if someone adds an unquote_plus (mutate it on a scratch copy and confirm red)? A key with a space/'+' must round-trip raw so the handler decodes exactly once. (4) --pipeline wo resolves the correct function name + bucket (not the PO defaults)? confirmed=true only with file:line or reproduced-failure evidence.` }, { key: 'dependency', prompt: `${PREAMBLE} ADVERSARIAL REVIEW — dependency lens. (1) moto: is tests/requirements.txt EXACT-pinned (moto==), not a floor (>=)? A floor makes the tests/ dependabot entry a no-op. (2) The three NEW dependabot dirs: do /lambdas/po/web_ui and /lambdas/po/site_extractor now have PINNED manifests that actually exist? A dependabot entry pointed at an absent or floor-pinned manifest does nothing — that is a finding. Cross-check dependabot.yml directory strings against the real file paths (a typo'd directory silently no-ops). (3) The email-processor + wo/web_ui requirements are the handbook empty-with-comment form — confirm no pinned third-party dep is left that would then require the pip step back (which would contradict the cp-only bundling). (4) Does the empty-with-comment wording match lambda-template.md (runtime provides boto3)? confirmed=true only with file:line evidence.` }, ] let round = 0 let checks = null let confirmed = [] while (round < 3) { phase('Verify') const results = await parallel([ () => agent(mechanicalPrompt, { label: `verify:mechanical-r${round}`, model: 'sonnet', phase: 'Verify', schema: CHECKS }), ...lenses.map(l => () => agent(l.prompt, { label: `verify:${l.key}-r${round}`, phase: 'Verify', schema: FINDINGS })), ]) checks = results[0] confirmed = results.slice(1).filter(Boolean) .flatMap(r => r.findings || []) .filter(f => f.confirmed && f.severity !== 'low') const green = checks && checks.passed log(`Verify round ${round}: mechanical ${checks && checks.passed ? 'GREEN' : 'RED'}, confirmed findings: ${confirmed.length}`) if (green && confirmed.length === 0) break round += 1 if (round >= 3) break phase('Fix') await agent(`${PREAMBLE} You are the fix agent — you may edit files under: ${EXPECTED_SCOPE.join(', ')}. Fix EVERY item below minimally; the binding spec and 10 pinned constraints still hold (a finding that conflicts with a constraint is reported, not "fixed" — the constraint wins, esp. constraint 5's cp-only-only-if-no-dep rule, constraint 8's do-NOT-git-rm-package, and constraint 9's no-handler / no-alarm change). Re-run the specific failing gate/test per fix. MECHANICAL:\n${checks ? checks.details : '(agent died — rerun all gates)'} CONFIRMED FINDINGS:\n${JSON.stringify(confirmed, null, 2)} ${specBlock}`, { label: `fix:round-${round}`, model: 'opus', phase: 'Fix', schema: IMPL }) } const verifyClean = checks && checks.passed && confirmed.length === 0 if (!verifyClean) { return { status: 'NEEDS ATTENTION — verify not clean after 3 rounds; branch left uncommitted', branch: BRANCH, mechanical: checks, unresolvedFindings: confirmed, implBlockers, reconBlockers, spec, } } // ----------------------------------------------------------------- package phase('Package') const commit = await agent(`${PREAMBLE.replace('do NOT commit, ', '')} YOU are the commit agent: 1. Read ~/Documents/repositories/seahaven/engineering-handbook/commit-messages.md and follow it exactly. 2. OPTIONAL LOCAL CLEANUP (constraint 8): rm -rf lambdas/po/email_processor/package/ — it is UNTRACKED, so this is a filesystem rm, NOT git rm; there is NOTHING to stage from it and it is asset-hash-neutral (Phase 2 exclude). PERFORM it and REPORT the reclaimed ~44 MB in your summary. It is NOT part of the commit. 3. git add only paths under: ${EXPECTED_SCOPE.join(', ')} and .claude/workflows/phase-7-ops-recovery.js. NOT .coverage, NOT package/ (it is gone / untracked either way). Verify the staged set with git status — the two NEW manifests and the new runbook MUST be staged. 4. ONE commit; write the message to /tmp/phase7-commit-msg.txt and use git commit -F /tmp/phase7-commit-msg.txt (backticks in -m get eaten by zsh). Suggested subject: "feat: ops/recovery tooling + dependency hygiene — generalized reprocess, DLQ runbook, cp-only bundling (refactor phase 7)" Body: the reprocess generalization (targeted-primary, --all demoted with caveats) + contract test, the DLQ runbook + windows, the dropped vendored boto3 / cp-only bundling, the moto exact-pin + three dependabot entries with pinned manifests, and the reported (not committed) package/ cleanup. NO AI attribution / Co-Authored-By lines. 5. Do NOT push. Return commit sha + shortstat + the package/ cleanup result in summary.`, { label: 'package:commit', model: 'sonnet', phase: 'Package', schema: IMPL }) return { status: 'BUILT — committed locally, NOT pushed', branch: BRANCH, base: BASE, commit: commit ? commit.summary : 'commit agent died — commit manually', spec: { reprocessCli: spec.reprocessCli, runbook: spec.runbook, dependencyEdits: spec.dependencyEdits, bundlingEdits: spec.bundlingEdits, packageCleanup: spec.packageCleanup }, implementation: implOk.map(r => r.summary), filesChanged: implOk.flatMap(r => r.filesChanged), verifyRounds: round + 1, blockers: implBlockers.concat(reconBlockers), outstandingGates: [ '/sh-security-review NOT required (ops tooling + dependency hygiene only — no untrusted-input parsing, no auth code, no IAM/policy change). The pre-push deterministic scanners still run as the unattended backstop.', 'cross-family cross_review.py NOT required (no IAM/policy change; no Lambda handler signature / event-shape / return-contract change — reprocess.py emits the SAME S3 event shape it always did, pinned by the new contract test).', 'push + PR + gh pr checks green', 'deploy-then-merge for the bundling/dep changes: deploy from branch, smoke green, one real email per pipeline (the dropped-vendored-boto3 email processors MUST still import boto3 at runtime from the Lambda runtime copy), expect ONE benign asset-hash redeploy from the package/ cleanup + cp-only change, THEN merge.', 'reprocess.py + docs/runbook-dlq-recovery.md are ops/docs (no deploy risk), but the runbook should be VALIDATED against the real DLQ / alarm / bucket names before it is relied on in an incident.', 'Confluence "AWS Architecture Map" — Phase 7 adds the DLQ-recovery runbook to the inventory; note the new runbook when the resource inventory is next refreshed.', ], }