fix(agent-team): a crashing pipeline node fails one task, not the whole daemon #53

Merged
amoussa1229 merged 1 commit from fix/coordinator-crash-on-node-exception into main 2026-06-23 20:22:53 +00:00

1 commit

Author SHA1 Message Date
f12dfecd95 fix(agent-team): a crashing pipeline node fails one task, not the whole daemon
A live task (d30b697c) on the R720 crashed the coordinator: the planner's
single-shot Claude call raised "Reached maximum number of turns (1)", the
exception propagated out of `drain_resumes` through the serve loop, and systemd
restarted the daemon — with no Slack notice, so the failure was silent.

Defense in depth:

1. invoker: `_collect_subscription_text` now tolerates the single-shot turn cap.
   When the Agent SDK raises "Reached maximum number of turns" mid-stream it
   salvages the assistant text already collected (the JSON the planner needs)
   instead of propagating. An empty salvage or any non-turn error still raises.

2. coordinator: `drain_resumes` wraps the per-job `resume()` so an unhandled
   node exception fails THAT task instead of the daemon — it supersedes the
   answered question (so the startup recovery sweep cannot re-drive it into the
   same crash on reboot), marks the task FAILED via `update_state` with a short
   `failure_reason`, and surfaces an honest "❌ FAILED" line to Slack.

3. task_model: add `failure_reason` to PipelineState + TaskRecord (kept in sync)
   so the terminal-failure detail persists as a real graph channel.

4. resume_worker: add `ResumeOutcome.FAILED`.

Tests: invoker salvage/re-raise/propagate paths; drain_resumes fails-not-crashes,
supersedes the question, notifies, and one failing task does not block others.
1189 passed.
2026-06-23 16:20:45 -04:00