Hybrid multi-model task orchestrator — routes coding/review/scan tasks across LLMs via LangGraph
This repository has been archived on 2026-08-04. You can view files and clone it, but cannot push or open issues or pull requests.
Find a file
Adam Moussa e85a56148e
fix(agent-team): handle real slack_bolt event envelope + map thread replies; bind start invoker (#26)
* fix(agent-team): handle real slack_bolt event envelope + map thread replies to open questions

The Socket Mode inbound listener was unit-tested against a SYNTHETIC payload
shape that does not match what slack_bolt actually delivers, so the suite was
green while a real Slack thread reply was silently dropped (the clarifier
question stayed `open`). Real slack_bolt delivers an Events API message /
app_mention as `{"type":"event_callback","event":{"type":"message",...}}` and
a free-text thread reply carries NO callback_id/question_id/metadata.

Three breaks fixed (all on the free-text reply path):

1. Type gate — handle_event gated on the OUTER `type`, which is
   "event_callback" for a real message/app_mention, so the event fell outside
   _ANSWER_BEARING_TYPES and was dropped. Now collapsed to the discriminating
   INNER `event.type` via _discriminating_type / _inner_event.

2. question_id recovery — a real reply has no callback_id/question_id/metadata
   (the bot's metadata is on the QUESTION message, not the reply). When explicit
   id recovery fails, the listener now resolves the question by the inner
   event's `thread_ts` against the OPEN ledger row whose `channel_ref` equals it
   (new schema helper find_open_question_by_channel_ref, constrained to
   status='open' as anti-replay). Explicit id recovery still takes precedence.

3. answer extraction — a real message event carries its text at `event.text`,
   not a top-level `answer`/`text`. The thread-reply path now takes the inner
   `event.text` (stripped) as the answer value.

AUTHZ-01 is unchanged and still runs FIRST: authorization gates on the sender's
Slack user id (`event.user` for the Events API shape) and fails closed on an
empty/unknown allowlist or unrecoverable sender. The new mapping only resolves
WHICH question is answered, never WHO may answer. Answers stay opaque DATA
(parameterized SQL + json.dumps; never eval/exec/interpolate).

Tests: replaced the synthetic events-API fixtures with REAL Bolt envelopes and
added regression coverage — real thread reply maps via channel_ref and is
accepted, text is stripped, non-owner reply rejected (row stays open), thread_ts
matching no open row is a no-op, reply to an already-answered row is a no-op
(anti-replay), and app_mention is normalized identically. block_actions /
slash_command paths retained.

* fix(agent-team): bind subscription invoker in the start CLI

`run-team.py start` runs the clarifier graph to the first human gate IN the CLI
process, and the clarifier calls Claude (assess_confidence). The invoker is a
process-local binding that only `serve` set, so `start` failed with
"claude_invoke has no invoker bound". Bind the real subscription invoker here,
mirroring Coordinator.serve(). Found during the live R720 P1 bring-up.
2026-06-22 15:30:48 -04:00
.github ci: add workflow permissions from GHAS notes 2026-06-17 17:56:32 -04:00
agent-team fix(agent-team): handle real slack_bolt event envelope + map thread replies; bind start invoker (#26) 2026-06-22 15:30:48 -04:00
docs fix(agent-team): repair Slack listener block_actions matcher; add dedicated Slack app (#25) 2026-06-22 13:16:35 -04:00
scripts Add retrieval-augmented routing tests, fix lazy loading and open items #2-3 2026-05-26 18:26:59 -04:00
security-review feat(secrev): wire full Plane-1 roster into checker coordinator + fix fixture SAM (#22) 2026-06-18 16:31:03 -04:00
tests Add retrieval-augmented routing tests, fix lazy loading and open items #2-3 2026-05-26 18:26:59 -04:00
.env.example Add README and fix .env.example project name 2026-05-08 13:12:02 -04:00
.gitignore feat(agent-team): P3-live CI apply/verify hardening + ci_fetcher (gate-passed, provisioning-gated) (#17) 2026-06-18 15:53:26 -04:00
agents.py Add memory retriever node (Phase 2) 2026-05-15 11:36:03 -04:00
conftest.py Fix CI collection: isolate agent-team tests; add agent-team CI job 2026-06-17 15:19:51 -04:00
graph.py Add retrieval-augmented routing tests, fix lazy loading and open items #2-3 2026-05-26 18:26:59 -04:00
models.py Bump Sonnet model ID to claude-sonnet-4-6 2026-05-15 13:14:18 -04:00
README.md Document the security-review subsystem in the root README 2026-06-16 15:55:01 -04:00
requirements.txt build(deps): bump the minor-and-patch group across 1 directory with 6 updates (#11) 2026-06-18 14:44:46 -04:00
retriever.py Add retrieval-augmented routing tests, fix lazy loading and open items #2-3 2026-05-26 18:26:59 -04:00
run.py Add retrieval-augmented routing tests, fix lazy loading and open items #2-3 2026-05-26 18:26:59 -04:00
state.py Add memory retriever node (Phase 2) 2026-05-15 11:36:03 -04:00
telemetry.py Address code review FIX items from PR #1 2026-05-15 13:01:14 -04:00
tools.py Add CLI entry point and fix Composio user_id 2026-05-08 13:10:39 -04:00

orchestrator

Multi-model AI agent orchestration via LangGraph + Composio. Routes tasks to the best-fit model and connects to external services (Slack, Notion, GitHub, Google Drive). Memory-aware — each run is enriched with the top-3 most relevant notes from Adam's project/feedback/reference memory store.

Architecture

Claude Code ──► run.py ──► LangGraph StateGraph
                              │
                              ▼
                          retriever  ──► top-3 memories from
                              │           ~/.claude/projects/.../memory/
                              ▼
                          router (Sonnet, structured output)
                              │
                ┌─────────────┼─────────────────────┐
                ▼             ▼                     ▼
        ┌──────────────┐  ┌───────────┐      ┌──────────┐
        │ implementer  │  │ connector │      │ unknown  │
        │ reviewer     │  │ (Composio)│      │ (no fit) │
        │ researcher   │  └───────────┘      └──────────┘
        │ cross_reviewer│       │
        │ scanner       │       ▼
        │ fast_coder    │  tool_executor ──► summarizer
        └──────────────┘

The retriever embeds Adam's memory files once and caches vectors to .cache/embeddings.json (mtime-keyed; only changed files re-embed). Each run picks the top-3 most relevant memories and surfaces them in the CLI output before the route line.

The router uses Pydantic structured output (RouteDecision) and returns an explicit "unknown" route when no agent fits — no silent fallback. All LLM invocations are wrapped with retry-on-transient-error.

Files

File Purpose
run.py CLI entry point — python3 run.py "<task>"
graph.py LangGraph graph: retriever, router, connector, summarizer, unknown nodes
agents.py AGENTS registry (label → model_fn, prompt, description) + make_agent_node factory
models.py LLM factories, model-ID constants, with_retries() helper
state.py OrchestratorState TypedDict
retriever.py Memory loader, embedder, cache, top-k retrieval
tools.py Composio tool loading (Slack, Notion, GitHub, Google Drive)
tests/test_routing_golden.py 20-case golden-set regression test for the router

Usage

# Full execution — retrieves memory, routes, and runs the task
python3 run.py "What is the LangGraph checkpoint API?"

# Route-only — retrieves memory and prints the agent that would handle the task
python3 run.py --route-only "Review this code for security issues"

Output shape:

[retrieved: project_seahaven_slack_bot, feedback_secrets_manager, reference_sea_haven_aws]
[reviewer]

<agent output>

From Claude Code (via CLAUDE.md hybrid delegation):

python3 ~/Documents/repositories/orchestrator/run.py "<task description>"
python3 ~/Documents/repositories/orchestrator/run.py --route-only "<task description>"

When Claude Code delegates vs. handles natively

Claude Code uses a hybrid model — it delegates to the orchestrator when a different model has a genuine advantage, and handles everything else natively:

Delegate to orchestrator Handle natively in Claude Code
Cross-family code review (GPT-4.1) File editing, refactoring, bug fixes
Large codebase scanning (Gemini) Git operations, PRs, merges
Quick bounded coding (DeepSeek) AWS/SAM/CDK deployments
External service actions (Composio) Shell commands, system admin
Interactive planning and conversation

Agents

Agent Model Use Case
implementer Claude Sonnet Write code with a clear spec
reviewer Claude Sonnet Code review (BLOCK/FIX/NIT/QUESTION)
researcher Claude Haiku Doc lookups, API research
cross_reviewer GPT-4.1 Independent second-opinion review
scanner Gemini 2.5 Pro Large codebase analysis
fast_coder DeepSeek Coder Quick, bounded coding tasks
connector Sonnet + Composio Slack, Notion, GitHub, Google Drive

The router can also return done (no agent needed) or unknown (no clear fit). Model IDs are centralized as constants in models.py.

Memory retrieval

The retriever reads ~/.claude/projects/-Users-adammoussa-Documents-repositories/memory/*.md (skipping the MEMORY.md index), embeds each file once with text-embedding-3-small, and caches the vectors to .cache/embeddings.json. On subsequent runs:

  • Only files whose mtime changed are re-embedded.
  • Top-3 memories by cosine similarity are injected as system context into both the router and the agent.
  • Retrieved names are printed as the first line of every run so bad retrieval is visible.
  • Retrieval is read-only. The orchestrator never writes back to the memory store.

If retrieval fails (network, missing key), the run continues with no memory context and logs the failure into the message trail.

Connectors (via Composio)

All connections authenticated under Composio user amoussa:

  • Slack: send messages, read channels/threads, find users, add reactions
  • Notion: search/read/create/update pages, add content
  • GitHub: create issues, list issues, get repo info
  • Google Drive: find files, get metadata

The connector node is restricted to one tool call per run — a load-bearing rule learned from a 1.9M-token incident with meta-tool routing.

Security Review

The security-review/ subsystem is a high-recall, anti-complacency security gate. It is separate from the router — it does not route through run.py or LangGraph. One pure-code script, review.sh, owns the block decision (confirmed critical/high → block); no agent decides.

  • Path A — interactive: the /sh-security-review Claude Code skill (Max-covered). Narrow fresh-context detector fan-out + a proof-or-kill verifier; emits the structured finding schema for review.sh to gate.
  • Path B — unattended: a nightly two-tier sweep on the sh-secrev R720 VM. Tier 1 runs deterministic scanners (review.sh --scanners-only) over every Sea-Haven-Industries org repo; Tier 2 is a budget-bounded agentic pass (run_headless.py) on a round-robin rotation. Clean-clone auto-discovery via a read-only GitHub PAT; ALARM-only Slack (a clean night posts nothing).
  • Git hooks: global pre-commit / pre-push hooks (install-hooks.sh --global) gate every local repo via review.sh --scanners-only.

See security-review/README.md for full detail and security-review/DEPLOY-R720.md for the VM runbook.

Setup

  1. Install dependencies: pip install -r requirements.txt
  2. Copy .env.example to .env and fill in API keys
  3. Authenticate Composio integrations at app.composio.dev

Configuration

All API keys are stored in .env (gitignored):

  • ANTHROPIC_API_KEY — Claude models + router
  • OPENAI_API_KEY — GPT-4.1 cross-reviewer + text-embedding-3-small
  • GOOGLE_API_KEY — Gemini scanner
  • DEEPSEEK_API_KEY — DeepSeek fast-coder
  • COMPOSIO_API_KEY — Composio connectors
  • LANGSMITH_API_KEY — LangSmith tracing

Tracing is enabled via LangSmith (project: orchestration).

Testing

pytest tests/test_routing_golden.py -v

20 labelled tasks → expected agent. Skipped cleanly if ANTHROPIC_API_KEY or COMPOSIO_API_KEY are unset.