2026-02-06 13:23:09 -08:00
|
|
|
from .fetch_url import fetch_url
|
feat: add reviewer graph + eval target wiring (#1241)
* feat: add reviewer graph + eval target wiring
- New `reviewer` graph (`agent/reviewer.py`) registered in langgraph.json
alongside the main `agent` graph. Reuses the same sandbox lifecycle,
GH proxy auth, and middleware primitives from `agent.server`, but with
a narrower tool set, a reviewer-specific system prompt, no
commit/push, and the `task` (subagent) tool stripped via
`_ToolExclusionMiddleware` so review stays in one context.
- New `github_comment` tool: agents call it once per issue with
`(file, line, body, severity)` and the eval scores those calls
against golden comments.
- `ensure_no_empty_msg` middleware (the no_op nudge) is intentionally
*not* on the reviewer's stack — that middleware exists to enforce the
main agent's "always finalize via Slack/Linear/PR" contract, which
the reviewer doesn't have. The main agent's behavior is unchanged.
- `evals/reviewer/target.py`: send PR info as a user message, extract
every `github_comment` tool call (multiple expected per review) into
the run output.
- `evals/reviewer/judge.py`: per-example evaluator now returns a list
of metrics under `{"results": [...]}` so LangSmith averages each
numeric key (f1/precision/recall/tp/fp/fn) across the experiment in
the UI. Dropped the broken `aggregate_pr` summary evaluator that
reached for an attribute that doesn't exist on `RunTree`.
- `evals/reviewer/run_eval.py`: `--limit` now slices the dataset via
`client.list_examples(limit=N)` since `aevaluate` doesn't accept
`max_examples`.
- Makefile: `dev` and `run` targets now use `uv run` so they work
without an activated venv.
* resolve comments
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-06 10:15:58 -07:00
|
|
|
from .github_comment import github_comment
|
2026-02-06 13:23:09 -08:00
|
|
|
from .http_request import http_request
|
2026-03-03 14:34:29 -08:00
|
|
|
from .linear_comment import linear_comment
|
2026-03-23 14:32:44 -07:00
|
|
|
from .linear_create_issue import linear_create_issue
|
|
|
|
|
from .linear_delete_issue import linear_delete_issue
|
|
|
|
|
from .linear_get_issue import linear_get_issue
|
|
|
|
|
from .linear_get_issue_comments import linear_get_issue_comments
|
|
|
|
|
from .linear_list_teams import linear_list_teams
|
|
|
|
|
from .linear_update_issue import linear_update_issue
|
2026-04-29 17:42:27 -07:00
|
|
|
from .slack_read_thread_messages import slack_read_thread_messages
|
2026-03-04 16:43:28 -08:00
|
|
|
from .slack_thread_reply import slack_thread_reply
|
2026-03-25 16:24:31 -07:00
|
|
|
from .web_search import web_search
|
2026-02-06 13:23:09 -08:00
|
|
|
|
2026-03-04 16:43:28 -08:00
|
|
|
__all__ = [
|
|
|
|
|
"fetch_url",
|
feat: add reviewer graph + eval target wiring (#1241)
* feat: add reviewer graph + eval target wiring
- New `reviewer` graph (`agent/reviewer.py`) registered in langgraph.json
alongside the main `agent` graph. Reuses the same sandbox lifecycle,
GH proxy auth, and middleware primitives from `agent.server`, but with
a narrower tool set, a reviewer-specific system prompt, no
commit/push, and the `task` (subagent) tool stripped via
`_ToolExclusionMiddleware` so review stays in one context.
- New `github_comment` tool: agents call it once per issue with
`(file, line, body, severity)` and the eval scores those calls
against golden comments.
- `ensure_no_empty_msg` middleware (the no_op nudge) is intentionally
*not* on the reviewer's stack — that middleware exists to enforce the
main agent's "always finalize via Slack/Linear/PR" contract, which
the reviewer doesn't have. The main agent's behavior is unchanged.
- `evals/reviewer/target.py`: send PR info as a user message, extract
every `github_comment` tool call (multiple expected per review) into
the run output.
- `evals/reviewer/judge.py`: per-example evaluator now returns a list
of metrics under `{"results": [...]}` so LangSmith averages each
numeric key (f1/precision/recall/tp/fp/fn) across the experiment in
the UI. Dropped the broken `aggregate_pr` summary evaluator that
reached for an attribute that doesn't exist on `RunTree`.
- `evals/reviewer/run_eval.py`: `--limit` now slices the dataset via
`client.list_examples(limit=N)` since `aevaluate` doesn't accept
`max_examples`.
- Makefile: `dev` and `run` targets now use `uv run` so they work
without an activated venv.
* resolve comments
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-06 10:15:58 -07:00
|
|
|
"github_comment",
|
2026-03-04 16:43:28 -08:00
|
|
|
"http_request",
|
|
|
|
|
"linear_comment",
|
2026-03-23 14:32:44 -07:00
|
|
|
"linear_create_issue",
|
|
|
|
|
"linear_delete_issue",
|
|
|
|
|
"linear_get_issue",
|
|
|
|
|
"linear_get_issue_comments",
|
|
|
|
|
"linear_list_teams",
|
|
|
|
|
"linear_update_issue",
|
2026-04-29 17:42:27 -07:00
|
|
|
"slack_read_thread_messages",
|
2026-03-04 16:43:28 -08:00
|
|
|
"slack_thread_reply",
|
2026-03-25 16:24:31 -07:00
|
|
|
"web_search",
|
2026-03-04 16:43:28 -08:00
|
|
|
]
|