mirror of
https://github.com/Sea-Haven-Industries/open-swe.git
synced 2026-10-02 00:23:16 +00:00
eb18b07b20
* feat: trigger reviewer evals from the admin page Add an admin-only "Reviewer eval" section + endpoints that launch the reviewer benchmark as an isolated subprocess against the running deployment, with live status and the LangSmith experiment link. Route eval traces to a dedicated open-swe-evals project so they stay out of the production tracing project. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> * fix: reconcile reviewer eval status via heartbeat, not local process The persisted record is shared across workers but _PROCS is process-local. The owning worker now refreshes a heartbeat while the subprocess runs, and status is only reconciled to failed once the heartbeat is stale, so a poll on a worker without the local handle no longer kills a live run (and a duplicate start is rejected across workers). Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
112 lines
4.2 KiB
Markdown
112 lines
4.2 KiB
Markdown
# Reviewer Eval
|
||
|
||
Offline LangSmith eval for the Open SWE Reviewer graph against the 50 PRs from
|
||
`withmartian/code-review-benchmark`.
|
||
|
||
## Layout
|
||
|
||
```
|
||
evals/reviewer/
|
||
├── golden_comments/ # 50 PRs × golden comments (copied from martian benchmark)
|
||
├── build_dataset.py # martian JSON → LangSmith dataset (resolves SHAs via gh)
|
||
├── config.toml # default benchmark run config
|
||
├── judge.py # claude-opus-4-5 pairwise match evaluator + aggregate
|
||
├── target.py # invokes the reviewer graph over langgraph_sdk
|
||
└── run_eval.py # client.aevaluate entrypoint
|
||
```
|
||
|
||
## Prerequisites
|
||
|
||
- `LANGSMITH_API_KEY` set in your env.
|
||
- `gh` authenticated (`gh auth status`) — needed for `build_dataset.py`.
|
||
- `ANTHROPIC_API_KEY` set — judge runs `claude-opus-4-5`.
|
||
- A running reviewer graph (local `langgraph dev` or deployed assistant id) with
|
||
`REVIEWER_ASSISTANT_ID` env var pointing at it. Defaults to assistant `reviewer`
|
||
on `http://localhost:2024`.
|
||
|
||
## 1. Build the dataset (once)
|
||
|
||
```bash
|
||
# Dry run — writes evals/reviewer/dataset_dryrun.json without uploading
|
||
uv run python -m evals.reviewer.build_dataset --dry-run
|
||
|
||
# Upload for real
|
||
uv run python -m evals.reviewer.build_dataset --dataset-name openswe-reviewer-v1
|
||
```
|
||
|
||
Each example carries: `repo`, `pr_number`, `pr_url`, `base_sha`, `head_sha`,
|
||
`base_ref`, `head_ref`, `pr_title`. The dataset is frozen at upload time —
|
||
upstream PR drift can't invalidate it.
|
||
|
||
## 2. Run the eval
|
||
|
||
The reviewer graph must be running and accept a `pr` input matching the
|
||
example schema, and must emit a `submit_review` tool call (or set
|
||
`state["review"]["comments"]`) with `[{file, line, severity, body}, ...]`.
|
||
|
||
```bash
|
||
uv run python -m evals.reviewer.run_eval
|
||
```
|
||
|
||
Smoke-test with 3 PRs first:
|
||
|
||
```bash
|
||
uv run python -m evals.reviewer.run_eval --limit 3
|
||
```
|
||
|
||
### From the admin dashboard
|
||
|
||
Admins can also kick off the eval from the **Reviewer eval** section on the
|
||
dashboard Admin page (no local shell needed). It launches the same
|
||
`run_eval` runner as a subprocess against the running deployment, with an
|
||
optional limit for a smoke test, and surfaces live status plus the LangSmith
|
||
experiment link. The deployment must have `LANGSMITH_API_KEY` / `ANTHROPIC_API_KEY`
|
||
in its environment.
|
||
|
||
### Tracing project
|
||
|
||
Eval traces are routed to the **`open-swe-evals`** LangSmith project (set via
|
||
`langsmith_project` in `config.toml`, default `open-swe-evals`) so they stay out
|
||
of the deployment's production tracing project. The admin-triggered run forces
|
||
the same project via the `LANGSMITH_PROJECT` env var; override the default with
|
||
`EVAL_LANGSMITH_PROJECT`.
|
||
|
||
The runner reads benchmark settings from `evals/reviewer/config.toml`. Set the
|
||
deployment URL there (or leave it blank to use `LANGGRAPH_URL` / local dev).
|
||
The target sets `reviewer_eval` for every run, so `publish_review` does not post
|
||
to GitHub.
|
||
|
||
## Per-repo review style prompts
|
||
|
||
At runtime the reviewer loads a custom style guide from LangGraph Store when
|
||
`configurable.repo` is set (`owner` + `name` → store key `owner/name`). This
|
||
applies to **eval runs too**, as long as a completed style profile exists for
|
||
that repo.
|
||
|
||
The Martian benchmark uses these upstream repos (10 PRs each):
|
||
|
||
- `getsentry/sentry`
|
||
- `keycloak/keycloak`
|
||
- `grafana/grafana`
|
||
- `discourse/discourse`
|
||
- `calcom/cal.com`
|
||
|
||
Before scoring with repo-specific styles, run **Review styles** analysis in the
|
||
dashboard for each repo (or copy prompts into store). Re-run `make dev` so the
|
||
reviewer graph sees the same store.
|
||
|
||
By default the judge scores final `add_finding` calls. Set
|
||
`score_mode = "surfaced_findings"` in the config to score only findings that
|
||
would pass the production threshold/cap.
|
||
|
||
`model_id` and `reasoning_effort` in the config are passed to the reviewer run,
|
||
so isolated benchmark deployments can test a specific model/effort without
|
||
changing deployment-wide defaults.
|
||
|
||
## Notes
|
||
|
||
- No GitHub forks needed — both upstream repos and martian's benchmark forks
|
||
(`ai-code-review-evaluation/*`) are public.
|
||
- `judge_match` charges judge LLM tokens proportional to
|
||
`n_candidates × n_goldens` per example. For 50 PRs with ~3 goldens each and
|
||
agents emitting ~10 candidates, expect ~1500 judge calls per experiment.
|