mirror of
https://github.com/Sea-Haven-Industries/open-swe.git
synced 2026-10-01 23:13:15 +00:00
82852f9eda
* feat: tune reviewer for precision — web/wiki tools + recalibrated prompt Reviewer agent now has web_search, fetch_url, and http_request alongside the finding tools, so it can verify library semantics and consult the DeepWiki auto-generated wiki for public repos (https://deepwiki.com/<owner>/<repo>) before flagging cross-file or architectural concerns. Prompt rewritten to push precision over recall: - explicit severity ladder pushing reviews toward bimodal high/low instead of defaulting to medium - ≤200-char description target (gold set averages ~186 chars; we were at ~436) - mandatory docs / wiki / code lookup before flagging concurrency, security, or perf — the three categories that dominated false positives - "do not flag" list covering compiler/linter-catchable nits, speculative claims without a concrete attacker/interleaving/scale, style preferences the codebase doesn't share, and test-quality nits on non-test diffs - smart file-selection guidance for large PRs (deprioritize generated / vendored / pure-rename hunks) Eval config switched to openai:gpt-5.5 + high reasoning effort for the next benchmark run. * trim prompt * subagent prompting * confidence ratings * added medium * enforce confidence threshold * . * reviewer: precision-tuned prompt + drop confidence gate Rewrites the reviewer system prompt around a defensibility bar (anchor + failure mode + maintainer wouldn't say "not a bug"), an explicit do-not-file list (style nits, speculation, scope-policing, same-bug fan-out), and a checklist of 10 bug archetypes drawn from a per-PR audit of the eval golden set. The audit showed 145 FPs in the last eval split ~28% speculative, ~26% style-nit, ~31% real-but-unscored (mostly same-archetype fan-out); the new prompt targets each class directly. Confidence is still recorded on every finding for post-hoc calibration but no longer gates publication — the audit showed the gate was a no-op (agent self-rated 65% of findings "high" regardless), and the prompt's defensibility bar is the actual discipline. Drops CONFIDENCE_ORDER, CONFIDENCE_THRESHOLD, the confidence_threshold kwarg on filter_findings_for_publish, the confidence_filtered score_mode, and the min_confidence kwarg on the eval target's _extract_comments — all dead once the gate is gone. Also removes the "informational" severity tier from the Severity enum, SEVERITY_ORDER, and all validators / tests / docstrings. It was reserved for FYI observations the dataset never rewards. * benchmax * adding google provider * slight steering * tuning * more tuning * fix * cleanup * reducing overfitting * Add per-repo review style profiles and inject them into the reviewer. Dashboard users can analyze historical PR review feedback per repository, edit the resulting style guide, and have it loaded from LangGraph Store at reviewer runtime (including Martian eval runs) keyed by owner/name. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix review style job errors leaking exception details to clients. Return generic dashboard messages while logging full stack traces server-side. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
104 lines
3.8 KiB
Markdown
104 lines
3.8 KiB
Markdown
# Reviewer Eval
|
||
|
||
Offline LangSmith eval for the Open SWE Reviewer graph against the 50 PRs from
|
||
`withmartian/code-review-benchmark`. See `REVIEWER_EVAL_PLAN.md` at the repo
|
||
root for the full design.
|
||
|
||
## Layout
|
||
|
||
```
|
||
evals/reviewer/
|
||
├── golden_comments/ # 50 PRs × golden comments (copied from martian benchmark)
|
||
├── build_dataset.py # martian JSON → LangSmith dataset (resolves SHAs via gh)
|
||
├── config.toml # default benchmark run config
|
||
├── judge.py # claude-opus-4-5 pairwise match evaluator + aggregate
|
||
├── target.py # invokes the reviewer graph over langgraph_sdk
|
||
└── run_eval.py # client.aevaluate entrypoint
|
||
```
|
||
|
||
## Prerequisites
|
||
|
||
- `LANGSMITH_API_KEY` set in your env.
|
||
- `gh` authenticated (`gh auth status`) — needed for `build_dataset.py`.
|
||
- `ANTHROPIC_API_KEY` set — judge runs `claude-opus-4-5`.
|
||
- A running reviewer graph (local `langgraph dev` or deployed assistant id) with
|
||
`REVIEWER_ASSISTANT_ID` env var pointing at it. Defaults to assistant `reviewer`
|
||
on `http://localhost:2024`.
|
||
|
||
## 1. Build the dataset (once)
|
||
|
||
```bash
|
||
# Dry run — writes evals/reviewer/dataset_dryrun.json without uploading
|
||
uv run python -m evals.reviewer.build_dataset --dry-run
|
||
|
||
# Upload for real
|
||
uv run python -m evals.reviewer.build_dataset --dataset-name openswe-reviewer-v1
|
||
```
|
||
|
||
Each example carries: `repo`, `pr_number`, `pr_url`, `base_sha`, `head_sha`,
|
||
`base_ref`, `head_ref`, `pr_title`. The dataset is frozen at upload time —
|
||
upstream PR drift can't invalidate it.
|
||
|
||
## 2. Run the eval
|
||
|
||
The reviewer graph must be running and accept a `pr` input matching the
|
||
example schema, and must emit a `submit_review` tool call (or set
|
||
`state["review"]["comments"]`) with `[{file, line, severity, body}, ...]`.
|
||
|
||
```bash
|
||
uv run python -m evals.reviewer.run_eval
|
||
```
|
||
|
||
Smoke-test with 3 PRs first:
|
||
|
||
```bash
|
||
uv run python -m evals.reviewer.run_eval --limit 3
|
||
```
|
||
|
||
The runner reads benchmark settings from `evals/reviewer/config.toml`. Set the
|
||
deployment URL there (or leave it blank to use `LANGGRAPH_URL` / local dev).
|
||
The target sets `reviewer_eval` for every run, so `publish_review` does not post
|
||
to GitHub.
|
||
|
||
## Per-repo review style prompts
|
||
|
||
At runtime the reviewer loads a custom style guide from LangGraph Store when
|
||
`configurable.repo` is set (`owner` + `name` → store key `owner/name`). This
|
||
applies to **eval runs too**, as long as a completed style profile exists for
|
||
that repo.
|
||
|
||
The Martian benchmark uses these upstream repos (10 PRs each):
|
||
|
||
- `getsentry/sentry`
|
||
- `keycloak/keycloak`
|
||
- `grafana/grafana`
|
||
- `discourse/discourse`
|
||
- `calcom/cal.com`
|
||
|
||
Before scoring with repo-specific styles, run **Review styles** analysis in the
|
||
dashboard for each repo (or copy prompts into store). Re-run `make dev` so the
|
||
reviewer graph sees the same store.
|
||
|
||
By default the judge scores final `add_finding` calls. Set
|
||
`score_mode = "surfaced_findings"` in the config to score only findings that
|
||
would pass the production threshold/cap.
|
||
|
||
`model_id` and `reasoning_effort` in the config are passed to the reviewer run,
|
||
so isolated benchmark deployments can test a specific model/effort without
|
||
changing deployment-wide defaults.
|
||
|
||
## Comparing against Devin Review
|
||
|
||
Both tools are scored on the same 50 PRs with the same judge model
|
||
(`claude-opus-4-5`) and the same judge prompt (verbatim from martian
|
||
`step3_judge_comments.py`). Pull martian's published Devin numbers from their
|
||
dashboard and compare against the LangSmith experiment's `micro_*` /
|
||
`macro_*` summary metrics.
|
||
|
||
## Notes
|
||
|
||
- No GitHub forks needed — both upstream repos and martian's benchmark forks
|
||
(`ai-code-review-evaluation/*`) are public.
|
||
- `judge_match` charges judge LLM tokens proportional to
|
||
`n_candidates × n_goldens` per example. For 50 PRs with ~3 goldens each and
|
||
agents emitting ~10 candidates, expect ~1500 judge calls per experiment.
|