Commit graph

1 commit

Author SHA1 Message Date
Johannes du Plessis
6cecf429c7
feat: add reviewer eval harness (#1239)
* feat: add reviewer eval harness

Offline LangSmith eval scaffolding for the upcoming Open SWE Reviewer graph.
Imports the 50 PRs from withmartian/code-review-benchmark goldens, resolves
base/head SHAs via gh, and runs a claude-opus-4-5 LLM judge using martian's
verbatim prompt so scores are directly comparable to their published Devin
Review numbers. Reviewer graph itself is not part of this change.

* fix: ruff lint and format on reviewer eval files
2026-05-05 13:04:23 -07:00