open-swe/tests/test_reviewer_tools.py
Johannes du Plessis 378b95266e
feat: implement reviewer findings, publish_review, and watch mode (#1253)
* feat: implement reviewer findings, publish_review, and watch mode

Build out the reviewer agent end-to-end against the design in
REVIEWER_DESIGN.md:

- Findings as first-class state on the reviewer thread metadata
  (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line
  ranges, suggestion text for ```suggestion blocks, github_review_comment_id
  for cross-run reconciliation, diff_hunk for UI rendering. Thread-level
  metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a
  future frontend can list reviewer threads via the langgraph SDK.
- Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff,
  compute_diff_line_set for in-diff validation, extract_diff_hunk for
  caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA
  diffs against the prepped repo.
- Tools: `add_finding` (validates against the diff line set so out-of-diff
  ranges fail at creation, not at GitHub-publish), `update_finding`,
  `list_findings`, `publish_review`. The reviewer agent's tool list is
  swapped from `[]` (direct shell `gh api` calls) to these four.
- Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`):
  one POST /reviews call with body + inline comments + ```suggestion blocks,
  per-comment IDs stored back on findings, GraphQL `resolveReviewThread`
  fired for findings transitioning open->resolved on a re-review.
- Reviewer graph: deterministic clone-or-fetch + checkout in the factory
  before the agent's first model call (warm- and cold-path symmetric);
  computed diff and in-diff line set passed via runnable config; system
  prompt rewritten for the single-evolving-findings model, severity ladder,
  in-diff-only discipline, and watch-mode reconciliation flow.
- Watch mode in webapp.py: `push` event + `pull_request` closed/reopened
  added to supported events. New `process_github_push_event` resolves the
  open PR for the pushed branch, gates on the reviewer thread's `watch`
  flag, builds a re-review configurable, and triggers a run on the same
  canonical thread. `process_github_pr_close` toggles watch on
  closed/reopened. `set_reviewer_thread_metadata` is called on first
  review to install `kind=reviewer` + PR identity + watch=True.
- Eval harness: target.py now extracts `add_finding` calls (mapped to the
  legacy {file, line, body, severity} shape the judge expects) and passes
  the right configurable so the prep step has base/head SHAs.
- Tests: new unit suites for findings helpers, diff parsing, finding tools,
  publish rendering + GraphQL resolve, and watch-mode webhook handlers
  (push triggers re-review only when watching, idempotent on unchanged
  head SHA, PR close disables watch). Updated existing reviewer-webhook
  tests to mock `set_reviewer_thread_metadata`.
- REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context
  into REVIEWER_DESIGN.md.

* fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL

Address PR #1253 review findings:

- compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag
  (`option no-prefix takes no value` — every prep run was failing
  silently and the agent saw an empty diff).
- compute_diff_in_sandbox grew a `merge_base` flag. First-review path
  now uses three-dot `base...head` (the merge-base diff GitHub renders
  on Files-changed) so we don't pick up changes that landed on the base
  branch after the PR diverged. Re-review delta keeps two-dot
  `last_reviewed_sha..head` since that's exactly the new commits.
- publish_review skips findings that already carry
  `github_review_comment_id`. Without this, watched re-reviews
  re-posted every previously surfaced finding, and only the most-recent
  duplicate's id would later resolve when the issue got addressed.
- fetch_review_comments URL now includes `{pull_number}` —
  `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments`
  is the canonical endpoint; the old form 404s, so comment ids were
  never stored and watch-mode resolution couldn't run.

Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix`
flag in the executed command, and that publish_review does not re-post
findings whose `github_review_comment_id` is set.

* fix(reviewer): default publish cap from 15 to 4

A clean PR with one critical issue padded out by three lower-severity
findings is fine; fifteen is review spam. The agent can override per
call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00

180 lines
5.9 KiB
Python

"""Unit tests for the add_finding / update_finding / list_findings tools."""
from __future__ import annotations
from typing import Any
from unittest.mock import AsyncMock, patch
from agent.tools.add_finding import add_finding
from agent.tools.list_findings import list_findings
from agent.tools.update_finding import update_finding
def _config(**configurable_overrides: Any) -> dict[str, Any]:
base: dict[str, Any] = {
"configurable": {
"thread_id": "tid-1",
"head_sha": "sha-head",
"diff_text": "",
"diff_line_set": {"foo.py": [10, 11, 12]},
},
"metadata": {},
}
base["configurable"].update(configurable_overrides)
return base
def test_add_finding_rejects_invalid_severity() -> None:
with patch("agent.tools.add_finding.get_config", return_value=_config()):
result = add_finding(
severity="trivial",
category="x",
file="foo.py",
description="d",
start_line=11,
end_line=11,
)
assert result["success"] is False
assert "severity" in result["error"].lower()
def test_add_finding_rejects_out_of_diff_lines() -> None:
with patch("agent.tools.add_finding.get_config", return_value=_config()):
result = add_finding(
severity="high",
category="correctness",
file="foo.py",
description="d",
start_line=99,
end_line=99,
)
assert result["success"] is False
assert "not part of the PR diff" in result["error"]
def test_add_finding_persists_to_thread_metadata() -> None:
captured: list[Any] = []
async def fake_append(thread_id: str, finding: Any) -> Any:
captured.append((thread_id, finding))
return finding
with (
patch("agent.tools.add_finding.get_config", return_value=_config()),
patch("agent.tools.add_finding.get_thread_id_from_runtime", return_value="tid-1"),
patch("agent.tools.add_finding.append_finding", side_effect=fake_append),
):
result = add_finding(
severity="medium",
category="style",
file="foo.py",
description="rename",
start_line=11,
end_line=12,
suggestion="renamed = 1",
)
assert result["success"] is True
assert "finding_id" in result
persisted_thread, persisted = captured[0]
assert persisted_thread == "tid-1"
assert persisted["file"] == "foo.py"
assert persisted["start_line"] == 11
assert persisted["end_line"] == 12
assert persisted["suggestion"] == "renamed = 1"
assert persisted["status"] == "open"
assert persisted["first_seen_sha"] == "sha-head"
def test_add_finding_allows_file_level_with_no_lines() -> None:
with (
patch("agent.tools.add_finding.get_config", return_value=_config()),
patch("agent.tools.add_finding.get_thread_id_from_runtime", return_value="tid-1"),
patch(
"agent.tools.add_finding.append_finding",
new_callable=AsyncMock,
side_effect=lambda _t, f: f,
),
):
result = add_finding(
severity="low",
category="style",
file="missing.py",
description="file-level note",
)
assert result["success"] is True
def test_update_finding_rejects_invalid_status() -> None:
with patch("agent.tools.update_finding.get_config", return_value=_config()):
result = update_finding(finding_id="f_x", status="archived")
assert result["success"] is False
def test_update_finding_rejects_empty_update() -> None:
with patch("agent.tools.update_finding.get_config", return_value=_config()):
result = update_finding(finding_id="f_x")
assert result["success"] is False
assert "No fields" in result["error"]
def test_update_finding_passes_through_fields() -> None:
captured: list[Any] = []
async def fake_update(thread_id: str, finding_id: str, updates: Any) -> Any:
captured.append((thread_id, finding_id, updates))
return {"id": finding_id, **updates}
with (
patch("agent.tools.update_finding.get_config", return_value=_config()),
patch("agent.tools.update_finding.get_thread_id_from_runtime", return_value="tid-1"),
patch("agent.tools.update_finding.update_finding_fields", side_effect=fake_update),
):
result = update_finding(
finding_id="f_a",
status="resolved",
note="addressed by new commit",
)
assert result["success"] is True
_t, fid, updates = captured[0]
assert fid == "f_a"
assert updates["status"] == "resolved"
assert updates["last_update_note"] == "addressed by new commit"
def test_list_findings_filters_by_status() -> None:
findings = [
{"id": "f_a", "status": "open"},
{"id": "f_b", "status": "resolved"},
{"id": "f_c", "status": "open"},
]
async def fake_list(_thread_id: str) -> list[Any]:
return findings
cfg = _config()
with (
patch("agent.tools.list_findings.get_thread_id_from_runtime", return_value="tid-1"),
patch("agent.tools.list_findings.list_findings_async", side_effect=fake_list),
patch("agent.tools.add_finding.get_config", return_value=cfg),
):
result = list_findings(status_filter="open")
assert result["count"] == 2
assert [f["id"] for f in result["findings"]] == ["f_a", "f_c"]
def test_list_findings_returns_all_when_filter_omitted() -> None:
findings = [{"id": "f_a", "status": "open"}, {"id": "f_b", "status": "resolved"}]
async def fake_list(_thread_id: str) -> list[Any]:
return findings
with (
patch("agent.tools.list_findings.get_thread_id_from_runtime", return_value="tid-1"),
patch("agent.tools.list_findings.list_findings_async", side_effect=fake_list),
):
result = list_findings()
assert result["count"] == 2