mirror of
https://github.com/Sea-Haven-Industries/open-swe.git
synced 2026-09-30 17:23:15 +00:00
378b95266e
* feat: implement reviewer findings, publish_review, and watch mode
Build out the reviewer agent end-to-end against the design in
REVIEWER_DESIGN.md:
- Findings as first-class state on the reviewer thread metadata
(`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line
ranges, suggestion text for ```suggestion blocks, github_review_comment_id
for cross-run reconciliation, diff_hunk for UI rendering. Thread-level
metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a
future frontend can list reviewer threads via the langgraph SDK.
- Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff,
compute_diff_line_set for in-diff validation, extract_diff_hunk for
caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA
diffs against the prepped repo.
- Tools: `add_finding` (validates against the diff line set so out-of-diff
ranges fail at creation, not at GitHub-publish), `update_finding`,
`list_findings`, `publish_review`. The reviewer agent's tool list is
swapped from `[]` (direct shell `gh api` calls) to these four.
- Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`):
one POST /reviews call with body + inline comments + ```suggestion blocks,
per-comment IDs stored back on findings, GraphQL `resolveReviewThread`
fired for findings transitioning open->resolved on a re-review.
- Reviewer graph: deterministic clone-or-fetch + checkout in the factory
before the agent's first model call (warm- and cold-path symmetric);
computed diff and in-diff line set passed via runnable config; system
prompt rewritten for the single-evolving-findings model, severity ladder,
in-diff-only discipline, and watch-mode reconciliation flow.
- Watch mode in webapp.py: `push` event + `pull_request` closed/reopened
added to supported events. New `process_github_push_event` resolves the
open PR for the pushed branch, gates on the reviewer thread's `watch`
flag, builds a re-review configurable, and triggers a run on the same
canonical thread. `process_github_pr_close` toggles watch on
closed/reopened. `set_reviewer_thread_metadata` is called on first
review to install `kind=reviewer` + PR identity + watch=True.
- Eval harness: target.py now extracts `add_finding` calls (mapped to the
legacy {file, line, body, severity} shape the judge expects) and passes
the right configurable so the prep step has base/head SHAs.
- Tests: new unit suites for findings helpers, diff parsing, finding tools,
publish rendering + GraphQL resolve, and watch-mode webhook handlers
(push triggers re-review only when watching, idempotent on unchanged
head SHA, PR close disables watch). Updated existing reviewer-webhook
tests to mock `set_reviewer_thread_metadata`.
- REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context
into REVIEWER_DESIGN.md.
* fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL
Address PR #1253 review findings:
- compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag
(`option no-prefix takes no value` — every prep run was failing
silently and the agent saw an empty diff).
- compute_diff_in_sandbox grew a `merge_base` flag. First-review path
now uses three-dot `base...head` (the merge-base diff GitHub renders
on Files-changed) so we don't pick up changes that landed on the base
branch after the PR diverged. Re-review delta keeps two-dot
`last_reviewed_sha..head` since that's exactly the new commits.
- publish_review skips findings that already carry
`github_review_comment_id`. Without this, watched re-reviews
re-posted every previously surfaced finding, and only the most-recent
duplicate's id would later resolve when the issue got addressed.
- fetch_review_comments URL now includes `{pull_number}` —
`/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments`
is the canonical endpoint; the old form 404s, so comment ids were
never stored and watch-mode resolution couldn't run.
Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix`
flag in the executed command, and that publish_review does not re-post
findings whose `github_review_comment_id` is set.
* fix(reviewer): default publish cap from 15 to 4
A clean PR with one critical issue padded out by three lower-severity
findings is fine; fifteen is review spam. The agent can override per
call when a PR genuinely warrants more.
232 lines
7.6 KiB
Python
232 lines
7.6 KiB
Python
"""Unit tests for the watch-mode webhook handlers."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
from unittest.mock import AsyncMock, MagicMock, patch
|
|
|
|
import pytest
|
|
|
|
from agent import webapp
|
|
|
|
|
|
def _push_payload(*, ref: str, after: str, owner: str = "lc", name: str = "repo") -> dict[str, Any]:
|
|
return {
|
|
"ref": ref,
|
|
"after": after,
|
|
"repository": {"owner": {"login": owner}, "name": name},
|
|
"sender": {"login": "alice", "id": 7},
|
|
}
|
|
|
|
|
|
def _pr_close_payload(*, action: str, number: int = 7) -> dict[str, Any]:
|
|
return {
|
|
"action": action,
|
|
"repository": {"owner": {"login": "lc"}, "name": "repo"},
|
|
"pull_request": {"number": number, "head": {"ref": "feat-x"}},
|
|
}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_push_event_skips_branch_deletion() -> None:
|
|
payload = _push_payload(
|
|
ref="refs/heads/feat-x", after="0000000000000000000000000000000000000000"
|
|
)
|
|
with patch("agent.webapp._is_repo_allowed_for_reviewer", return_value=True):
|
|
await webapp.process_github_push_event(payload)
|
|
# If we got here without crashing and with no other patches needed, the
|
|
# function returned early on the deletion check.
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_push_event_skips_when_thread_not_watching() -> None:
|
|
payload = _push_payload(ref="refs/heads/feat-x", after="newsha")
|
|
pr = {
|
|
"number": 7,
|
|
"html_url": "https://github.com/lc/repo/pull/7",
|
|
"title": "T",
|
|
"head": {"sha": "newsha", "ref": "feat-x"},
|
|
"base": {"sha": "basesha", "ref": "main"},
|
|
}
|
|
fake_client = MagicMock()
|
|
fake_client.runs.create = AsyncMock()
|
|
|
|
with (
|
|
patch("agent.webapp._is_repo_allowed_for_reviewer", return_value=True),
|
|
patch(
|
|
"agent.webapp.get_github_app_installation_token",
|
|
new_callable=AsyncMock,
|
|
return_value="t",
|
|
),
|
|
patch(
|
|
"agent.webapp._fetch_open_pr_for_branch",
|
|
new_callable=AsyncMock,
|
|
return_value=pr,
|
|
),
|
|
patch(
|
|
"agent.webapp._get_thread_metadata_safe",
|
|
new_callable=AsyncMock,
|
|
return_value={"kind": "reviewer", "watch": False},
|
|
),
|
|
patch("agent.webapp.get_client", return_value=fake_client),
|
|
):
|
|
await webapp.process_github_push_event(payload)
|
|
fake_client.runs.create.assert_not_called()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_push_event_triggers_re_review_run_when_watching() -> None:
|
|
payload = _push_payload(ref="refs/heads/feat-x", after="newsha")
|
|
pr = {
|
|
"number": 7,
|
|
"html_url": "https://github.com/lc/repo/pull/7",
|
|
"title": "T",
|
|
"head": {"sha": "newsha", "ref": "feat-x"},
|
|
"base": {"sha": "basesha", "ref": "main"},
|
|
}
|
|
fake_client = MagicMock()
|
|
fake_client.runs.create = AsyncMock()
|
|
|
|
with (
|
|
patch("agent.webapp._is_repo_allowed_for_reviewer", return_value=True),
|
|
patch(
|
|
"agent.webapp.get_github_app_installation_token",
|
|
new_callable=AsyncMock,
|
|
return_value="t",
|
|
),
|
|
patch(
|
|
"agent.webapp._fetch_open_pr_for_branch",
|
|
new_callable=AsyncMock,
|
|
return_value=pr,
|
|
),
|
|
patch(
|
|
"agent.webapp._get_thread_metadata_safe",
|
|
new_callable=AsyncMock,
|
|
return_value={
|
|
"kind": "reviewer",
|
|
"watch": True,
|
|
"last_reviewed_sha": "oldsha",
|
|
},
|
|
),
|
|
patch(
|
|
"agent.webapp._ensure_thread_exists_for_metadata",
|
|
new_callable=AsyncMock,
|
|
return_value=True,
|
|
),
|
|
patch(
|
|
"agent.webapp.persist_encrypted_github_token",
|
|
new_callable=AsyncMock,
|
|
return_value="enc",
|
|
),
|
|
patch(
|
|
"agent.webapp.set_reviewer_thread_metadata",
|
|
new_callable=AsyncMock,
|
|
),
|
|
patch("agent.webapp.is_thread_active", new_callable=AsyncMock, return_value=False),
|
|
patch("agent.webapp.get_client", return_value=fake_client),
|
|
):
|
|
await webapp.process_github_push_event(payload)
|
|
|
|
fake_client.runs.create.assert_awaited_once()
|
|
args, kwargs = fake_client.runs.create.await_args
|
|
assert args[1] == "reviewer"
|
|
configurable = kwargs["config"]["configurable"]
|
|
assert configurable["re_review"] is True
|
|
assert configurable["last_reviewed_sha"] == "oldsha"
|
|
assert configurable["head_sha"] == "newsha"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_push_event_idempotent_when_head_unchanged() -> None:
|
|
payload = _push_payload(ref="refs/heads/feat-x", after="samesha")
|
|
pr = {
|
|
"number": 7,
|
|
"html_url": "https://github.com/lc/repo/pull/7",
|
|
"title": "T",
|
|
"head": {"sha": "samesha", "ref": "feat-x"},
|
|
"base": {"sha": "basesha", "ref": "main"},
|
|
}
|
|
fake_client = MagicMock()
|
|
fake_client.runs.create = AsyncMock()
|
|
|
|
with (
|
|
patch("agent.webapp._is_repo_allowed_for_reviewer", return_value=True),
|
|
patch(
|
|
"agent.webapp.get_github_app_installation_token",
|
|
new_callable=AsyncMock,
|
|
return_value="t",
|
|
),
|
|
patch(
|
|
"agent.webapp._fetch_open_pr_for_branch",
|
|
new_callable=AsyncMock,
|
|
return_value=pr,
|
|
),
|
|
patch(
|
|
"agent.webapp._get_thread_metadata_safe",
|
|
new_callable=AsyncMock,
|
|
return_value={
|
|
"kind": "reviewer",
|
|
"watch": True,
|
|
"last_reviewed_sha": "samesha",
|
|
},
|
|
),
|
|
patch("agent.webapp.get_client", return_value=fake_client),
|
|
):
|
|
await webapp.process_github_push_event(payload)
|
|
fake_client.runs.create.assert_not_called()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_pr_close_disables_watch() -> None:
|
|
captured: list[Any] = []
|
|
|
|
async def fake_set(thread_id: str, **kwargs: Any) -> None:
|
|
captured.append((thread_id, kwargs))
|
|
|
|
with (
|
|
patch("agent.webapp._is_repo_allowed_for_reviewer", return_value=True),
|
|
patch(
|
|
"agent.webapp._get_thread_metadata_safe",
|
|
new_callable=AsyncMock,
|
|
return_value={"kind": "reviewer", "watch": True},
|
|
),
|
|
patch("agent.webapp.set_reviewer_thread_metadata", side_effect=fake_set),
|
|
):
|
|
await webapp.process_github_pr_close(_pr_close_payload(action="closed"))
|
|
assert captured and captured[0][1]["watch"] is False
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_pr_reopened_re_enables_watch() -> None:
|
|
captured: list[Any] = []
|
|
|
|
async def fake_set(thread_id: str, **kwargs: Any) -> None:
|
|
captured.append((thread_id, kwargs))
|
|
|
|
with (
|
|
patch("agent.webapp._is_repo_allowed_for_reviewer", return_value=True),
|
|
patch(
|
|
"agent.webapp._get_thread_metadata_safe",
|
|
new_callable=AsyncMock,
|
|
return_value={"kind": "reviewer", "watch": False},
|
|
),
|
|
patch("agent.webapp.set_reviewer_thread_metadata", side_effect=fake_set),
|
|
):
|
|
await webapp.process_github_pr_close(_pr_close_payload(action="reopened"))
|
|
assert captured and captured[0][1]["watch"] is True
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_pr_close_skips_non_reviewer_threads() -> None:
|
|
fake_set = AsyncMock()
|
|
with (
|
|
patch("agent.webapp._is_repo_allowed_for_reviewer", return_value=True),
|
|
patch(
|
|
"agent.webapp._get_thread_metadata_safe",
|
|
new_callable=AsyncMock,
|
|
return_value={"kind": "agent"},
|
|
),
|
|
patch("agent.webapp.set_reviewer_thread_metadata", new=fake_set),
|
|
):
|
|
await webapp.process_github_pr_close(_pr_close_payload(action="closed"))
|
|
fake_set.assert_not_called()
|