open-swe/agent/reviewer_publish.py

503 lines
18 KiB
Python
Raw Normal View History

feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
"""GitHub Reviews API + GraphQL resolveReviewThread for the reviewer agent.
The reviewer agent calls ``publish_review`` at the end of a run. That tool
batches eligible findings (severity ≥ threshold, status=open, capped) into a
single GitHub PR Review:
- Review body: a fixed, host-formatted summary line. The agent never writes
prose here — it's either "no issues found" or "found N potential issue(s)".
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
- Inline comments: one per surfaced finding, anchored to ``path`` + ``line``
(+ ``start_line`` for ranges) + ``side``.
- Suggestion: when ``finding.suggestion`` is set, appended to the comment body
as a fenced ```suggestion``` block — gives the user the "Commit suggestion"
button on GitHub.
After publish, the returned per-comment IDs get stored back on each Finding as
``github_review_comment_id``. On a re-review run, when a finding moves
``open`` → ``resolved``, ``resolve_review_thread`` is called for that ID via
the GraphQL ``resolveReviewThread`` mutation (REST doesn't expose this).
"""
from __future__ import annotations
import json
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
import logging
from typing import Any
import httpx
from .reviewer_findings import Finding
from .utils.github_token import GitHubAuthError
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
logger = logging.getLogger(__name__)
_GITHUB_API_BASE = "https://api.github.com"
_GITHUB_GRAPHQL = "https://api.github.com/graphql"
_GITHUB_HEADERS_VERSION = "2022-11-28"
def render_inline_comment_body(finding: Finding) -> str:
"""Render the body of one inline review comment.
Format:
<description>
```suggestion
<replacement>
```
The suggestion block is only included when ``finding.suggestion`` is set.
Multi-line suggestions just become multi-line ```suggestion``` blocks.
"""
description = finding.get("description", "") or ""
marker_payload = {
"id": finding.get("id", ""),
"file_path": finding.get("file", ""),
"start_line": finding.get("start_line"),
"end_line": finding.get("end_line"),
"side": finding.get("side", "RIGHT"),
}
marker = f"<!-- open-swe-review-comment {json.dumps(marker_payload, separators=(',', ':'))} -->"
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
suggestion = finding.get("suggestion")
body = f"{marker}\n\n{description}"
if suggestion:
body = f"{body}\n\n```suggestion\n{suggestion}\n```"
return body
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
def render_inline_comment_payload(finding: Finding) -> dict[str, Any] | None:
"""Render one finding into the payload shape GitHub's Reviews API expects.
Returns ``None`` for file-level findings (no line range), since the Reviews
API requires inline comments to be anchored to a line.
"""
file = finding.get("file")
start_line = finding.get("start_line")
end_line = finding.get("end_line")
side = finding.get("side", "RIGHT")
if not file or end_line is None:
return None
payload: dict[str, Any] = {
"path": file,
"line": end_line,
"side": side,
"body": render_inline_comment_body(finding),
}
if start_line is not None and start_line != end_line:
payload["start_line"] = start_line
payload["start_side"] = side
return payload
def render_review_body(*, pr_number: int, surfaced_count: int) -> str:
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
"""Compose the top-level review body.
Two fixed shapes — no agent prose:
- 0 surfaced findings: a single "no issues" line.
- N surfaced findings: a single "found N potential issue(s)" line.
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
"""
if surfaced_count == 0:
headline = (
"## ✅ Open SWE Review: No issues found\n\n"
"Open SWE reviewed this PR and found no potential bugs to report."
fix(open-swe): always post a review summary, even with no findings (#1256) * fix(reviewer): log every push/close early-return so 'silent ignore' is debuggable Pushes to PRs that haven't had a first review fall through the watch handler because the reviewer thread doesn't have kind=reviewer set. Without log lines on the early-return paths, this scenario was indistinguishable from 'webhook reached the handler at all' in the hosted log stream. Now every early-return logs at info or debug: - info when a real PR exists but the reviewer thread isn't set up (with a hint pointing at the trigger paths the user can use) - info when the repo isn't in the reviewer allowlist - debug for benign skips (non-branch refs, branch deletions, already-reviewed head_sha) * fix(reviewer): always post a summary review, even with no findings The publish_review tool gated POSTing on `inline_comments or summary`, so when the agent called publish_review() with no args on a clean PR the result returned `success: true` but no GitHub review was posted — the user got silence instead of a "no issues found" comment. - Drop the gate so publish_review always POSTs. - Friendlier no-findings render: `**No issues found.**` when the findings list is empty, vs. `**No issues at or above \`<sev>\` severity.**` with hidden count when only sub-threshold findings exist. Agent summary renders below. - Prompt now requires the agent to always pass a `summary` so the body is meaningful; calls out specifically not to skip on a clean PR.
2026-05-07 15:16:20 -07:00
)
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
else:
issue_word = "issue" if surfaced_count == 1 else "issues"
headline = f"**Open SWE Review** found {surfaced_count} potential {issue_word}."
return f"{headline}\n\n<!-- open-swe-reviewer pr={pr_number} -->"
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
async def post_pull_request_review(
*,
owner: str,
repo: str,
pr_number: int,
head_sha: str,
body: str,
inline_comments: list[dict[str, Any]],
token: str,
) -> dict[str, Any] | None:
"""POST one GitHub PR Review with inline comments. Returns the API response or None."""
url = f"{_GITHUB_API_BASE}/repos/{owner}/{repo}/pulls/{pr_number}/reviews"
payload: dict[str, Any] = {
"commit_id": head_sha,
"event": "COMMENT",
"body": body,
"comments": inline_comments,
}
headers = _github_headers(token)
async with httpx.AsyncClient() as client:
try:
response = await client.post(url, headers=headers, json=payload, timeout=30)
if response.status_code == 401:
raise GitHubAuthError(
f"GitHub returned 401 posting PR review for {owner}/{repo}#{pr_number}"
)
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
response.raise_for_status()
except GitHubAuthError:
raise
except httpx.HTTPStatusError as e:
body = (e.response.text or "")[:500]
logger.exception(
"Failed to POST PR review for %s/%s#%s: %s %s",
owner,
repo,
pr_number,
e.response.status_code,
body,
)
return {"_error": f"HTTP {e.response.status_code}: {body}"}
except httpx.HTTPError as e:
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
logger.exception("Failed to POST PR review for %s/%s#%s", owner, repo, pr_number)
return {"_error": f"{type(e).__name__}: {e}"}
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
data = response.json()
if isinstance(data, dict):
return data
body_excerpt = (response.text or "")[:500]
logger.error(
"POST PR review for %s/%s#%s returned non-dict body: %s",
owner,
repo,
pr_number,
body_excerpt,
)
return {"_error": (f"HTTP {response.status_code}: non-dict response body: {body_excerpt}")}
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
async def fetch_review_comments(
*,
owner: str,
repo: str,
pr_number: int,
review_id: int,
token: str,
) -> list[dict[str, Any]]:
"""List the inline comments for a posted review.
GitHub's review-creation response includes a ``comments`` count but not the
per-comment IDs in all paths; this paginates the canonical list endpoint.
"""
url = f"{_GITHUB_API_BASE}/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments"
headers = _github_headers(token)
out: list[dict[str, Any]] = []
params: dict[str, Any] = {"per_page": 100, "page": 1}
async with httpx.AsyncClient() as client:
while True:
try:
response = await client.get(url, headers=headers, params=params, timeout=30)
response.raise_for_status()
except httpx.HTTPError:
logger.exception(
"Failed to list review comments for review %s on %s/%s",
review_id,
owner,
repo,
)
break
data = response.json()
if not isinstance(data, list) or not data:
break
out.extend(item for item in data if isinstance(item, dict))
if len(data) < 100: # noqa: PLR2004
break
params["page"] += 1
return out
fix: inject existing PR review threads into reviewer context (#1331) * fix(reviewer): inject existing PR review threads into reviewer context The reviewer agent was filing the same inline comment on every re-review because it only saw findings recorded on its own thread metadata — not the live PR review-thread state on GitHub. When a previous finding was still open (code unchanged, or a human reply explained it), the agent rediscovered the same defect on the next push and called `add_finding` again, producing duplicate comments. This change fetches the PR's review threads (across all reviewers, with replies and isResolved status) via GraphQL and renders them into the first-review and re-review contexts as a "Pre-existing PR review threads" block. The system prompt now lists overlap with that block as a hard "Do NOT file" rule, and treats threads addressed by a human reply as resolved. This also gives the reviewer comment-awareness on its very first run on a PR, so it skips findings already raised by another reviewer or bot. * fix(reviewer): wrap PR review threads in untrusted-data XML block Addresses the reviewer comment on this PR (https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533): PR review comment bodies are attacker-controlled (anyone who can comment on the PR can put anything in them), and they were being concatenated into the reviewer's system prompt with instruction-priority. Switches the existing-threads section from a Markdown block to an XML data block: <pr_review_threads> <thread location="path:line" status="open"> <comment author="open-swe[bot]"> <body>...</body> </comment> <comment author="romain-priour-lc"> <body>We added defaults in the template</body> </comment> </thread> </pr_review_threads> The system prompt now explicitly names the wrapper, tells the agent that everything inside it is untrusted data from the PR (not instructions), and that prompt-injection payloads inside bodies must be disregarded. We keep the bodies so the agent can actually read engineer replies — that's the whole point of comment-awareness — but they're delimited as data, not concatenated as prose. Modern frontier models are well-trained to honor this contract. Additional defenses: - Author logins are validated against the GitHub username grammar; any unexpected value is rendered as "unknown" so the `author` attribute can't smuggle freeform text. - Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in bodies are neutered so a body can't break out of its wrapper. - Body length is capped at 4000 chars per comment to bound the prompt. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-24 16:36:50 -07:00
async def fetch_pr_review_threads(
*,
owner: str,
repo: str,
pr_number: int,
token: str,
max_threads: int = 100,
max_comments_per_thread: int = 20,
) -> list[dict[str, Any]]:
"""Fetch all inline review threads on a PR (across reviewers, with replies).
Returned shape per thread:
{
"path": str,
"line": int | None,
"original_line": int | None,
"is_resolved": bool,
"is_outdated": bool,
"comments": [{"author": str, "body": str, "created_at": str}, ...],
}
Used to give the reviewer agent comment-awareness: it should not re-file a
finding that already appears as an open thread (its own or another
reviewer's), and should treat a thread as addressed when a human reply
explains the code or the thread is resolved.
"""
query = """
query Threads($owner: String!, $repo: String!, $pr: Int!, $cursor: String, $perThread: Int!) {
repository(owner: $owner, name: $repo) {
pullRequest(number: $pr) {
reviewThreads(first: 50, after: $cursor) {
pageInfo { hasNextPage endCursor }
nodes {
id
fix: inject existing PR review threads into reviewer context (#1331) * fix(reviewer): inject existing PR review threads into reviewer context The reviewer agent was filing the same inline comment on every re-review because it only saw findings recorded on its own thread metadata — not the live PR review-thread state on GitHub. When a previous finding was still open (code unchanged, or a human reply explained it), the agent rediscovered the same defect on the next push and called `add_finding` again, producing duplicate comments. This change fetches the PR's review threads (across all reviewers, with replies and isResolved status) via GraphQL and renders them into the first-review and re-review contexts as a "Pre-existing PR review threads" block. The system prompt now lists overlap with that block as a hard "Do NOT file" rule, and treats threads addressed by a human reply as resolved. This also gives the reviewer comment-awareness on its very first run on a PR, so it skips findings already raised by another reviewer or bot. * fix(reviewer): wrap PR review threads in untrusted-data XML block Addresses the reviewer comment on this PR (https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533): PR review comment bodies are attacker-controlled (anyone who can comment on the PR can put anything in them), and they were being concatenated into the reviewer's system prompt with instruction-priority. Switches the existing-threads section from a Markdown block to an XML data block: <pr_review_threads> <thread location="path:line" status="open"> <comment author="open-swe[bot]"> <body>...</body> </comment> <comment author="romain-priour-lc"> <body>We added defaults in the template</body> </comment> </thread> </pr_review_threads> The system prompt now explicitly names the wrapper, tells the agent that everything inside it is untrusted data from the PR (not instructions), and that prompt-injection payloads inside bodies must be disregarded. We keep the bodies so the agent can actually read engineer replies — that's the whole point of comment-awareness — but they're delimited as data, not concatenated as prose. Modern frontier models are well-trained to honor this contract. Additional defenses: - Author logins are validated against the GitHub username grammar; any unexpected value is rendered as "unknown" so the `author` attribute can't smuggle freeform text. - Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in bodies are neutered so a body can't break out of its wrapper. - Body length is capped at 4000 chars per comment to bound the prompt. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-24 16:36:50 -07:00
isResolved
isOutdated
path
line
originalLine
comments(first: $perThread) {
nodes {
databaseId
fix: inject existing PR review threads into reviewer context (#1331) * fix(reviewer): inject existing PR review threads into reviewer context The reviewer agent was filing the same inline comment on every re-review because it only saw findings recorded on its own thread metadata — not the live PR review-thread state on GitHub. When a previous finding was still open (code unchanged, or a human reply explained it), the agent rediscovered the same defect on the next push and called `add_finding` again, producing duplicate comments. This change fetches the PR's review threads (across all reviewers, with replies and isResolved status) via GraphQL and renders them into the first-review and re-review contexts as a "Pre-existing PR review threads" block. The system prompt now lists overlap with that block as a hard "Do NOT file" rule, and treats threads addressed by a human reply as resolved. This also gives the reviewer comment-awareness on its very first run on a PR, so it skips findings already raised by another reviewer or bot. * fix(reviewer): wrap PR review threads in untrusted-data XML block Addresses the reviewer comment on this PR (https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533): PR review comment bodies are attacker-controlled (anyone who can comment on the PR can put anything in them), and they were being concatenated into the reviewer's system prompt with instruction-priority. Switches the existing-threads section from a Markdown block to an XML data block: <pr_review_threads> <thread location="path:line" status="open"> <comment author="open-swe[bot]"> <body>...</body> </comment> <comment author="romain-priour-lc"> <body>We added defaults in the template</body> </comment> </thread> </pr_review_threads> The system prompt now explicitly names the wrapper, tells the agent that everything inside it is untrusted data from the PR (not instructions), and that prompt-injection payloads inside bodies must be disregarded. We keep the bodies so the agent can actually read engineer replies — that's the whole point of comment-awareness — but they're delimited as data, not concatenated as prose. Modern frontier models are well-trained to honor this contract. Additional defenses: - Author logins are validated against the GitHub username grammar; any unexpected value is rendered as "unknown" so the `author` attribute can't smuggle freeform text. - Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in bodies are neutered so a body can't break out of its wrapper. - Body length is capped at 4000 chars per comment to bound the prompt. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-24 16:36:50 -07:00
author { login }
authorAssociation
fix: inject existing PR review threads into reviewer context (#1331) * fix(reviewer): inject existing PR review threads into reviewer context The reviewer agent was filing the same inline comment on every re-review because it only saw findings recorded on its own thread metadata — not the live PR review-thread state on GitHub. When a previous finding was still open (code unchanged, or a human reply explained it), the agent rediscovered the same defect on the next push and called `add_finding` again, producing duplicate comments. This change fetches the PR's review threads (across all reviewers, with replies and isResolved status) via GraphQL and renders them into the first-review and re-review contexts as a "Pre-existing PR review threads" block. The system prompt now lists overlap with that block as a hard "Do NOT file" rule, and treats threads addressed by a human reply as resolved. This also gives the reviewer comment-awareness on its very first run on a PR, so it skips findings already raised by another reviewer or bot. * fix(reviewer): wrap PR review threads in untrusted-data XML block Addresses the reviewer comment on this PR (https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533): PR review comment bodies are attacker-controlled (anyone who can comment on the PR can put anything in them), and they were being concatenated into the reviewer's system prompt with instruction-priority. Switches the existing-threads section from a Markdown block to an XML data block: <pr_review_threads> <thread location="path:line" status="open"> <comment author="open-swe[bot]"> <body>...</body> </comment> <comment author="romain-priour-lc"> <body>We added defaults in the template</body> </comment> </thread> </pr_review_threads> The system prompt now explicitly names the wrapper, tells the agent that everything inside it is untrusted data from the PR (not instructions), and that prompt-injection payloads inside bodies must be disregarded. We keep the bodies so the agent can actually read engineer replies — that's the whole point of comment-awareness — but they're delimited as data, not concatenated as prose. Modern frontier models are well-trained to honor this contract. Additional defenses: - Author logins are validated against the GitHub username grammar; any unexpected value is rendered as "unknown" so the `author` attribute can't smuggle freeform text. - Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in bodies are neutered so a body can't break out of its wrapper. - Body length is capped at 4000 chars per comment to bound the prompt. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-24 16:36:50 -07:00
body
createdAt
}
}
}
}
}
}
}
"""
out: list[dict[str, Any]] = []
cursor: str | None = None
async with httpx.AsyncClient() as client:
while len(out) < max_threads:
try:
response = await client.post(
_GITHUB_GRAPHQL,
headers={"Authorization": f"Bearer {token}"},
json={
"query": query,
"variables": {
"owner": owner,
"repo": repo,
"pr": pr_number,
"cursor": cursor,
"perThread": max_comments_per_thread,
},
},
timeout=30,
)
response.raise_for_status()
except httpx.HTTPError:
logger.exception(
"Failed to fetch PR review threads for %s/%s#%s",
owner,
repo,
pr_number,
)
return out
data = response.json()
threads = (
data.get("data", {})
.get("repository", {})
.get("pullRequest", {})
.get("reviewThreads", {})
)
for thread in threads.get("nodes", []) or []:
if not isinstance(thread, dict):
continue
comments_block = thread.get("comments") or {}
comments_nodes = comments_block.get("nodes") or []
comments: list[dict[str, Any]] = []
for c in comments_nodes:
if not isinstance(c, dict):
continue
author_block = c.get("author") or {}
login = author_block.get("login") if isinstance(author_block, dict) else None
comments.append(
{
"id": c.get("databaseId")
if isinstance(c.get("databaseId"), int)
else None,
fix: inject existing PR review threads into reviewer context (#1331) * fix(reviewer): inject existing PR review threads into reviewer context The reviewer agent was filing the same inline comment on every re-review because it only saw findings recorded on its own thread metadata — not the live PR review-thread state on GitHub. When a previous finding was still open (code unchanged, or a human reply explained it), the agent rediscovered the same defect on the next push and called `add_finding` again, producing duplicate comments. This change fetches the PR's review threads (across all reviewers, with replies and isResolved status) via GraphQL and renders them into the first-review and re-review contexts as a "Pre-existing PR review threads" block. The system prompt now lists overlap with that block as a hard "Do NOT file" rule, and treats threads addressed by a human reply as resolved. This also gives the reviewer comment-awareness on its very first run on a PR, so it skips findings already raised by another reviewer or bot. * fix(reviewer): wrap PR review threads in untrusted-data XML block Addresses the reviewer comment on this PR (https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533): PR review comment bodies are attacker-controlled (anyone who can comment on the PR can put anything in them), and they were being concatenated into the reviewer's system prompt with instruction-priority. Switches the existing-threads section from a Markdown block to an XML data block: <pr_review_threads> <thread location="path:line" status="open"> <comment author="open-swe[bot]"> <body>...</body> </comment> <comment author="romain-priour-lc"> <body>We added defaults in the template</body> </comment> </thread> </pr_review_threads> The system prompt now explicitly names the wrapper, tells the agent that everything inside it is untrusted data from the PR (not instructions), and that prompt-injection payloads inside bodies must be disregarded. We keep the bodies so the agent can actually read engineer replies — that's the whole point of comment-awareness — but they're delimited as data, not concatenated as prose. Modern frontier models are well-trained to honor this contract. Additional defenses: - Author logins are validated against the GitHub username grammar; any unexpected value is rendered as "unknown" so the `author` attribute can't smuggle freeform text. - Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in bodies are neutered so a body can't break out of its wrapper. - Body length is capped at 4000 chars per comment to bound the prompt. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-24 16:36:50 -07:00
"author": login if isinstance(login, str) else "unknown",
"author_association": c.get("authorAssociation", "")
if isinstance(c.get("authorAssociation"), str)
else "",
fix: inject existing PR review threads into reviewer context (#1331) * fix(reviewer): inject existing PR review threads into reviewer context The reviewer agent was filing the same inline comment on every re-review because it only saw findings recorded on its own thread metadata — not the live PR review-thread state on GitHub. When a previous finding was still open (code unchanged, or a human reply explained it), the agent rediscovered the same defect on the next push and called `add_finding` again, producing duplicate comments. This change fetches the PR's review threads (across all reviewers, with replies and isResolved status) via GraphQL and renders them into the first-review and re-review contexts as a "Pre-existing PR review threads" block. The system prompt now lists overlap with that block as a hard "Do NOT file" rule, and treats threads addressed by a human reply as resolved. This also gives the reviewer comment-awareness on its very first run on a PR, so it skips findings already raised by another reviewer or bot. * fix(reviewer): wrap PR review threads in untrusted-data XML block Addresses the reviewer comment on this PR (https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533): PR review comment bodies are attacker-controlled (anyone who can comment on the PR can put anything in them), and they were being concatenated into the reviewer's system prompt with instruction-priority. Switches the existing-threads section from a Markdown block to an XML data block: <pr_review_threads> <thread location="path:line" status="open"> <comment author="open-swe[bot]"> <body>...</body> </comment> <comment author="romain-priour-lc"> <body>We added defaults in the template</body> </comment> </thread> </pr_review_threads> The system prompt now explicitly names the wrapper, tells the agent that everything inside it is untrusted data from the PR (not instructions), and that prompt-injection payloads inside bodies must be disregarded. We keep the bodies so the agent can actually read engineer replies — that's the whole point of comment-awareness — but they're delimited as data, not concatenated as prose. Modern frontier models are well-trained to honor this contract. Additional defenses: - Author logins are validated against the GitHub username grammar; any unexpected value is rendered as "unknown" so the `author` attribute can't smuggle freeform text. - Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in bodies are neutered so a body can't break out of its wrapper. - Body length is capped at 4000 chars per comment to bound the prompt. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-24 16:36:50 -07:00
"body": c.get("body", "") if isinstance(c.get("body"), str) else "",
"created_at": c.get("createdAt", "")
if isinstance(c.get("createdAt"), str)
else "",
}
)
out.append(
{
"id": thread.get("id") if isinstance(thread.get("id"), str) else "",
fix: inject existing PR review threads into reviewer context (#1331) * fix(reviewer): inject existing PR review threads into reviewer context The reviewer agent was filing the same inline comment on every re-review because it only saw findings recorded on its own thread metadata — not the live PR review-thread state on GitHub. When a previous finding was still open (code unchanged, or a human reply explained it), the agent rediscovered the same defect on the next push and called `add_finding` again, producing duplicate comments. This change fetches the PR's review threads (across all reviewers, with replies and isResolved status) via GraphQL and renders them into the first-review and re-review contexts as a "Pre-existing PR review threads" block. The system prompt now lists overlap with that block as a hard "Do NOT file" rule, and treats threads addressed by a human reply as resolved. This also gives the reviewer comment-awareness on its very first run on a PR, so it skips findings already raised by another reviewer or bot. * fix(reviewer): wrap PR review threads in untrusted-data XML block Addresses the reviewer comment on this PR (https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533): PR review comment bodies are attacker-controlled (anyone who can comment on the PR can put anything in them), and they were being concatenated into the reviewer's system prompt with instruction-priority. Switches the existing-threads section from a Markdown block to an XML data block: <pr_review_threads> <thread location="path:line" status="open"> <comment author="open-swe[bot]"> <body>...</body> </comment> <comment author="romain-priour-lc"> <body>We added defaults in the template</body> </comment> </thread> </pr_review_threads> The system prompt now explicitly names the wrapper, tells the agent that everything inside it is untrusted data from the PR (not instructions), and that prompt-injection payloads inside bodies must be disregarded. We keep the bodies so the agent can actually read engineer replies — that's the whole point of comment-awareness — but they're delimited as data, not concatenated as prose. Modern frontier models are well-trained to honor this contract. Additional defenses: - Author logins are validated against the GitHub username grammar; any unexpected value is rendered as "unknown" so the `author` attribute can't smuggle freeform text. - Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in bodies are neutered so a body can't break out of its wrapper. - Body length is capped at 4000 chars per comment to bound the prompt. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-05-24 16:36:50 -07:00
"path": thread.get("path", "")
if isinstance(thread.get("path"), str)
else "",
"line": thread.get("line") if isinstance(thread.get("line"), int) else None,
"original_line": thread.get("originalLine")
if isinstance(thread.get("originalLine"), int)
else None,
"is_resolved": bool(thread.get("isResolved")),
"is_outdated": bool(thread.get("isOutdated")),
"comments": comments,
}
)
if len(out) >= max_threads:
break
page_info = threads.get("pageInfo") or {}
if not page_info.get("hasNextPage"):
break
cursor = page_info.get("endCursor")
return out
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
async def fetch_review_thread_id_for_comment(
*,
owner: str,
repo: str,
pr_number: int,
review_comment_id: int,
token: str,
) -> str | None:
"""Resolve the GraphQL review-thread node id for a REST review-comment id.
GitHub's GraphQL API resolves "threads" rather than individual comments; to
resolve a thread we need its node id. The REST review-comment id is mapped
to the thread by walking the PR's review threads.
"""
query = """
query Threads($owner: String!, $repo: String!, $pr: Int!, $cursor: String) {
repository(owner: $owner, name: $repo) {
pullRequest(number: $pr) {
reviewThreads(first: 50, after: $cursor) {
pageInfo { hasNextPage endCursor }
nodes {
id
comments(first: 50) { nodes { databaseId } }
}
}
}
}
}
"""
cursor: str | None = None
async with httpx.AsyncClient() as client:
while True:
try:
response = await client.post(
_GITHUB_GRAPHQL,
headers={"Authorization": f"Bearer {token}"},
json={
"query": query,
"variables": {
"owner": owner,
"repo": repo,
"pr": pr_number,
"cursor": cursor,
},
},
timeout=30,
)
response.raise_for_status()
except httpx.HTTPError:
logger.exception(
"Failed to fetch review threads for %s/%s#%s",
owner,
repo,
pr_number,
)
return None
data = response.json()
threads = (
data.get("data", {})
.get("repository", {})
.get("pullRequest", {})
.get("reviewThreads", {})
)
for thread in threads.get("nodes", []) or []:
comment_ids = {
c.get("databaseId") for c in (thread.get("comments", {}).get("nodes") or [])
}
if review_comment_id in comment_ids:
node_id = thread.get("id")
return node_id if isinstance(node_id, str) else None
page_info = threads.get("pageInfo") or {}
if not page_info.get("hasNextPage"):
return None
cursor = page_info.get("endCursor")
async def resolve_review_thread(*, thread_node_id: str, token: str) -> bool:
"""Mark a review thread as resolved via the GraphQL ``resolveReviewThread`` mutation."""
mutation = """
mutation Resolve($threadId: ID!) {
resolveReviewThread(input: {threadId: $threadId}) {
thread { id isResolved }
}
}
"""
async with httpx.AsyncClient() as client:
try:
response = await client.post(
_GITHUB_GRAPHQL,
headers={"Authorization": f"Bearer {token}"},
json={"query": mutation, "variables": {"threadId": thread_node_id}},
timeout=30,
)
response.raise_for_status()
except httpx.HTTPError:
logger.exception("Failed to resolve review thread %s", thread_node_id)
return False
data = response.json()
if data.get("errors"):
logger.warning("resolveReviewThread errors: %s", data["errors"])
return False
thread = data.get("data", {}).get("resolveReviewThread", {}).get("thread", {})
return bool(thread.get("isResolved"))
async def reply_to_review_comment(
*,
owner: str,
repo: str,
pr_number: int,
review_comment_id: int,
body: str,
token: str,
) -> dict[str, Any] | None:
"""Reply to an existing pull request review comment thread."""
url = (
f"{_GITHUB_API_BASE}/repos/{owner}/{repo}/pulls/"
f"{pr_number}/comments/{review_comment_id}/replies"
)
async with httpx.AsyncClient() as client:
try:
response = await client.post(
url,
headers=_github_headers(token),
json={"body": body},
timeout=30,
)
if response.status_code == 401:
raise GitHubAuthError(
f"GitHub returned 401 replying to review comment {review_comment_id}"
)
response.raise_for_status()
except GitHubAuthError:
raise
except httpx.HTTPError:
logger.exception(
"Failed to reply to review comment %s on %s/%s#%s",
review_comment_id,
owner,
repo,
pr_number,
)
return None
data = response.json()
return data if isinstance(data, dict) else None
feat: implement reviewer findings, publish_review, and watch mode (#1253) * feat: implement reviewer findings, publish_review, and watch mode Build out the reviewer agent end-to-end against the design in REVIEWER_DESIGN.md: - Findings as first-class state on the reviewer thread metadata (`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line ranges, suggestion text for ```suggestion blocks, github_review_comment_id for cross-run reconciliation, diff_hunk for UI rendering. Thread-level metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a future frontend can list reviewer threads via the langgraph SDK. - Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff, compute_diff_line_set for in-diff validation, extract_diff_hunk for caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA diffs against the prepped repo. - Tools: `add_finding` (validates against the diff line set so out-of-diff ranges fail at creation, not at GitHub-publish), `update_finding`, `list_findings`, `publish_review`. The reviewer agent's tool list is swapped from `[]` (direct shell `gh api` calls) to these four. - Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`): one POST /reviews call with body + inline comments + ```suggestion blocks, per-comment IDs stored back on findings, GraphQL `resolveReviewThread` fired for findings transitioning open->resolved on a re-review. - Reviewer graph: deterministic clone-or-fetch + checkout in the factory before the agent's first model call (warm- and cold-path symmetric); computed diff and in-diff line set passed via runnable config; system prompt rewritten for the single-evolving-findings model, severity ladder, in-diff-only discipline, and watch-mode reconciliation flow. - Watch mode in webapp.py: `push` event + `pull_request` closed/reopened added to supported events. New `process_github_push_event` resolves the open PR for the pushed branch, gates on the reviewer thread's `watch` flag, builds a re-review configurable, and triggers a run on the same canonical thread. `process_github_pr_close` toggles watch on closed/reopened. `set_reviewer_thread_metadata` is called on first review to install `kind=reviewer` + PR identity + watch=True. - Eval harness: target.py now extracts `add_finding` calls (mapped to the legacy {file, line, body, severity} shape the judge expects) and passes the right configurable so the prep step has base/head SHAs. - Tests: new unit suites for findings helpers, diff parsing, finding tools, publish rendering + GraphQL resolve, and watch-mode webhook handlers (push triggers re-review only when watching, idempotent on unchanged head SHA, PR close disables watch). Updated existing reviewer-webhook tests to mock `set_reviewer_thread_metadata`. - REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context into REVIEWER_DESIGN.md. * fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL Address PR #1253 review findings: - compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag (`option no-prefix takes no value` — every prep run was failing silently and the agent saw an empty diff). - compute_diff_in_sandbox grew a `merge_base` flag. First-review path now uses three-dot `base...head` (the merge-base diff GitHub renders on Files-changed) so we don't pick up changes that landed on the base branch after the PR diverged. Re-review delta keeps two-dot `last_reviewed_sha..head` since that's exactly the new commits. - publish_review skips findings that already carry `github_review_comment_id`. Without this, watched re-reviews re-posted every previously surfaced finding, and only the most-recent duplicate's id would later resolve when the issue got addressed. - fetch_review_comments URL now includes `{pull_number}` — `/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments` is the canonical endpoint; the old form 404s, so comment ids were never stored and watch-mode resolution couldn't run. Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix` flag in the executed command, and that publish_review does not re-post findings whose `github_review_comment_id` is set. * fix(reviewer): default publish cap from 15 to 4 A clean PR with one critical issue padded out by three lower-severity findings is fine; fifteen is review spam. The agent can override per call when a PR genuinely warrants more.
2026-05-07 14:48:43 -07:00
def _github_headers(token: str) -> dict[str, str]:
return {
"Authorization": f"Bearer {token}",
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": _GITHUB_HEADERS_VERSION,
}