* fix(reviewer): inject existing PR review threads into reviewer context
The reviewer agent was filing the same inline comment on every re-review
because it only saw findings recorded on its own thread metadata — not
the live PR review-thread state on GitHub. When a previous finding was
still open (code unchanged, or a human reply explained it), the agent
rediscovered the same defect on the next push and called `add_finding`
again, producing duplicate comments.
This change fetches the PR's review threads (across all reviewers, with
replies and isResolved status) via GraphQL and renders them into the
first-review and re-review contexts as a "Pre-existing PR review
threads" block. The system prompt now lists overlap with that block as
a hard "Do NOT file" rule, and treats threads addressed by a human
reply as resolved.
This also gives the reviewer comment-awareness on its very first run
on a PR, so it skips findings already raised by another reviewer or
bot.
* fix(reviewer): wrap PR review threads in untrusted-data XML block
Addresses the reviewer comment on this PR
(https://github.com/langchain-ai/open-swe/pull/1331#discussion_r3295497533):
PR review comment bodies are attacker-controlled (anyone who can comment
on the PR can put anything in them), and they were being concatenated
into the reviewer's system prompt with instruction-priority.
Switches the existing-threads section from a Markdown block to an XML
data block:
<pr_review_threads>
<thread location="path:line" status="open">
<comment author="open-swe[bot]">
<body>...</body>
</comment>
<comment author="romain-priour-lc">
<body>We added defaults in the template</body>
</comment>
</thread>
</pr_review_threads>
The system prompt now explicitly names the wrapper, tells the agent that
everything inside it is untrusted data from the PR (not instructions),
and that prompt-injection payloads inside bodies must be disregarded.
We keep the bodies so the agent can actually read engineer replies —
that's the whole point of comment-awareness — but they're delimited as
data, not concatenated as prose. Modern frontier models are well-trained
to honor this contract.
Additional defenses:
- Author logins are validated against the GitHub username grammar; any
unexpected value is rendered as "unknown" so the `author` attribute
can't smuggle freeform text.
- Literal closing tags (`</body>`, `</pr_review_threads>`, etc.) in
bodies are neutered so a body can't break out of its wrapper.
- Body length is capped at 4000 chars per comment to bound the prompt.
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
* feat: auto-review PRs on opened / ready-for-review
Trigger Open SWE Review on `pull_request` actions `opened` and
`ready_for_review` against the canonical reviewer thread (no need to
request open-swe[bot] as a reviewer). `converted_to_draft` now also
flips watch=False on the existing reviewer thread.
Draft PRs are gated by a tri-state user setting on the profile:
inherit team default, always on, or always off. The team-wide
`review_draft_prs` setting is the org-wide default; each user can
override it in My Settings.
External contributors with no Open SWE profile fall back to the team
default.
* fix: PR review comments — auth source + draft-aware watch toggle
- `process_github_pr_ready` now dispatches with `source="github"` so the
auth resolver finds the bot token persisted on the thread. The previous
`source="github_auto"` fell through to the email-based path in non
bot-token-only deployments and failed with a missing-user-email error.
- `converted_to_draft` no longer unconditionally clears `watch`. When the
PR author's effective `review_draft_prs` setting is on, watch stays on
so subsequent pushes still trigger re-reviews while the PR is in draft.
* feat(reviewer): skip "no issues found" comment on empty re-reviews
A re-review run with no new findings to surface no longer posts another
"Open SWE Review: No issues found" comment on the PR. The "no issues"
summary now only appears on the first review of a PR — matching Devin's
behavior, where subsequent reviews are silent unless there's something
new to flag.
Resolved-thread reconciliation and ``last_reviewed_sha`` persistence
still happen on the skipped path, so findings the user just fixed still
get their GitHub threads marked resolved, and the next push event sees
an up-to-date dedup SHA.
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
* feat: tune reviewer for precision — web/wiki tools + recalibrated prompt
Reviewer agent now has web_search, fetch_url, and http_request alongside the
finding tools, so it can verify library semantics and consult the DeepWiki
auto-generated wiki for public repos (https://deepwiki.com/<owner>/<repo>)
before flagging cross-file or architectural concerns.
Prompt rewritten to push precision over recall:
- explicit severity ladder pushing reviews toward bimodal high/low instead of
defaulting to medium
- ≤200-char description target (gold set averages ~186 chars; we were at ~436)
- mandatory docs / wiki / code lookup before flagging concurrency, security,
or perf — the three categories that dominated false positives
- "do not flag" list covering compiler/linter-catchable nits, speculative
claims without a concrete attacker/interleaving/scale, style preferences
the codebase doesn't share, and test-quality nits on non-test diffs
- smart file-selection guidance for large PRs (deprioritize generated /
vendored / pure-rename hunks)
Eval config switched to openai:gpt-5.5 + high reasoning effort for the next
benchmark run.
* trim prompt
* subagent prompting
* confidence ratings
* added medium
* enforce confidence threshold
* .
* reviewer: precision-tuned prompt + drop confidence gate
Rewrites the reviewer system prompt around a defensibility bar (anchor +
failure mode + maintainer wouldn't say "not a bug"), an explicit do-not-file
list (style nits, speculation, scope-policing, same-bug fan-out), and a
checklist of 10 bug archetypes drawn from a per-PR audit of the eval golden
set. The audit showed 145 FPs in the last eval split ~28% speculative, ~26%
style-nit, ~31% real-but-unscored (mostly same-archetype fan-out); the new
prompt targets each class directly.
Confidence is still recorded on every finding for post-hoc calibration but
no longer gates publication — the audit showed the gate was a no-op (agent
self-rated 65% of findings "high" regardless), and the prompt's defensibility
bar is the actual discipline. Drops CONFIDENCE_ORDER, CONFIDENCE_THRESHOLD,
the confidence_threshold kwarg on filter_findings_for_publish, the
confidence_filtered score_mode, and the min_confidence kwarg on the eval
target's _extract_comments — all dead once the gate is gone.
Also removes the "informational" severity tier from the Severity enum,
SEVERITY_ORDER, and all validators / tests / docstrings. It was reserved for
FYI observations the dataset never rewards.
* benchmax
* adding google provider
* slight steering
* tuning
* more tuning
* fix
* cleanup
* reducing overfitting
* Add per-repo review style profiles and inject them into the reviewer.
Dashboard users can analyze historical PR review feedback per repository,
edit the resulting style guide, and have it loaded from LangGraph Store at
reviewer runtime (including Martian eval runs) keyed by owner/name.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix review style job errors leaking exception details to clients.
Return generic dashboard messages while logging full stack traces server-side.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: tighten reviewer eval workflow
Require the reviewer to verify and dedupe findings before recording them, and make benchmark runs safe to execute against deployed reviewer graphs without posting GitHub reviews.
* chore: move reviewer eval settings to config
Load reviewer benchmark settings from the default eval config file so deployed eval runs do not require a wide CLI surface.
* feat: allow reviewer eval model overrides
Pass reviewer model and reasoning effort from the eval config into reviewer runs so isolated benchmark deployments can test Opus 4.7 high thinking.
* fix: use adaptive thinking for Opus 4.7
Switch Opus 4.7 model overrides to Anthropic adaptive thinking with effort instead of the deprecated budgeted thinking payload rejected by the API.
* refactor: use latest Anthropic effort API
Remove legacy Anthropic budget-token thinking support and route Anthropic efforts through adaptive thinking plus effort.
* revert prompting
* fix(reviewer): surface HTTP status and body excerpt for non-dict GitHub PR review responses
When post_pull_request_review received a non-dict body, it returned
None and publish_review surfaced a generic 'Failed to POST PR review'
string with no signal for the agent to adapt — leading to blind
retries with permuted cap/severity_threshold args.
Now the non-dict-body path mirrors the existing HTTPStatusError /
HTTPError paths: it returns {'_error': 'HTTP <status>: non-dict
response body: <excerpt>'} so the user-facing tool can include the
underlying detail. The bare-None branch in publish_review.py is kept
as a defensive guard with a clearer message.
* ci: apply ruff format to reviewer_publish.py
Collapse the multi-line return dict into a single line so it matches the
output of `ruff format`, unblocking the Agent lint / format-check CI jobs.
Co-authored-by: Johannes du Plessis <51395795+johannes117@users.noreply.github.com>
---------
Co-authored-by: LangSmith Issues Agent <issues-agent@langsmith.dev>
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Co-authored-by: Johannes du Plessis <51395795+johannes117@users.noreply.github.com>
When a Slack user kicks off a PR review with `@open-swe review <pr-url>`,
the reviewer agent now posts a one-line summary back to the Slack thread
when it finishes — either "No issues found" or "found N potential
issue(s)" with a link to the GitHub review.
The reviewer agent has no Slack tools by design, so the summary is sent
host-side from `publish_review` after the GitHub review POST succeeds.
The Slack channel/thread_ts is persisted on reviewer thread metadata at
trigger time and read back on publish. Re-reviews triggered by push
events stay silent in Slack to avoid noise on the original thread.
The reviewer agent was writing a 1–2 sentence "top-level take" as the
review body, which produced noisy paragraph-style summaries on PRs
("Reviewed the PR. The new ALLOWED_GITHUB_REPOS allowlist…"). Devin's
review comment is just a one-liner (`✅ No Issues Found` or
`**Devin Review** found N potential issue.`), and was preferred in the
internal A/B vs Graphite.
Drop the `summary` parameter from `publish_review`; render a fixed,
host-formatted body in `render_review_body` instead. Update the reviewer
prompt to forbid prose summaries.
* fix(reviewer): log every push/close early-return so 'silent ignore' is debuggable
Pushes to PRs that haven't had a first review fall through the watch
handler because the reviewer thread doesn't have kind=reviewer set.
Without log lines on the early-return paths, this scenario was
indistinguishable from 'webhook reached the handler at all' in the
hosted log stream.
Now every early-return logs at info or debug:
- info when a real PR exists but the reviewer thread isn't set up
(with a hint pointing at the trigger paths the user can use)
- info when the repo isn't in the reviewer allowlist
- debug for benign skips (non-branch refs, branch deletions,
already-reviewed head_sha)
* fix(reviewer): always post a summary review, even with no findings
The publish_review tool gated POSTing on `inline_comments or summary`,
so when the agent called publish_review() with no args on a clean PR
the result returned `success: true` but no GitHub review was posted —
the user got silence instead of a "no issues found" comment.
- Drop the gate so publish_review always POSTs.
- Friendlier no-findings render: `**No issues found.**` when the
findings list is empty, vs. `**No issues at or above \`<sev>\`
severity.**` with hidden count when only sub-threshold findings
exist. Agent summary renders below.
- Prompt now requires the agent to always pass a `summary` so the
body is meaningful; calls out specifically not to skip on a clean PR.
* feat: implement reviewer findings, publish_review, and watch mode
Build out the reviewer agent end-to-end against the design in
REVIEWER_DESIGN.md:
- Findings as first-class state on the reviewer thread metadata
(`agent/reviewer_findings.py`): Finding TypedDict with start_line/end_line
ranges, suggestion text for ```suggestion blocks, github_review_comment_id
for cross-run reconciliation, diff_hunk for UI rendering. Thread-level
metadata gets `kind=reviewer`, `pr`, `last_reviewed_sha`, `watch` so a
future frontend can list reviewer threads via the langgraph SDK.
- Diff utilities (`agent/reviewer_diff.py`): parse_unified_diff,
compute_diff_line_set for in-diff validation, extract_diff_hunk for
caching the hunk on a Finding, compute_diff_in_sandbox for SHA-to-SHA
diffs against the prepped repo.
- Tools: `add_finding` (validates against the diff line set so out-of-diff
ranges fail at creation, not at GitHub-publish), `update_finding`,
`list_findings`, `publish_review`. The reviewer agent's tool list is
swapped from `[]` (direct shell `gh api` calls) to these four.
- Publish path (`agent/reviewer_publish.py` + `agent/tools/publish_review.py`):
one POST /reviews call with body + inline comments + ```suggestion blocks,
per-comment IDs stored back on findings, GraphQL `resolveReviewThread`
fired for findings transitioning open->resolved on a re-review.
- Reviewer graph: deterministic clone-or-fetch + checkout in the factory
before the agent's first model call (warm- and cold-path symmetric);
computed diff and in-diff line set passed via runnable config; system
prompt rewritten for the single-evolving-findings model, severity ladder,
in-diff-only discipline, and watch-mode reconciliation flow.
- Watch mode in webapp.py: `push` event + `pull_request` closed/reopened
added to supported events. New `process_github_push_event` resolves the
open PR for the pushed branch, gates on the reviewer thread's `watch`
flag, builds a re-review configurable, and triggers a run on the same
canonical thread. `process_github_pr_close` toggles watch on
closed/reopened. `set_reviewer_thread_metadata` is called on first
review to install `kind=reviewer` + PR identity + watch=True.
- Eval harness: target.py now extracts `add_finding` calls (mapped to the
legacy {file, line, body, severity} shape the judge expects) and passes
the right configurable so the prep step has base/head SHAs.
- Tests: new unit suites for findings helpers, diff parsing, finding tools,
publish rendering + GraphQL resolve, and watch-mode webhook handlers
(push triggers re-review only when watching, idempotent on unchanged
head SHA, PR close disables watch). Updated existing reviewer-webhook
tests to mock `set_reviewer_thread_metadata`.
- REVIEWER_EVAL_PLAN.md removed per user request; folded relevant context
into REVIEWER_DESIGN.md.
* fix(reviewer): correct git diff flags, scope, dedup, and review-comments URL
Address PR #1253 review findings:
- compute_diff_in_sandbox dropped the invalid `--no-prefix=false` flag
(`option no-prefix takes no value` — every prep run was failing
silently and the agent saw an empty diff).
- compute_diff_in_sandbox grew a `merge_base` flag. First-review path
now uses three-dot `base...head` (the merge-base diff GitHub renders
on Files-changed) so we don't pick up changes that landed on the base
branch after the PR diverged. Re-review delta keeps two-dot
`last_reviewed_sha..head` since that's exactly the new commits.
- publish_review skips findings that already carry
`github_review_comment_id`. Without this, watched re-reviews
re-posted every previously surfaced finding, and only the most-recent
duplicate's id would later resolve when the issue got addressed.
- fetch_review_comments URL now includes `{pull_number}` —
`/repos/{owner}/{repo}/pulls/{pr_number}/reviews/{review_id}/comments`
is the canonical endpoint; the old form 404s, so comment ids were
never stored and watch-mode resolution couldn't run.
Three new tests cover: three-dot vs two-dot wiring, no `--no-prefix`
flag in the executed command, and that publish_review does not re-post
findings whose `github_review_comment_id` is set.
* fix(reviewer): default publish cap from 15 to 4
A clean PR with one critical issue padded out by three lower-severity
findings is fine; fifteen is review spam. The agent can override per
call when a PR genuinely warrants more.
