open-swe/agent/tools/add_finding.py
Johannes du Plessis 82852f9eda
feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312)
* feat: tune reviewer for precision — web/wiki tools + recalibrated prompt

Reviewer agent now has web_search, fetch_url, and http_request alongside the
finding tools, so it can verify library semantics and consult the DeepWiki
auto-generated wiki for public repos (https://deepwiki.com/<owner>/<repo>)
before flagging cross-file or architectural concerns.

Prompt rewritten to push precision over recall:
- explicit severity ladder pushing reviews toward bimodal high/low instead of
  defaulting to medium
- ≤200-char description target (gold set averages ~186 chars; we were at ~436)
- mandatory docs / wiki / code lookup before flagging concurrency, security,
  or perf — the three categories that dominated false positives
- "do not flag" list covering compiler/linter-catchable nits, speculative
  claims without a concrete attacker/interleaving/scale, style preferences
  the codebase doesn't share, and test-quality nits on non-test diffs
- smart file-selection guidance for large PRs (deprioritize generated /
  vendored / pure-rename hunks)

Eval config switched to openai:gpt-5.5 + high reasoning effort for the next
benchmark run.

* trim prompt

* subagent prompting

* confidence ratings

* added medium

* enforce confidence threshold

* .

* reviewer: precision-tuned prompt + drop confidence gate

Rewrites the reviewer system prompt around a defensibility bar (anchor +
failure mode + maintainer wouldn't say "not a bug"), an explicit do-not-file
list (style nits, speculation, scope-policing, same-bug fan-out), and a
checklist of 10 bug archetypes drawn from a per-PR audit of the eval golden
set. The audit showed 145 FPs in the last eval split ~28% speculative, ~26%
style-nit, ~31% real-but-unscored (mostly same-archetype fan-out); the new
prompt targets each class directly.

Confidence is still recorded on every finding for post-hoc calibration but
no longer gates publication — the audit showed the gate was a no-op (agent
self-rated 65% of findings "high" regardless), and the prompt's defensibility
bar is the actual discipline. Drops CONFIDENCE_ORDER, CONFIDENCE_THRESHOLD,
the confidence_threshold kwarg on filter_findings_for_publish, the
confidence_filtered score_mode, and the min_confidence kwarg on the eval
target's _extract_comments — all dead once the gate is gone.

Also removes the "informational" severity tier from the Severity enum,
SEVERITY_ORDER, and all validators / tests / docstrings. It was reserved for
FYI observations the dataset never rewards.

* benchmax

* adding google provider

* slight steering

* tuning

* more tuning

* fix

* cleanup

* reducing overfitting

* Add per-repo review style profiles and inject them into the reviewer.

Dashboard users can analyze historical PR review feedback per repository,
edit the resulting style guide, and have it loaded from LangGraph Store at
reviewer runtime (including Martian eval runs) keyed by owner/name.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix review style job errors leaking exception details to clients.

Return generic dashboard messages while logging full stack traces server-side.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-20 18:35:00 +00:00

158 lines
6.2 KiB
Python

"""Tool: ``add_finding``. Records one review finding on the reviewer thread."""
from __future__ import annotations
import asyncio
from typing import Any
from langgraph.config import get_config
from ..reviewer_diff import is_range_in_diff
from ..reviewer_findings import (
MAX_SUGGESTION_LINES,
Confidence,
DiffSide,
Finding,
Severity,
append_finding,
clip_suggestion,
get_thread_id_from_runtime,
new_finding,
)
def add_finding(
severity: str,
confidence: str,
category: str,
file: str,
description: str,
start_line: int | None = None,
end_line: int | None = None,
suggestion: str | None = None,
side: str = "RIGHT",
) -> dict[str, Any]:
"""Record a review finding on the reviewer thread.
Findings persist on the reviewer thread's metadata so they survive sandbox
eviction and are queryable across runs by the watch-mode reconciliation
flow and the future UI.
**When to use:** Once per distinct issue you find while reviewing the
diff. Prefer one finding per issue, with a clear ``description`` and, when
you can offer a concrete fix, a ``suggestion`` that exactly replaces lines
``start_line..end_line``.
**In-diff only:** ``start_line..end_line`` must be inside the PR diff.
File-level findings (both ``start_line`` and ``end_line`` None) are
accepted but won't render as inline GitHub comments — only use when the
issue truly isn't anchored to a line.
Args:
severity: One of ``low``, ``medium``, ``high``, ``critical``.
confidence: One of ``low``, ``medium``, ``high``.
category: Short category label (``correctness``, ``security``, ``perf``,
``style``, ``flag``, etc.). Free-form; used for grouping in the UI.
file: Repo-relative path of the file the finding refers to.
description: Markdown body the user sees.
start_line: 1-based line in the new (post-PR) file to anchor the
comment to. Pick the single most relevant line — the call site,
the signature, or the line the issue is actually about. Omit
(with ``end_line``) for file-level findings.
end_line: Accepted for API compatibility, but findings are always
anchored to a single line (``start_line``). GitHub renders
multi-line ranges as walls of context that bury the comment, so
we collapse to one line for a cleaner comment.
suggestion: Replacement text for ``start_line..end_line``. When set,
the published GitHub comment includes a ```suggestion``` block so
the user can click "Commit suggestion". **Only set this for small,
obvious fixes that fit in 4 lines or fewer** (e.g. a one-liner
rename, a missing guard, a typo). Longer suggestions are dropped
because they read as rewrites rather than reviews — leave those
cases as a description-only finding so the author can decide how
to fix it.
side: ``RIGHT`` (post-PR file, default) or ``LEFT`` (base file). Almost
always ``RIGHT``.
Returns:
Dictionary with ``success``, ``finding_id`` and (on rejection) ``error``.
"""
if start_line is not None and end_line is None:
end_line = start_line
if start_line is None and end_line is not None:
start_line = end_line
if severity not in {"low", "medium", "high", "critical"}:
return {"success": False, "error": f"Invalid severity: {severity}"}
if confidence not in {"low", "medium", "high"}:
return {"success": False, "error": f"Invalid confidence: {confidence}"}
if side not in {"LEFT", "RIGHT"}:
return {"success": False, "error": f"Invalid side: {side}"}
if start_line is not None and end_line is not None and end_line < start_line:
return {"success": False, "error": "end_line must be >= start_line"}
if start_line is not None:
end_line = start_line
config = get_config()
configurable = config.get("configurable", {}) if isinstance(config, dict) else {}
diff_line_set = configurable.get("diff_line_set") if isinstance(configurable, dict) else None
head_sha = configurable.get("head_sha", "") if isinstance(configurable, dict) else ""
diff_text = configurable.get("diff_text", "") if isinstance(configurable, dict) else ""
if isinstance(diff_line_set, dict) and not is_range_in_diff(
diff_line_set, file, start_line, end_line
):
return {
"success": False,
"error": (
f"Finding range {file}:{start_line}-{end_line} is not part of the PR diff. "
"Only review changes the PR introduces; do not flag pre-existing code."
),
}
diff_hunk: str | None = None
if isinstance(diff_text, str) and diff_text:
from ..reviewer_diff import extract_diff_hunk
diff_hunk = extract_diff_hunk(diff_text, file, start_line, end_line)
clipped_suggestion, suggestion_dropped = clip_suggestion(suggestion)
finding: Finding = new_finding(
severity=_cast_severity(severity),
confidence=_cast_confidence(confidence),
category=category,
file=file,
start_line=start_line,
end_line=end_line,
description=description,
sha=str(head_sha) if isinstance(head_sha, str) else "",
side=_cast_side(side),
suggestion=clipped_suggestion,
diff_hunk=diff_hunk,
)
thread_id = get_thread_id_from_runtime()
asyncio.run(append_finding(thread_id, finding))
result: dict[str, Any] = {"success": True, "finding_id": finding["id"]}
if suggestion_dropped:
result["suggestion_dropped"] = True
result["warning"] = (
f"Suggestion exceeded the {MAX_SUGGESTION_LINES}-line cap and was "
"dropped — the finding was recorded with description only. Only "
"include `suggestion` for small, obvious fixes."
)
return result
def _cast_severity(value: str) -> Severity:
return value # type: ignore[return-value]
def _cast_confidence(value: str) -> Confidence:
return value # type: ignore[return-value]
def _cast_side(value: str) -> DiffSide:
return value # type: ignore[return-value]