open-swe/agent/tools/update_finding.py
Johannes du Plessis 82852f9eda
feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312)
* feat: tune reviewer for precision — web/wiki tools + recalibrated prompt

Reviewer agent now has web_search, fetch_url, and http_request alongside the
finding tools, so it can verify library semantics and consult the DeepWiki
auto-generated wiki for public repos (https://deepwiki.com/<owner>/<repo>)
before flagging cross-file or architectural concerns.

Prompt rewritten to push precision over recall:
- explicit severity ladder pushing reviews toward bimodal high/low instead of
  defaulting to medium
- ≤200-char description target (gold set averages ~186 chars; we were at ~436)
- mandatory docs / wiki / code lookup before flagging concurrency, security,
  or perf — the three categories that dominated false positives
- "do not flag" list covering compiler/linter-catchable nits, speculative
  claims without a concrete attacker/interleaving/scale, style preferences
  the codebase doesn't share, and test-quality nits on non-test diffs
- smart file-selection guidance for large PRs (deprioritize generated /
  vendored / pure-rename hunks)

Eval config switched to openai:gpt-5.5 + high reasoning effort for the next
benchmark run.

* trim prompt

* subagent prompting

* confidence ratings

* added medium

* enforce confidence threshold

* .

* reviewer: precision-tuned prompt + drop confidence gate

Rewrites the reviewer system prompt around a defensibility bar (anchor +
failure mode + maintainer wouldn't say "not a bug"), an explicit do-not-file
list (style nits, speculation, scope-policing, same-bug fan-out), and a
checklist of 10 bug archetypes drawn from a per-PR audit of the eval golden
set. The audit showed 145 FPs in the last eval split ~28% speculative, ~26%
style-nit, ~31% real-but-unscored (mostly same-archetype fan-out); the new
prompt targets each class directly.

Confidence is still recorded on every finding for post-hoc calibration but
no longer gates publication — the audit showed the gate was a no-op (agent
self-rated 65% of findings "high" regardless), and the prompt's defensibility
bar is the actual discipline. Drops CONFIDENCE_ORDER, CONFIDENCE_THRESHOLD,
the confidence_threshold kwarg on filter_findings_for_publish, the
confidence_filtered score_mode, and the min_confidence kwarg on the eval
target's _extract_comments — all dead once the gate is gone.

Also removes the "informational" severity tier from the Severity enum,
SEVERITY_ORDER, and all validators / tests / docstrings. It was reserved for
FYI observations the dataset never rewards.

* benchmax

* adding google provider

* slight steering

* tuning

* more tuning

* fix

* cleanup

* reducing overfitting

* Add per-repo review style profiles and inject them into the reviewer.

Dashboard users can analyze historical PR review feedback per repository,
edit the resulting style guide, and have it loaded from LangGraph Store at
reviewer runtime (including Martian eval runs) keyed by owner/name.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix review style job errors leaking exception details to clients.

Return generic dashboard messages while logging full stack traces server-side.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-20 18:35:00 +00:00

111 lines
4.5 KiB
Python

"""Tool: ``update_finding``. Mutate an existing finding by id."""
from __future__ import annotations
import asyncio
from typing import Any
from langgraph.config import get_config
from ..reviewer_findings import (
MAX_SUGGESTION_LINES,
clip_suggestion,
get_thread_id_from_runtime,
update_finding_fields,
)
def update_finding(
finding_id: str,
status: str | None = None,
severity: str | None = None,
confidence: str | None = None,
description: str | None = None,
suggestion: str | None = None,
note: str | None = None,
) -> dict[str, Any]:
"""Update fields on an existing finding.
Use this on a re-review run to mark an existing finding as resolved or
dismissed, or to revise its severity/description/suggestion if the new
commits changed the situation.
Args:
finding_id: The id returned by ``add_finding`` (or shown in the
``Existing findings`` block of the re-review user message).
status: New status (``open``, ``resolved``, ``dismissed``).
Use ``resolved`` when the new commits address the issue.
severity: New severity, if reassessing.
confidence: New confidence rating (``low``, ``medium``, ``high``), if
new commits change how sure you are the finding is a real issue.
description: New description body, if revising.
suggestion: New replacement text. Pass an empty string to clear it.
Capped at 4 lines — longer values are dropped (the finding keeps
its description). Only set this for small, obvious fixes.
note: Optional free-form note explaining the change. Persisted on the
finding under ``last_update_note``.
Returns:
Dictionary with ``success`` and (on success) the updated ``finding``.
"""
if status is not None and status not in {"open", "resolved", "dismissed"}:
return {"success": False, "error": f"Invalid status: {status}"}
if severity is not None and severity not in {"low", "medium", "high", "critical"}:
return {"success": False, "error": f"Invalid severity: {severity}"}
if confidence is not None and confidence not in {"low", "medium", "high"}:
return {"success": False, "error": f"Invalid confidence: {confidence}"}
updates: dict[str, Any] = {}
suggestion_dropped = False
if status is not None:
updates["status"] = status
if severity is not None:
updates["severity"] = severity
if confidence is not None:
updates["confidence"] = confidence
if description is not None:
updates["description"] = description
if suggestion is not None:
if suggestion == "":
updates["suggestion"] = None
else:
clipped, suggestion_dropped = clip_suggestion(suggestion)
if not suggestion_dropped:
updates["suggestion"] = clipped
if note is not None:
updates["last_update_note"] = note
config = get_config()
configurable = config.get("configurable", {}) if isinstance(config, dict) else {}
head_sha = configurable.get("head_sha", "") if isinstance(configurable, dict) else ""
if status == "open" and isinstance(head_sha, str) and head_sha:
updates["last_confirmed_sha"] = head_sha
if not updates:
if suggestion_dropped:
return {
"success": False,
"suggestion_dropped": True,
"error": (
f"Suggestion exceeded the {MAX_SUGGESTION_LINES}-line cap "
"and was rejected; no other fields were provided, so "
"nothing was updated. Only include `suggestion` for "
"small, obvious fixes."
),
}
return {"success": False, "error": "No fields provided to update"}
thread_id = get_thread_id_from_runtime()
updated = asyncio.run(update_finding_fields(thread_id, finding_id, updates))
if updated is None:
return {"success": False, "error": f"No finding found with id {finding_id}"}
result: dict[str, Any] = {"success": True, "finding": updated}
if suggestion_dropped:
result["suggestion_dropped"] = True
result["warning"] = (
f"Suggestion exceeded the {MAX_SUGGESTION_LINES}-line cap and was "
"rejected — the finding's prior `suggestion` was left unchanged "
"and other fields were updated normally. Only include "
"`suggestion` for small, obvious fixes."
)
return result