mirror of
https://github.com/Sea-Haven-Industries/open-swe.git
synced 2026-09-30 06:53:14 +00:00
82852f9eda
* feat: tune reviewer for precision — web/wiki tools + recalibrated prompt Reviewer agent now has web_search, fetch_url, and http_request alongside the finding tools, so it can verify library semantics and consult the DeepWiki auto-generated wiki for public repos (https://deepwiki.com/<owner>/<repo>) before flagging cross-file or architectural concerns. Prompt rewritten to push precision over recall: - explicit severity ladder pushing reviews toward bimodal high/low instead of defaulting to medium - ≤200-char description target (gold set averages ~186 chars; we were at ~436) - mandatory docs / wiki / code lookup before flagging concurrency, security, or perf — the three categories that dominated false positives - "do not flag" list covering compiler/linter-catchable nits, speculative claims without a concrete attacker/interleaving/scale, style preferences the codebase doesn't share, and test-quality nits on non-test diffs - smart file-selection guidance for large PRs (deprioritize generated / vendored / pure-rename hunks) Eval config switched to openai:gpt-5.5 + high reasoning effort for the next benchmark run. * trim prompt * subagent prompting * confidence ratings * added medium * enforce confidence threshold * . * reviewer: precision-tuned prompt + drop confidence gate Rewrites the reviewer system prompt around a defensibility bar (anchor + failure mode + maintainer wouldn't say "not a bug"), an explicit do-not-file list (style nits, speculation, scope-policing, same-bug fan-out), and a checklist of 10 bug archetypes drawn from a per-PR audit of the eval golden set. The audit showed 145 FPs in the last eval split ~28% speculative, ~26% style-nit, ~31% real-but-unscored (mostly same-archetype fan-out); the new prompt targets each class directly. Confidence is still recorded on every finding for post-hoc calibration but no longer gates publication — the audit showed the gate was a no-op (agent self-rated 65% of findings "high" regardless), and the prompt's defensibility bar is the actual discipline. Drops CONFIDENCE_ORDER, CONFIDENCE_THRESHOLD, the confidence_threshold kwarg on filter_findings_for_publish, the confidence_filtered score_mode, and the min_confidence kwarg on the eval target's _extract_comments — all dead once the gate is gone. Also removes the "informational" severity tier from the Severity enum, SEVERITY_ORDER, and all validators / tests / docstrings. It was reserved for FYI observations the dataset never rewards. * benchmax * adding google provider * slight steering * tuning * more tuning * fix * cleanup * reducing overfitting * Add per-repo review style profiles and inject them into the reviewer. Dashboard users can analyze historical PR review feedback per repository, edit the resulting style guide, and have it loaded from LangGraph Store at reviewer runtime (including Martian eval runs) keyed by owner/name. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix review style job errors leaking exception details to clients. Return generic dashboard messages while logging full stack traces server-side. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
54 lines
1.7 KiB
Python
54 lines
1.7 KiB
Python
from __future__ import annotations
|
|
|
|
from unittest.mock import AsyncMock, patch
|
|
|
|
import pytest
|
|
|
|
from agent.dashboard.review_styles import (
|
|
create_review_style,
|
|
get_repo_custom_prompt,
|
|
set_custom_prompt,
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_repo_custom_prompt_returns_trimmed_text() -> None:
|
|
with patch(
|
|
"agent.dashboard.review_styles.get_review_style",
|
|
new_callable=AsyncMock,
|
|
return_value={"custom_prompt": " Flag nil deref aggressively.\n"},
|
|
):
|
|
prompt = await get_repo_custom_prompt("acme", "repo")
|
|
assert prompt == "Flag nil deref aggressively."
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_create_review_style_puts_new_record() -> None:
|
|
mock_put = AsyncMock()
|
|
with (
|
|
patch(
|
|
"agent.dashboard.review_styles._get_value", new_callable=AsyncMock, return_value=None
|
|
),
|
|
patch("agent.dashboard.review_styles._client") as mock_client,
|
|
):
|
|
mock_client.return_value.store.put_item = mock_put
|
|
record = await create_review_style("acme/repo", "octo")
|
|
assert record["full_name"] == "acme/repo"
|
|
assert record["status"] == "idle"
|
|
mock_put.assert_awaited_once()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_set_custom_prompt_updates_store() -> None:
|
|
with (
|
|
patch(
|
|
"agent.dashboard.review_styles.get_review_style",
|
|
new_callable=AsyncMock,
|
|
return_value={"full_name": "acme/repo", "status": "completed"},
|
|
),
|
|
patch(
|
|
"agent.dashboard.review_styles.update_review_style", new_callable=AsyncMock
|
|
) as mock_up,
|
|
):
|
|
await set_custom_prompt("acme/repo", "Use direct tone.")
|
|
mock_up.assert_awaited_once()
|