open-swe/tests/test_analyzer_skills.py
Johannes du Plessis 4a55145bb1
feat: outcomes dataset + bootstrap/continual split via skills (#1365)
* fix: reset stale sandbox creation sentinel

Co-authored-by: Johannes du Plessis <51395795+johannes117@users.noreply.github.com>

* fix: treat SANDBOX_CREATING as a timestamped cross-process lock

Only reset the sentinel when proven stale (older than the creation
timeout); otherwise wait for the worker that holds the lock so a
concurrent run does not create a duplicate sandbox.

* feat(analyzer): outcomes dataset + bootstrap/continual split via skills

Rename the review_style_analyzer graph to `analyzer` and split it into two
modes, plus capture reviewer finding outcomes for continual learning.

- Outcomes dataset: upsert resolved-by-commit (positive), dismissed (false
  positive), and GitHub/Slack thumbs findings into a single LangSmith dataset
  (openswe-reviewer-outcomes), keyed deterministically per finding+source.
  Emit points wired into update_finding, resolve_finding_thread, and the
  GitHub/Slack reaction handlers.
- Two playbooks delivered as deepagents skills (bootstrap-repo-analysis,
  continual-learning), served as virtual files via a CompositeBackend /skills/
  route + StateBackend (seeded into the run files channel at invoke time, never
  written to the sandbox). Mode is set by the launcher; continual runs fall
  back to the GitHub App installation token.
- Split launcher into start_bootstrap_analysis + start_continual_run; register
  a per-repo nightly continual-learning cron when bootstrap completes.
- New read_finding_outcomes tool feeds confirmed/dismissed findings back to the
  continual playbook.

Tests for outcome label mapping, skills helper, and cron idempotency.

* fix(analyzer): anchor continual cron runs to a real thread_id

The nightly continual-learning cron is threadless, and get_analyzer
early-returns an empty agent when configurable.thread_id is missing — so
every cron-launched run no-op'd before reading outcomes or saving a refined
prompt. Include the repo's deterministic analyzer thread_id in the continual
run configurable so the run executes; the threadless run carries no message
history, so nightly runs don't accumulate context.

* refactor(analyzer): move cron lifecycle calls out of the review-styles store

Drop the inline `analyzer_cron` imports from review_styles.py (added only to
dodge a circular import) by relocating the cron-trigger calls to the layer
above the store: registration to the save_review_style tool (after a prompt is
saved) and removal to the dashboard delete route. review_styles.py is now a
pure store again with top-level imports only.

* refactor: hoist reviewer_outcomes imports to module level

Move the two inline emit_finding_status_outcome imports introduced in this PR
(update_finding, resolve_finding_thread) to top-level imports. reviewer_outcomes
only depends on langsmith, so there is no circular import to avoid.

---------

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-06-01 13:25:12 -07:00

60 lines
2.1 KiB
Python

from __future__ import annotations
import pathlib
from deepagents.middleware.skills import _parse_skill_metadata
from agent.dashboard.review_style_jobs import (
build_continual_run_configurable,
build_continual_run_input,
)
from agent.utils.analyzer_skills import (
ANALYZER_MODES,
SKILLS_DIR,
build_skill_files,
skill_path_for_mode,
)
def test_build_skill_files_stripped_keys_and_valid_file_data() -> None:
files = build_skill_files()
assert set(files) == {
"/bootstrap-repo-analysis/SKILL.md",
"/continual-learning/SKILL.md",
}
for entry in files.values():
assert entry["encoding"] == "utf-8"
assert isinstance(entry["content"], str) and entry["content"].strip()
assert "created_at" in entry and "modified_at" in entry
def test_skill_path_for_mode() -> None:
assert skill_path_for_mode("bootstrap") == "/skills/bootstrap-repo-analysis/SKILL.md"
assert skill_path_for_mode("continual") == "/skills/continual-learning/SKILL.md"
# Unknown modes fall back to bootstrap.
assert skill_path_for_mode("whatever") == "/skills/bootstrap-repo-analysis/SKILL.md"
def test_bundled_skill_md_parse() -> None:
for skill in ANALYZER_MODES.values():
path = SKILLS_DIR / skill / "SKILL.md"
meta = _parse_skill_metadata(path.read_text(), str(path), path.parent.name)
assert meta["name"] == skill
assert meta["description"].strip()
def test_skills_dir_resolves() -> None:
assert SKILLS_DIR.name == "skills"
assert (SKILLS_DIR / "bootstrap-repo-analysis" / "SKILL.md").exists()
assert isinstance(SKILLS_DIR, pathlib.Path)
def test_continual_run_payload_carries_mode_and_skill_files() -> None:
configurable = build_continual_run_configurable("o/r")
assert configurable["analyzer_mode"] == "continual"
assert configurable["review_style_full_name"] == "o/r"
assert configurable.get("thread_id")
run_input = build_continual_run_input("o/r")
assert "/continual-learning/SKILL.md" in run_input["files"]
assert run_input["messages"][0]["role"] == "user"