open-swe/tests
Johannes du Plessis 4a55145bb1
feat: outcomes dataset + bootstrap/continual split via skills (#1365)
* fix: reset stale sandbox creation sentinel

Co-authored-by: Johannes du Plessis <51395795+johannes117@users.noreply.github.com>

* fix: treat SANDBOX_CREATING as a timestamped cross-process lock

Only reset the sentinel when proven stale (older than the creation
timeout); otherwise wait for the worker that holds the lock so a
concurrent run does not create a duplicate sandbox.

* feat(analyzer): outcomes dataset + bootstrap/continual split via skills

Rename the review_style_analyzer graph to `analyzer` and split it into two
modes, plus capture reviewer finding outcomes for continual learning.

- Outcomes dataset: upsert resolved-by-commit (positive), dismissed (false
  positive), and GitHub/Slack thumbs findings into a single LangSmith dataset
  (openswe-reviewer-outcomes), keyed deterministically per finding+source.
  Emit points wired into update_finding, resolve_finding_thread, and the
  GitHub/Slack reaction handlers.
- Two playbooks delivered as deepagents skills (bootstrap-repo-analysis,
  continual-learning), served as virtual files via a CompositeBackend /skills/
  route + StateBackend (seeded into the run files channel at invoke time, never
  written to the sandbox). Mode is set by the launcher; continual runs fall
  back to the GitHub App installation token.
- Split launcher into start_bootstrap_analysis + start_continual_run; register
  a per-repo nightly continual-learning cron when bootstrap completes.
- New read_finding_outcomes tool feeds confirmed/dismissed findings back to the
  continual playbook.

Tests for outcome label mapping, skills helper, and cron idempotency.

* fix(analyzer): anchor continual cron runs to a real thread_id

The nightly continual-learning cron is threadless, and get_analyzer
early-returns an empty agent when configurable.thread_id is missing — so
every cron-launched run no-op'd before reading outcomes or saving a refined
prompt. Include the repo's deterministic analyzer thread_id in the continual
run configurable so the run executes; the threadless run carries no message
history, so nightly runs don't accumulate context.

* refactor(analyzer): move cron lifecycle calls out of the review-styles store

Drop the inline `analyzer_cron` imports from review_styles.py (added only to
dodge a circular import) by relocating the cron-trigger calls to the layer
above the store: registration to the save_review_style tool (after a prompt is
saved) and removal to the dashboard delete route. review_styles.py is now a
pure store again with top-level imports only.

* refactor: hoist reviewer_outcomes imports to module level

Move the two inline emit_finding_status_outcome imports introduced in this PR
(update_finding, resolve_finding_thread) to top-level imports. reviewer_outcomes
only depends on langsmith, so there is no circular import to avoid.

---------

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-06-01 13:25:12 -07:00
..
middleware fix: keep sandbox backend stable across recovery (#1294)w 2026-05-11 16:03:38 -07:00
conftest.py Remove reviewer env allowlist (#1353) 2026-05-28 14:29:33 -07:00
test_agent_subagent_models.py feat: upgrade default agent + reviewer model to Opus 4.8 (#1350) 2026-05-28 10:59:29 -07:00
test_analyzer_cron.py feat: outcomes dataset + bootstrap/continual split via skills (#1365) 2026-06-01 13:25:12 -07:00
test_analyzer_skills.py feat: outcomes dataset + bootstrap/continual split via skills (#1365) 2026-06-01 13:25:12 -07:00
test_anthropic_effort.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_auth_sources.py chore: Drop monorepo (#1029) 2026-03-06 16:10:34 -08:00
test_dashboard_message_adapter.py feat: add Agents chat UI for cloud threads (#1323) 2026-05-22 18:15:59 +00:00
test_daytona_integration.py fix(daytona): make sandbox snapshot configurable (#1220) 2026-05-01 22:51:34 +00:00
test_encryption.py feat: support TOKEN_ENCRYPTION_KEY rotation via MultiFernet [closes AB-2323] (#1275) 2026-05-08 14:29:23 -07:00
test_ensure_no_empty_msg.py feat: add Agents chat UI for cloud threads (#1323) 2026-05-22 18:15:59 +00:00
test_github_comment_prompts.py fix: preserve pushed commit history (#1360) 2026-05-29 15:07:18 -07:00
test_github_feedback.py feat: reconcile reviewer comment lifecycle (#1332) 2026-05-26 16:24:34 -07:00
test_github_issue_webhook.py Remove reviewer env allowlist (#1353) 2026-05-28 14:29:33 -07:00
test_github_oauth_refresh.py fix: review style prompts UX, stale runs, and OAuth refresh (#1321) 2026-05-21 17:44:04 +00:00
test_github_token_ttl.py feat: TTL and revocation handling for cached GitHub OAuth tokens [closes AB-2322] (#1280) 2026-05-08 22:57:01 +00:00
test_google_model.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_http_security.py fix: harden http_request SSRF guard against DNS rebinding [closes AB-2321] (#1277) 2026-05-08 22:06:25 +00:00
test_langsmith_sandbox_config.py feat: add idle TTL and delete-after-stop sandbox lifecycle controls (#1265) 2026-05-08 00:30:14 -04:00
test_model_fallback_middleware.py feat: cross-provider model fallback on transient errors (#1281) 2026-05-08 15:35:13 -07:00
test_model_fallback_resolution.py feat: upgrade default agent + reviewer model to Opus 4.8 (#1350) 2026-05-28 10:59:29 -07:00
test_multimodal.py chore: Drop monorepo (#1029) 2026-03-06 16:10:34 -08:00
test_normalize_repo.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_notify_step_limit_middleware.py fix: notify users via Slack when agent hits model call step limit (#1204) 2026-05-01 14:24:25 -07:00
test_pr_ready_auto_review.py Remove reviewer env allowlist (#1353) 2026-05-28 14:29:33 -07:00
test_proxy_auth.py feat: add Agents chat UI for cloud threads (#1323) 2026-05-22 18:15:59 +00:00
test_public_repo_org_gate.py Remove reviewer env allowlist (#1353) 2026-05-28 14:29:33 -07:00
test_recent_comments.py chore: Drop monorepo (#1029) 2026-03-06 16:10:34 -08:00
test_refresh_slack_status_middleware.py feat: add optional Slack Assistants API typing status indicator (#1269) 2026-05-08 10:21:55 -07:00
test_repo_extraction.py feat: support allowed repos in addition to allowed orgs for webhook filtering (#1092) 2026-05-08 13:26:30 -07:00
test_review_style_collector.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_review_style_sync.py fix: review style prompts UX, stale runs, and OAuth refresh (#1321) 2026-05-21 17:44:04 +00:00
test_review_styles_store.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_reviewer.py feat: upgrade default agent + reviewer model to Opus 4.8 (#1350) 2026-05-28 10:59:29 -07:00
test_reviewer_diff.py fix: fetch PR diff via GitHub API to re-enable add_finding validation (#1339) 2026-05-27 17:03:33 +00:00
test_reviewer_eval_run.py feat: upgrade default agent + reviewer model to Opus 4.8 (#1350) 2026-05-28 10:59:29 -07:00
test_reviewer_eval_target.py feat: upgrade default agent + reviewer model to Opus 4.8 (#1350) 2026-05-28 10:59:29 -07:00
test_reviewer_findings.py feat: let reviewer set comment titles (#1356) 2026-05-28 16:04:38 -07:00
test_reviewer_outcomes.py feat: outcomes dataset + bootstrap/continual split via skills (#1365) 2026-06-01 13:25:12 -07:00
test_reviewer_publish.py feat: let reviewer set comment titles (#1356) 2026-05-28 16:04:38 -07:00
test_reviewer_reconcile.py feat: let reviewer set comment titles (#1356) 2026-05-28 16:04:38 -07:00
test_reviewer_tools.py feat: let reviewer set comment titles (#1356) 2026-05-28 16:04:38 -07:00
test_reviewer_watch.py Remove reviewer env allowlist (#1353) 2026-05-28 14:29:33 -07:00
test_sandbox_paths.py fix: better custom backend support (#1071) 2026-03-17 11:55:36 -07:00
test_sanitize_thinking_blocks.py fix: sanitize malformed Anthropic thinking blocks (#1357) 2026-05-28 16:15:13 -07:00
test_sanitize_tool_inputs.py fix: coerce malformed integer strings in read_file offset/limit params (#1216) 2026-05-01 14:29:48 -07:00
test_slack_assistants_status.py fix: propagate Slack reply errors (#1358) 2026-05-28 23:11:42 +00:00
test_slack_context.py fix: propagate Slack reply errors (#1358) 2026-05-28 23:11:42 +00:00
test_slack_feedback.py feat: add Slack reaction feedback to LangSmith (#1231) 2026-05-08 14:24:12 -07:00
test_slack_thread_reply_tool.py fix: propagate Slack reply errors (#1358) 2026-05-28 23:11:42 +00:00
test_stale_sandbox_creating.py feat: outcomes dataset + bootstrap/continual split via skills (#1365) 2026-06-01 13:25:12 -07:00