open-swe/tests
Ramon Nogueira 3a0e2b4672
feat: plan mode with model-driven entry and collaborative review (#1580)
* feat: add plan mode for read-only research and planning

Adds a per-run plan_mode flag that puts the agent in a read-only
research phase: a strong prompt section is injected and mutating tools
are stripped via ExcludeToolsMiddleware so the agent proposes a
reviewable implementation plan before any edits. Surfaced in the
dashboard UI with a Plan toggle (Shift+Tab) wired through the thread API.

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

* fix: enforce plan-mode read-only at tool layer and disable subagents

Addresses PR review: plan mode previously relied on prompt text to keep
the shell read-only and left the task subagent (built with its own
write/PR/Linear tools) unrestricted. Now `task` is excluded so research
cannot be delegated to a mutating subagent, and a new
PlanModeShellGuardMiddleware enforces a read-only command allowlist on
`execute`, blocking writes, git state changes, installs, redirection,
and command substitution regardless of model/prompt-injection compliance.

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

* fix: harden plan-mode shell guard against wrapped mutations

Block git global options that take values (-C, --git-dir, ...) from being
misread as the subcommand, reject config-injection options (-c,
--config-env, --exec-path), and drop the env command wrapper that could
run arbitrary commands.

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

* feat: add plan mode with enter_plan_mode tool, profile/team defaults, Slack commands and approval flow

- enter_plan_mode tool: agent self-activates plan mode via Command(update={'plan_mode': True})
- Plan mode resolution: per-thread > profile default > team default > False
- PLAN_MODE_GUIDANCE_SECTION: always-present prompt section telling agent about the tool
- profile_plan_mode_default and team plan_mode_default settings
- Slack plan on/off/status commands with thread metadata persistence
- slack_thread_reply plan_approval=True renders Approve/Revise/Cancel buttons
- Interactivity handler: approve triggers implementation run, cancel posts confirmation
- Frontend: plan_mode_default in Profile/ProfileUpdate/TeamSettings types and UI toggles

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

* test: add tests for enter_plan_mode tool, profile/team defaults, Slack plan commands, approval blocks

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

* refactor(plan-mode): drop shell guard, rely on prompt for read-only discipline

Remove PlanModeShellGuardMiddleware and its enforcement of read-only shell
commands during plan mode. Plan mode now relies on the system prompt to
instruct the agent not to run mutating commands; the mutating-tool exclusion
(ExcludeToolsMiddleware) is retained.

* test(open-swe): add Playwright E2E for the Slack → PR → web handoff

Local, secrets-free end-to-end suite that drives the full happy path through mock Slack/GitHub control panels and the real dashboard UI. Only the LLM and external SaaS HTTP boundaries (GitHub/Slack APIs, OAuth token mint) are faked — the real process_slack_mention, get_agent, deepagents loop, tools, middleware, and dashboard authorization all run under `langgraph dev` with a scripted fake chat model and a local temp-dir sandbox.

- full_flow: a Slack mention runs the agent, which implements a change in the sandbox, opens a PR against a fake GitHub remote, and replies with the PR link in the same thread.
- dashboard: clicking the bot's real "Open in Web" link loads the built ui/ app (served same-origin); the thread owner can continue the conversation, while a different user sees the same thread read-only (no composer).

Wired into Agent CI as a `Playwright E2E` job that runs on pull requests.

* fix(open-swe): serve E2E UI assets via explicit route; pin Playwright

The dashboard E2E served the built ui/ SPA's /assets via app.mount(StaticFiles), but LangGraph's custom-app loader serves APIRoutes and drops sub-app Mounts, so /assets 404'd under `langgraph dev` in CI — the React app never booted and the composer/transcript never rendered. Serve assets via an explicit route instead.

Also pin @playwright/test to the latest (1.61.0) for reproducible runs, and make the owner composer assertion tolerant of either hydration state.

* test(open-swe): record Playwright trace + video on every E2E run

Capture a replayable trace (DOM snapshots, network, console, source) and a screen recording for every test, not just retries, plus a screenshot on failure. The CI job already uploads playwright-report/ and test-results/, so each run now has a downloadable replay; documented how to open it.

* feat(plan-mode): collaborative plan review with BlockNote + Yjs

When the agent enters plan mode it writes the plan as a markdown file in the
sandbox (save_plan tool), publishes it, and posts a review link to the source
channel. Reviewers open the plan inside the dashboard (under the /agents shell),
read it rendered in a BlockNote editor, and leave inline comments synced live
over Yjs. Only the thread owner can approve; any reviewer can request changes.
On approve/reject the comments are harvested and handed to the agent for the
follow-up run; the agent never sees comments mid-review.

- agent: enter_plan_mode persists plan state; new save_plan tool; prompt shares
  the plan-review link.
- dashboard: Yjs WebSocket collab server (pycrdt-websocket) with store-backed
  snapshots; plan content/status store; plan REST API (get/approve/reject,
  owner-only approve, client-harvested comments); planStatus on thread summaries.
- ui: BlockNote native comments (CommentsExtension + YjsThreadStore) plan page
  mounted under the agents shell, with a "Review plan" banner in the thread view
  and a back-link; theme-aware (dark mode) using the dashboard tokens.
- e2e: Playwright coverage of the full Slack -> plan -> review -> approve -> PR
  flow, including cross-user comment sync and owner-only approval.

* fix(plan-mode): address review feedback (authz, overrides, leaks, deps)

- plan-collab WS: authorize per-thread before joining a room (same read gate as
  the REST API) — previously any logged-in user could join any thread (IDOR).
- plan-collab: tie the snapshot flusher to active connections (refcount) so each
  opened plan no longer leaks a permanent 1.5s task on the shared event loop.
- plan decisions: include thread_id in the follow-up run configurable so the run
  resumes the existing thread; set plan_mode explicitly so approve forces it off.
- get_agent: an explicit per-thread plan_mode (Slack `plan off`, approved plan,
  dashboard toggle) now overrides profile/team defaults instead of falling back.
- plan mode tool gating moved to a state-aware PlanModeMiddleware installed
  unconditionally, so a mid-run enter_plan_mode restricts the next model turn;
  before_agent resets stale plan_mode so a later run isn't forced back into it.
- exclude write-capable http_request from plan mode.
- pin pycrdt / pycrdt-websocket with upper bounds.

Includes the latest base (#1583): E2E UI assets served via explicit route
(fixes the Playwright CI failure — LangGraph's app loader drops sub-app mounts).

* style: ruff format plan_collab.py

* fix(plan-mode): owner-gate Slack approval + same-origin check on collab WS

- Slack "Approve & Implement" now verifies the clicking user is the plan
  requester (owner, via the stored triggering_user_id) before implementing —
  matching the dashboard API's owner-only approval. Non-owners are pointed to
  Revise / feedback.
- The plan-collab WebSocket validates the handshake Origin against the dashboard
  allowlist before accept() (no-op when unconfigured, e.g. local/dev), mirroring
  the REST require_same_origin CSRF defense.

* fix(plan-mode): enter plan mode only via the model + local mock dev harness

Plan mode is now entered solely when the model calls enter_plan_mode.
Removed the per-user and team plan_mode_default settings (backend + UI)
and the Slack `plan on/off/status` toggle.

- enter_plan_mode returns a terminating ToolMessage, fixing the missing
  ToolMessage error that silently dropped plan mode mid-run.
- PlanReview: defer Yjs provider/doc teardown so React StrictMode's dev
  remount doesn't destroy and then reuse the collaboration provider.
- e2e plan_review spec asserts plan_mode actually engages.
- LangSmith trace-url resolution is best-effort: bail before any API
  call when the tenant is unset, cache failures, log at debug.
- Add `pnpm run dev:mock`: same-origin Vite HMR harness with a real LLM,
  Alice/Bob mock users, and a GitHub login picker.

* docs(plan-mode): drop stale references to removed profile/team defaults

The plan_mode middleware docstring and the approve/reject dispatch comment
still described the profile/team plan_mode_default resolution that no longer
exists; reword to match model-driven entry + the per-thread carry.

* feat(plan-mode): let any reviewer edit the plan, not just comment

Drop the owner/commenter split for the plan document: everyone with read
access edits and comments alike (DefaultThreadStoreAuth "editor" for all,
editor always editable until a decision, anyone seeds the empty doc). This
matches the collab WS, which already relays frames to every readable user.
Plan approval stays owner-gated.

* test(plan-mode): assert plan-mode entry via the tool's success message

plan_mode lives only in run state for tool gating; it is not a persisted
thread-state channel, so the previous `values.plan_mode === true` poll
could never pass. Assert instead that enter_plan_mode's success ToolMessage
("Plan mode is active …") lands in the thread — which only happens when the
tool's Command applies cleanly, the exact regression this guards.

---------

Co-authored-by: Johannes du Plessis <51395795+johannes117@users.noreply.github.com>
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-06-23 12:06:58 -07:00
..
e2e feat: plan mode with model-driven entry and collaborative review (#1580) 2026-06-23 12:06:58 -07:00
middleware fix: keep sandbox backend stable across recovery (#1294)w 2026-05-11 16:03:38 -07:00
conftest.py Remove reviewer env allowlist (#1353) 2026-05-28 14:29:33 -07:00
test_account_link.py fix: use Slack OIDC mappings for Slack thread ownership (#1410) 2026-06-04 19:58:30 +00:00
test_agent_instructions.py feat: per-repo custom instructions for the coding agent (#1460) 2026-06-09 15:46:29 -07:00
test_agent_schedules.py feat: add scheduled web agents (#1422) 2026-06-05 02:20:24 +00:00
test_agent_subagent_models.py feat(open-swe): stream agent chat via @langchain/react v2 protocol (#1475) 2026-06-11 09:54:35 -07:00
test_agent_thread_pr_state.py feat: track PR lifecycle state per thread for sidebar (#1492) 2026-06-11 12:21:59 -07:00
test_agent_usage.py Revert out-of-process usage-snapshot builder (#1473) 2026-06-09 13:19:11 -07:00
test_agents_md.py feat: reviewer enforces AGENTS.md/CLAUDE.md repo rules as mandatory pass (#1569) 2026-06-18 11:13:58 -07:00
test_analyzer_cron.py feat: outcomes dataset + bootstrap/continual split via skills (#1365) 2026-06-01 13:25:12 -07:00
test_analyzer_skills.py feat: outcomes dataset + bootstrap/continual split via skills (#1365) 2026-06-01 13:25:12 -07:00
test_anthropic_effort.py feat(open-swe): stream agent chat via @langchain/react v2 protocol (#1475) 2026-06-11 09:54:35 -07:00
test_api_standards_skill.py feat: apply API standards skill in PR reviews for API changes (#1452) 2026-06-08 14:37:04 -07:00
test_auth_sources.py fix: stop storing GitHub tokens in metadata (#1405) 2026-06-04 09:33:51 -07:00
test_authorship.py feat: open Slack-triggered PRs as the triggering user (#1375) 2026-06-02 15:04:20 -07:00
test_autofix_state.py feat: activate PR babysitting UI toggles for autofix and trigger mode (#1561) 2026-06-17 14:12:04 -07:00
test_autofix_webhook.py feat: activate PR babysitting UI toggles for autofix and trigger mode (#1561) 2026-06-17 14:12:04 -07:00
test_check_message_queue.py feat: activate PR babysitting UI toggles for autofix and trigger mode (#1561) 2026-06-17 14:12:04 -07:00
test_ci_autofix.py feat: activate PR babysitting UI toggles for autofix and trigger mode (#1561) 2026-06-17 14:12:04 -07:00
test_corridor_mcp.py feat: Add Corridor MCP analyzePlan integration (#1572) 2026-06-18 14:01:25 -07:00
test_currents_tools.py feat: add user-scoped Currents.dev API key for e2e test investigation (#1566) 2026-06-17 13:59:55 -07:00
test_dashboard_admin.py fix: allow GitHub logins for dashboard admins (#1582) 2026-06-20 09:44:12 -07:00
test_dashboard_csrf.py fix: harden origin parsing and home prompt submit failure (#1499) 2026-06-11 11:54:25 -07:00
test_dashboard_links.py fix: point reviewer "Open in Web" link to the review page (#1519) 2026-06-12 14:15:58 -07:00
test_dashboard_org_login_gate.py fix: Lock dashboard login to GitHub org members (#1367) 2026-06-01 20:56:30 +00:00
test_dashboard_repo_optional.py feat(open-swe): stream agent chat via @langchain/react v2 protocol (#1475) 2026-06-11 09:54:35 -07:00
test_dashboard_repos.py fix: Reduce graph load and dashboard repo failures (#1412) 2026-06-04 13:54:06 -07:00
test_dashboard_reviews.py perf: speed up Reviews list (My PRs + All PRs) (#1518) 2026-06-12 14:14:50 -07:00
test_dashboard_run_email.py fix: use Slack OIDC mappings for Slack thread ownership (#1410) 2026-06-04 19:58:30 +00:00
test_dashboard_thread_api.py feat(open-swe): let any org member post to a thread, with attribution (#1594) 2026-06-23 11:06:53 -07:00
test_dashboard_thread_api_activity.py fix: restore org-wide read access to agent threads (#1474) 2026-06-09 16:41:07 -07:00
test_dashboard_web_handoff.py feat(open-swe): stream agent chat via @langchain/react v2 protocol (#1475) 2026-06-11 09:54:35 -07:00
test_daytona_integration.py fix(daytona): make sandbox snapshot configurable (#1220) 2026-05-01 22:51:34 +00:00
test_encryption.py feat: support TOKEN_ENCRYPTION_KEY rotation via MultiFernet [closes AB-2323] (#1275) 2026-05-08 14:29:23 -07:00
test_eval_jobs.py feat: Run reviewer eval in a GitHub Action; dashboard becomes read-only (#1556) 2026-06-16 19:38:36 -07:00
test_eval_store_reporter.py feat: Run reviewer eval in a GitHub Action; dashboard becomes read-only (#1556) 2026-06-16 19:38:36 -07:00
test_fireworks_model.py Remove Kimi K2.6 from supported model selector (#1522) 2026-06-15 09:39:08 -07:00
test_github_app.py fix: scope public reviewer tokens (#1389) 2026-06-03 09:25:17 -07:00
test_github_checks.py feat: shared GitHub HTTP helper with retries, rate-limit handling [closes OPE-45] (#1565) 2026-06-17 14:45:57 -07:00
test_github_ci.py feat: shared GitHub HTTP helper with retries, rate-limit handling [closes OPE-45] (#1565) 2026-06-17 14:45:57 -07:00
test_github_comment_prompts.py fix: allow agent to pause and ask before adding a dependency (#1578) 2026-06-19 17:04:20 -07:00
test_github_feedback.py feat: reconcile reviewer comment lifecycle (#1332) 2026-05-26 16:24:34 -07:00
test_github_http.py feat: shared GitHub HTTP helper with retries, rate-limit handling [closes OPE-45] (#1565) 2026-06-17 14:45:57 -07:00
test_github_issue_webhook.py feat: surface dashboard UI link on PR reviews [INF-0000] (#1440) 2026-06-07 05:17:09 +00:00
test_github_oauth_refresh.py fix: auto-recover from expired GitHub refresh tokens (#1491) 2026-06-11 12:05:27 -07:00
test_github_proxy_refresh.py fix: refresh sandbox GitHub proxy token before mid-run expiry (#1496) 2026-06-11 10:59:21 -07:00
test_github_token_ttl.py fix: stop storing GitHub tokens in metadata (#1405) 2026-06-04 09:33:51 -07:00
test_google_model.py feat: add Gemini 3.5 Flash provider (#1420) 2026-06-04 17:38:54 -07:00
test_http_security.py fix: harden http_request SSRF guard against DNS rebinding [closes AB-2321] (#1277) 2026-05-08 22:06:25 +00:00
test_langsmith_sandbox_config.py feat: add idle TTL and delete-after-stop sandbox lifecycle controls (#1265) 2026-05-08 00:30:14 -04:00
test_langsmith_sandbox_timeout.py fix: enforce client-side deadline on sandbox execute (#1451) 2026-06-08 11:16:08 -07:00
test_langsmith_trace_url.py feat: plan mode with model-driven entry and collaborative review (#1580) 2026-06-23 12:06:58 -07:00
test_local_integration.py fix: auto-create local sandbox root dir (#1402) 2026-06-03 23:47:08 +00:00
test_model_fallback_middleware.py feat(open-swe): stream agent chat via @langchain/react v2 protocol (#1475) 2026-06-11 09:54:35 -07:00
test_model_fallback_resolution.py feat: upgrade default agent + reviewer model to Opus 4.8 (#1350) 2026-05-28 10:59:29 -07:00
test_multimodal.py feat: handle images sent to non-vision models in Slack, Linear, and web UI (#1560) 2026-06-17 09:12:52 -07:00
test_normalize_repo.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_notify_step_limit_middleware.py fix: notify users via Slack when agent hits model call step limit (#1204) 2026-05-01 14:24:25 -07:00
test_observability_tools.py fix: allow GitHub logins for dashboard admins (#1582) 2026-06-20 09:44:12 -07:00
test_open_pull_request.py feat: link Slack thread/Linear ticket in PRs and append ticket to title (#1504) 2026-06-11 15:52:40 -07:00
test_plan_mode.py feat: plan mode with model-driven entry and collaborative review (#1580) 2026-06-23 12:06:58 -07:00
test_plan_review.py feat: plan mode with model-driven entry and collaborative review (#1580) 2026-06-23 12:06:58 -07:00
test_pr_ready_auto_review.py fix: reviewer resolves GitHub App token at run start, not via cross-process cache (#1409) 2026-06-04 12:11:23 -07:00
test_prompt_default_repo.py fix: make default repository configurable (#1429) 2026-06-05 13:48:47 -07:00
test_proxy_auth.py fix: refresh sandbox GitHub proxy token before mid-run expiry (#1496) 2026-06-11 10:59:21 -07:00
test_public_repo_org_gate.py fix: remove manual review trigger surfaces (#1396) 2026-06-03 12:33:22 -07:00
test_recent_comments.py chore: Drop monorepo (#1029) 2026-03-06 16:10:34 -08:00
test_refresh_slack_status_middleware.py feat: add optional Slack Assistants API typing status indicator (#1269) 2026-05-08 10:21:55 -07:00
test_repo_extraction.py feat: resolve Slack repo from channel topic/purpose (#1453) 2026-06-08 13:53:16 -07:00
test_repo_prep.py fix: reviewer can silently review a stale checkout on reused sandboxes (#1503) 2026-06-11 13:42:10 -07:00
test_review_api.py fix: render GitHub-hosted images in PR descriptions on reviews page (#1589) 2026-06-22 14:01:06 -07:00
test_review_chat.py feat: chat with your PR on the review page (#1534) 2026-06-15 17:17:30 -07:00
test_review_style_collector.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_review_style_sync.py fix: review style prompts UX, stale runs, and OAuth refresh (#1321) 2026-05-21 17:44:04 +00:00
test_review_styles_store.py feat: tune reviewer for precision — web/wiki tools + recalibrated prompt (#1312) 2026-05-20 18:35:00 +00:00
test_reviewer.py chore: install sfw in agent image (#1577) 2026-06-19 15:39:09 -07:00
test_reviewer_diff.py fix: fetch PR diff via GitHub API to re-enable add_finding validation (#1339) 2026-05-27 17:03:33 +00:00
test_reviewer_eval_run.py Reviewer eval admin: configurable runs + stacked form layout (#1540) 2026-06-16 10:12:35 -07:00
test_reviewer_eval_target.py feat: Run reviewer eval in a GitHub Action; dashboard becomes read-only (#1556) 2026-06-16 19:38:36 -07:00
test_reviewer_findings.py fix: honest publish_review reporting + structured thread-not-found errors (#1481) 2026-06-10 10:44:13 -07:00
test_reviewer_groups.py fix: Simplify review explanation: full-width, plain prose, no diff links (#1547) 2026-06-16 15:32:24 -07:00
test_reviewer_outcomes.py feat: outcomes dataset + bootstrap/continual split via skills (#1365) 2026-06-01 13:25:12 -07:00
test_reviewer_publish.py feat: surface sub-threshold findings in review summary with web app link (#1571) 2026-06-18 11:15:42 -07:00
test_reviewer_reconcile.py feat: let reviewer set comment titles (#1356) 2026-05-28 16:04:38 -07:00
test_reviewer_tools.py feat: disable out-of-diff reviewer findings (#1516) 2026-06-12 10:19:30 -07:00
test_reviewer_watch.py fix: keep review check on follow-up commits, drop neutral conclusion (#1486) 2026-06-10 13:33:34 -07:00
test_sandbox_paths.py fix: better custom backend support (#1071) 2026-03-17 11:55:36 -07:00
test_sanitize_thinking_blocks.py fix: sanitize malformed Anthropic thinking blocks (#1357) 2026-05-28 16:15:13 -07:00
test_sanitize_tool_inputs.py fix: coerce malformed integer strings in read_file offset/limit params (#1216) 2026-05-01 14:29:48 -07:00
test_schedule_thread_wakeup.py feat: add schedule_thread_wakeup tool for self-polling (#1592) 2026-06-23 11:06:01 -07:00
test_slack_assistants_status.py feat: add Slack Block Kit reply options (#1407) 2026-06-04 10:26:09 -07:00
test_slack_context.py fix: serialize Slack run dispatch (#1591) 2026-06-23 12:04:08 -07:00
test_slack_feedback.py feat: add Slack reaction feedback to LangSmith (#1231) 2026-05-08 14:24:12 -07:00
test_slack_oauth.py feat: open Slack-triggered PRs as the triggering user (#1375) 2026-06-02 15:04:20 -07:00
test_slack_thread_reply_tool.py feat: add Slack Block Kit reply options (#1407) 2026-06-04 10:26:09 -07:00
test_stale_sandbox_creating.py fix: refresh sandbox GitHub proxy token before mid-run expiry (#1496) 2026-06-11 10:59:21 -07:00
test_team_credentials.py feat: server-side Datadog/LangSmith observability tools + team creds [closes OPE-54] (#1476) 2026-06-10 11:07:42 -07:00
test_team_settings_grouping.py feat: AI-sorted PR review view with diff grouping (#1544) 2026-06-16 13:59:37 -07:00
test_team_settings_org_guidelines.py feat: chat with your PR on the review page (#1534) 2026-06-15 17:17:30 -07:00
test_tool_artifact_middleware.py feat(open-swe): stream agent chat via @langchain/react v2 protocol (#1475) 2026-06-11 09:54:35 -07:00
test_user_credentials.py feat: add user-scoped Currents.dev API key for e2e test investigation (#1566) 2026-06-17 13:59:55 -07:00
test_user_mappings.py fix: use Slack OIDC mappings for Slack thread ownership (#1410) 2026-06-04 19:58:30 +00:00