open-swe/tests/e2e/e2e_env.py

95 lines
3.7 KiB
Python
Raw Normal View History

test(open-swe): add Playwright E2E for the Slack → PR → web handoff (#1583) * test(open-swe): add Playwright E2E for the Slack → PR → web handoff Local, secrets-free end-to-end suite that drives the full happy path through mock Slack/GitHub control panels and the real dashboard UI. Only the LLM and external SaaS HTTP boundaries (GitHub/Slack APIs, OAuth token mint) are faked — the real process_slack_mention, get_agent, deepagents loop, tools, middleware, and dashboard authorization all run under `langgraph dev` with a scripted fake chat model and a local temp-dir sandbox. - full_flow: a Slack mention runs the agent, which implements a change in the sandbox, opens a PR against a fake GitHub remote, and replies with the PR link in the same thread. - dashboard: clicking the bot's real "Open in Web" link loads the built ui/ app (served same-origin); the thread owner can continue the conversation, while a different user sees the same thread read-only (no composer). Wired into Agent CI as a `Playwright E2E` job that runs on pull requests. * fix(open-swe): serve E2E UI assets via explicit route; pin Playwright The dashboard E2E served the built ui/ SPA's /assets via app.mount(StaticFiles), but LangGraph's custom-app loader serves APIRoutes and drops sub-app Mounts, so /assets 404'd under `langgraph dev` in CI — the React app never booted and the composer/transcript never rendered. Serve assets via an explicit route instead. Also pin @playwright/test to the latest (1.61.0) for reproducible runs, and make the owner composer assertion tolerant of either hydration state. * test(open-swe): record Playwright trace + video on every E2E run Capture a replayable trace (DOM snapshots, network, console, source) and a screen recording for every test, not just retries, plus a screenshot on failure. The CI job already uploads playwright-report/ and test-results/, so each run now has a downloadable replay; documented how to open it.
2026-06-22 15:54:46 -04:00
"""Shared environment + constants for the full-flow E2E.
Imported FIRST by both the agent graph entrypoint and the HTTP harness, before
refactor: split webapp.py into api/ + per-source webhook routes Plan step C4 (docs/upstream-sync/domain-reorg/reorg-build-plan.md, approved decisions 1-2): split the 2,590-line agent/webapp.py monolith into agent/webhooks/common.py (shared verify/dispatch helpers), agent/api/app.py (composition), agent/api/health.py (/health + /webhooks/run-complete), and per-source {github,linear,slack,jira,confluence}_routes.py. Atlassian Connect lifecycle + descriptor routes (/connect/*) fold into confluence_routes.py; webapp.py becomes the upstream-shaped compatibility shim (from .api.app import app). langgraph.json http.app stays agent.webapp:app via the shim. Fork content, upstream layout: linear/slack route files verified content-identical to upstream 8356eb34 and taken verbatim; github_routes is upstream + the fork's CI auto-fix trigger wiring; jira/confluence routes are fork-only, transformed to the same common.X / service.X module-attribute style. All signature verification (GitHub HMAC, Slack, Linear timestamp-freshness, verify_jira_secret + opt-in HMAC/timestamp/IP allowlist, Connect JWT/qsh), token-attribution gating, TID-COLLIDE-01 repo binding, _is_repo_auto_review_enabled gates, and public-repo org gate move unchanged. Handlers rewired from webapp.X to common.X; test monkeypatch sites across 26 files + conftest.py + e2e/harness.py retargeted to webhook_common/handler/route modules per upstream's pattern. Residual agent.webapp importers: only the shim, langgraph.json http.app, Makefile uvicorn target, and docs (doc-path updates land in C7). Gates: ruff check + format, pytest --co, full unit (1637 passed), full Playwright E2E vs real langgraph dev (9/9), residual-importer sweep.
2026-07-17 14:30:05 -04:00
any ``agent.*`` module — several webhook/auth/slack constants are read into module
test(open-swe): add Playwright E2E for the Slack → PR → web handoff (#1583) * test(open-swe): add Playwright E2E for the Slack → PR → web handoff Local, secrets-free end-to-end suite that drives the full happy path through mock Slack/GitHub control panels and the real dashboard UI. Only the LLM and external SaaS HTTP boundaries (GitHub/Slack APIs, OAuth token mint) are faked — the real process_slack_mention, get_agent, deepagents loop, tools, middleware, and dashboard authorization all run under `langgraph dev` with a scripted fake chat model and a local temp-dir sandbox. - full_flow: a Slack mention runs the agent, which implements a change in the sandbox, opens a PR against a fake GitHub remote, and replies with the PR link in the same thread. - dashboard: clicking the bot's real "Open in Web" link loads the built ui/ app (served same-origin); the thread owner can continue the conversation, while a different user sees the same thread read-only (no composer). Wired into Agent CI as a `Playwright E2E` job that runs on pull requests. * fix(open-swe): serve E2E UI assets via explicit route; pin Playwright The dashboard E2E served the built ui/ SPA's /assets via app.mount(StaticFiles), but LangGraph's custom-app loader serves APIRoutes and drops sub-app Mounts, so /assets 404'd under `langgraph dev` in CI — the React app never booted and the composer/transcript never rendered. Serve assets via an explicit route instead. Also pin @playwright/test to the latest (1.61.0) for reproducible runs, and make the owner composer assertion tolerant of either hydration state. * test(open-swe): record Playwright trace + video on every E2E run Capture a replayable trace (DOM snapshots, network, console, source) and a screen recording for every test, not just retries, plus a screenshot on failure. The CI job already uploads playwright-report/ and test-results/, so each run now has a downloadable replay; documented how to open it.
2026-06-22 15:54:46 -04:00
globals at import time, so the env must be set beforehand.
Everything here only configures *boundaries* (which sandbox, which fake API
URLs, where git writes its global config). The agent code itself is unchanged.
"""
from __future__ import annotations
import os
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
TMP = Path(os.environ.setdefault("E2E_TMP", str(Path(__file__).parent / ".e2e-tmp")))
# Demo repo the fake GitHub serves and the agent operates on.
OWNER = "fakeorg"
REPO = "demo"
BASE_BRANCH = "main"
FEATURE_BRANCH = "add-greet"
PR_TITLE = "Add greet() helper"
FEATURE_FILE = "greet.py"
# Fixed Slack identifiers so the mock UI and assertions are deterministic.
BOT_USER_ID = "U0BOT"
BOT_USERNAME = "open-swe"
DEMO_CHANNEL = "C_DEMO"
HUMAN_USER = "U_HUMAN"
PORT = os.environ.setdefault("E2E_PORT", "2024")
BASE_URL = os.environ.setdefault("E2E_BASE", f"http://127.0.0.1:{PORT}")
_GH_DIR = TMP / "github"
_WORK_DIR = TMP / "work"
BARE_REMOTE = _GH_DIR / f"{OWNER}__{REPO}.git"
_DEFAULTS = {
# Sandbox: real local provider, rooted in a throwaway temp dir.
"SANDBOX_TYPE": "local",
"LOCAL_SANDBOX_ROOT_DIR": str(_WORK_DIR),
# Keep git's --global writes (bot identity) out of the user's ~/.gitconfig.
"GIT_CONFIG_GLOBAL": str(TMP / "gitconfig-global"),
"GIT_CONFIG_SYSTEM": "/dev/null",
# Path the scripted agent clones from (a local bare repo = "fake GitHub").
"E2E_REMOTE": str(BARE_REMOTE),
# Webhook signing + bot identity.
"GITHUB_WEBHOOK_SECRET": "test-github-secret",
"SLACK_SIGNING_SECRET": "test-slack-secret",
"SLACK_BOT_TOKEN": "xoxb-test-token",
"SLACK_BOT_USER_ID": BOT_USER_ID,
"SLACK_BOT_USERNAME": BOT_USERNAME,
# Slack runs resolve the repo from this when the channel/thread carry none.
"DEFAULT_REPO_OWNER": OWNER,
"DEFAULT_REPO_NAME": REPO,
# Bot-token-only mode: lets Slack runs proceed without a per-user OAuth token.
"LANGSMITH_API_KEY_PROD": "test-bot-mode",
# SDK client target (same dev server).
"LANGGRAPH_URL": BASE_URL,
# Dashboard: the "Open in Web" link target + session-cookie signing. Use
# 127.0.0.1 (not localhost) so the local-dev LLM-key check stays skipped.
"DASHBOARD_BASE_URL": BASE_URL,
"DASHBOARD_API_BASE_URL": BASE_URL,
"DASHBOARD_ALLOWED_ORIGINS": BASE_URL,
"DASHBOARD_JWT_SECRET": "test-dashboard-jwt-secret",
}
feat: plan mode with model-driven entry and collaborative review (#1580) * feat: add plan mode for read-only research and planning Adds a per-run plan_mode flag that puts the agent in a read-only research phase: a strong prompt section is injected and mutating tools are stripped via ExcludeToolsMiddleware so the agent proposes a reviewable implementation plan before any edits. Surfaced in the dashboard UI with a Plan toggle (Shift+Tab) wired through the thread API. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> * fix: enforce plan-mode read-only at tool layer and disable subagents Addresses PR review: plan mode previously relied on prompt text to keep the shell read-only and left the task subagent (built with its own write/PR/Linear tools) unrestricted. Now `task` is excluded so research cannot be delegated to a mutating subagent, and a new PlanModeShellGuardMiddleware enforces a read-only command allowlist on `execute`, blocking writes, git state changes, installs, redirection, and command substitution regardless of model/prompt-injection compliance. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> * fix: harden plan-mode shell guard against wrapped mutations Block git global options that take values (-C, --git-dir, ...) from being misread as the subcommand, reject config-injection options (-c, --config-env, --exec-path), and drop the env command wrapper that could run arbitrary commands. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> * feat: add plan mode with enter_plan_mode tool, profile/team defaults, Slack commands and approval flow - enter_plan_mode tool: agent self-activates plan mode via Command(update={'plan_mode': True}) - Plan mode resolution: per-thread > profile default > team default > False - PLAN_MODE_GUIDANCE_SECTION: always-present prompt section telling agent about the tool - profile_plan_mode_default and team plan_mode_default settings - Slack plan on/off/status commands with thread metadata persistence - slack_thread_reply plan_approval=True renders Approve/Revise/Cancel buttons - Interactivity handler: approve triggers implementation run, cancel posts confirmation - Frontend: plan_mode_default in Profile/ProfileUpdate/TeamSettings types and UI toggles Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> * test: add tests for enter_plan_mode tool, profile/team defaults, Slack plan commands, approval blocks Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> * refactor(plan-mode): drop shell guard, rely on prompt for read-only discipline Remove PlanModeShellGuardMiddleware and its enforcement of read-only shell commands during plan mode. Plan mode now relies on the system prompt to instruct the agent not to run mutating commands; the mutating-tool exclusion (ExcludeToolsMiddleware) is retained. * test(open-swe): add Playwright E2E for the Slack → PR → web handoff Local, secrets-free end-to-end suite that drives the full happy path through mock Slack/GitHub control panels and the real dashboard UI. Only the LLM and external SaaS HTTP boundaries (GitHub/Slack APIs, OAuth token mint) are faked — the real process_slack_mention, get_agent, deepagents loop, tools, middleware, and dashboard authorization all run under `langgraph dev` with a scripted fake chat model and a local temp-dir sandbox. - full_flow: a Slack mention runs the agent, which implements a change in the sandbox, opens a PR against a fake GitHub remote, and replies with the PR link in the same thread. - dashboard: clicking the bot's real "Open in Web" link loads the built ui/ app (served same-origin); the thread owner can continue the conversation, while a different user sees the same thread read-only (no composer). Wired into Agent CI as a `Playwright E2E` job that runs on pull requests. * fix(open-swe): serve E2E UI assets via explicit route; pin Playwright The dashboard E2E served the built ui/ SPA's /assets via app.mount(StaticFiles), but LangGraph's custom-app loader serves APIRoutes and drops sub-app Mounts, so /assets 404'd under `langgraph dev` in CI — the React app never booted and the composer/transcript never rendered. Serve assets via an explicit route instead. Also pin @playwright/test to the latest (1.61.0) for reproducible runs, and make the owner composer assertion tolerant of either hydration state. * test(open-swe): record Playwright trace + video on every E2E run Capture a replayable trace (DOM snapshots, network, console, source) and a screen recording for every test, not just retries, plus a screenshot on failure. The CI job already uploads playwright-report/ and test-results/, so each run now has a downloadable replay; documented how to open it. * feat(plan-mode): collaborative plan review with BlockNote + Yjs When the agent enters plan mode it writes the plan as a markdown file in the sandbox (save_plan tool), publishes it, and posts a review link to the source channel. Reviewers open the plan inside the dashboard (under the /agents shell), read it rendered in a BlockNote editor, and leave inline comments synced live over Yjs. Only the thread owner can approve; any reviewer can request changes. On approve/reject the comments are harvested and handed to the agent for the follow-up run; the agent never sees comments mid-review. - agent: enter_plan_mode persists plan state; new save_plan tool; prompt shares the plan-review link. - dashboard: Yjs WebSocket collab server (pycrdt-websocket) with store-backed snapshots; plan content/status store; plan REST API (get/approve/reject, owner-only approve, client-harvested comments); planStatus on thread summaries. - ui: BlockNote native comments (CommentsExtension + YjsThreadStore) plan page mounted under the agents shell, with a "Review plan" banner in the thread view and a back-link; theme-aware (dark mode) using the dashboard tokens. - e2e: Playwright coverage of the full Slack -> plan -> review -> approve -> PR flow, including cross-user comment sync and owner-only approval. * fix(plan-mode): address review feedback (authz, overrides, leaks, deps) - plan-collab WS: authorize per-thread before joining a room (same read gate as the REST API) — previously any logged-in user could join any thread (IDOR). - plan-collab: tie the snapshot flusher to active connections (refcount) so each opened plan no longer leaks a permanent 1.5s task on the shared event loop. - plan decisions: include thread_id in the follow-up run configurable so the run resumes the existing thread; set plan_mode explicitly so approve forces it off. - get_agent: an explicit per-thread plan_mode (Slack `plan off`, approved plan, dashboard toggle) now overrides profile/team defaults instead of falling back. - plan mode tool gating moved to a state-aware PlanModeMiddleware installed unconditionally, so a mid-run enter_plan_mode restricts the next model turn; before_agent resets stale plan_mode so a later run isn't forced back into it. - exclude write-capable http_request from plan mode. - pin pycrdt / pycrdt-websocket with upper bounds. Includes the latest base (#1583): E2E UI assets served via explicit route (fixes the Playwright CI failure — LangGraph's app loader drops sub-app mounts). * style: ruff format plan_collab.py * fix(plan-mode): owner-gate Slack approval + same-origin check on collab WS - Slack "Approve & Implement" now verifies the clicking user is the plan requester (owner, via the stored triggering_user_id) before implementing — matching the dashboard API's owner-only approval. Non-owners are pointed to Revise / feedback. - The plan-collab WebSocket validates the handshake Origin against the dashboard allowlist before accept() (no-op when unconfigured, e.g. local/dev), mirroring the REST require_same_origin CSRF defense. * fix(plan-mode): enter plan mode only via the model + local mock dev harness Plan mode is now entered solely when the model calls enter_plan_mode. Removed the per-user and team plan_mode_default settings (backend + UI) and the Slack `plan on/off/status` toggle. - enter_plan_mode returns a terminating ToolMessage, fixing the missing ToolMessage error that silently dropped plan mode mid-run. - PlanReview: defer Yjs provider/doc teardown so React StrictMode's dev remount doesn't destroy and then reuse the collaboration provider. - e2e plan_review spec asserts plan_mode actually engages. - LangSmith trace-url resolution is best-effort: bail before any API call when the tenant is unset, cache failures, log at debug. - Add `pnpm run dev:mock`: same-origin Vite HMR harness with a real LLM, Alice/Bob mock users, and a GitHub login picker. * docs(plan-mode): drop stale references to removed profile/team defaults The plan_mode middleware docstring and the approve/reject dispatch comment still described the profile/team plan_mode_default resolution that no longer exists; reword to match model-driven entry + the per-thread carry. * feat(plan-mode): let any reviewer edit the plan, not just comment Drop the owner/commenter split for the plan document: everyone with read access edits and comments alike (DefaultThreadStoreAuth "editor" for all, editor always editable until a decision, anyone seeds the empty doc). This matches the collab WS, which already relays frames to every readable user. Plan approval stays owner-gated. * test(plan-mode): assert plan-mode entry via the tool's success message plan_mode lives only in run state for tool gating; it is not a persisted thread-state channel, so the previous `values.plan_mode === true` poll could never pass. Assert instead that enter_plan_mode's success ToolMessage ("Plan mode is active …") lands in the thread — which only happens when the tool's Command applies cleanly, the exact regression this guards. --------- Co-authored-by: Johannes du Plessis <51395795+johannes117@users.noreply.github.com> Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
2026-06-23 15:06:58 -04:00
# Named test users (the Slack sender dropdown + the dashboard login picker, and
# the identities the automated tests log in as). Each maps a Slack sender id to
# a dashboard login with a matching email, so the Slack thread's owner (resolved
# by email) is the same person when they sign in. The first (Alice) is the
# default Slack sender, hence the default thread owner.
TEST_USERS = [
{"name": "Alice", "slack_id": "U_ALICE", "login": "alice", "email": "alice@example.com"},
{"name": "Bob", "slack_id": "U_BOB", "login": "bob", "email": "bob@example.com"},
]
# The default Slack sender / thread owner; a session with this email may continue
# the thread on the web. Any other logged-in user is read-only.
SAME_USER = {"login": TEST_USERS[0]["login"], "email": TEST_USERS[0]["email"]}
OTHER_USER = {"login": TEST_USERS[1]["login"], "email": TEST_USERS[1]["email"]}
test(open-swe): add Playwright E2E for the Slack → PR → web handoff (#1583) * test(open-swe): add Playwright E2E for the Slack → PR → web handoff Local, secrets-free end-to-end suite that drives the full happy path through mock Slack/GitHub control panels and the real dashboard UI. Only the LLM and external SaaS HTTP boundaries (GitHub/Slack APIs, OAuth token mint) are faked — the real process_slack_mention, get_agent, deepagents loop, tools, middleware, and dashboard authorization all run under `langgraph dev` with a scripted fake chat model and a local temp-dir sandbox. - full_flow: a Slack mention runs the agent, which implements a change in the sandbox, opens a PR against a fake GitHub remote, and replies with the PR link in the same thread. - dashboard: clicking the bot's real "Open in Web" link loads the built ui/ app (served same-origin); the thread owner can continue the conversation, while a different user sees the same thread read-only (no composer). Wired into Agent CI as a `Playwright E2E` job that runs on pull requests. * fix(open-swe): serve E2E UI assets via explicit route; pin Playwright The dashboard E2E served the built ui/ SPA's /assets via app.mount(StaticFiles), but LangGraph's custom-app loader serves APIRoutes and drops sub-app Mounts, so /assets 404'd under `langgraph dev` in CI — the React app never booted and the composer/transcript never rendered. Serve assets via an explicit route instead. Also pin @playwright/test to the latest (1.61.0) for reproducible runs, and make the owner composer assertion tolerant of either hydration state. * test(open-swe): record Playwright trace + video on every E2E run Capture a replayable trace (DOM snapshots, network, console, source) and a screen recording for every test, not just retries, plus a screenshot on failure. The CI job already uploads playwright-report/ and test-results/, so each run now has a downloadable replay; documented how to open it.
2026-06-22 15:54:46 -04:00
for _k, _v in _DEFAULTS.items():
os.environ.setdefault(_k, _v)
os.environ["CONFIGURED_ADMINS"] = "alice,alice@example.com"
test(open-swe): add Playwright E2E for the Slack → PR → web handoff (#1583) * test(open-swe): add Playwright E2E for the Slack → PR → web handoff Local, secrets-free end-to-end suite that drives the full happy path through mock Slack/GitHub control panels and the real dashboard UI. Only the LLM and external SaaS HTTP boundaries (GitHub/Slack APIs, OAuth token mint) are faked — the real process_slack_mention, get_agent, deepagents loop, tools, middleware, and dashboard authorization all run under `langgraph dev` with a scripted fake chat model and a local temp-dir sandbox. - full_flow: a Slack mention runs the agent, which implements a change in the sandbox, opens a PR against a fake GitHub remote, and replies with the PR link in the same thread. - dashboard: clicking the bot's real "Open in Web" link loads the built ui/ app (served same-origin); the thread owner can continue the conversation, while a different user sees the same thread read-only (no composer). Wired into Agent CI as a `Playwright E2E` job that runs on pull requests. * fix(open-swe): serve E2E UI assets via explicit route; pin Playwright The dashboard E2E served the built ui/ SPA's /assets via app.mount(StaticFiles), but LangGraph's custom-app loader serves APIRoutes and drops sub-app Mounts, so /assets 404'd under `langgraph dev` in CI — the React app never booted and the composer/transcript never rendered. Serve assets via an explicit route instead. Also pin @playwright/test to the latest (1.61.0) for reproducible runs, and make the owner composer assertion tolerant of either hydration state. * test(open-swe): record Playwright trace + video on every E2E run Capture a replayable trace (DOM snapshots, network, console, source) and a screen recording for every test, not just retries, plus a screenshot on failure. The CI job already uploads playwright-report/ and test-results/, so each run now has a downloadable replay; documented how to open it.
2026-06-22 15:54:46 -04:00
for _d in (TMP, _GH_DIR, _WORK_DIR):
_d.mkdir(parents=True, exist_ok=True)
FAKE_GITHUB_API = f"{BASE_URL}/fake-gh"
FAKE_SLACK_API = f"{BASE_URL}/fake-slack"