mirror of
https://github.com/Sea-Haven-Industries/open-swe.git
synced 2026-09-30 12:43:16 +00:00
ddaa87b97d
* fix: preserve dashboard redirect after login Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> * test: cover plan login redirect in e2e Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> --------- Co-authored-by: Ramon Nogueira <270434257+ramon-langchain@users.noreply.github.com> Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> (cherry picked from commit bc7ce59169b5350da7286164afb83a7b037b528d)
96 lines
5.1 KiB
Markdown
96 lines
5.1 KiB
Markdown
# Playwright E2E — the full Slack → implement → PR → reply flow
|
|
|
|
This drives the **whole happy path** through two mock UIs:
|
|
|
|
1. A user asks Open SWE to implement something in a **mock Slack** thread.
|
|
2. The **real agent** runs (via `langgraph dev`): it implements the change in a
|
|
**local temp-dir sandbox**, pushes a branch, and opens a PR on a **fake GitHub**.
|
|
3. It posts the PR link back to the **same Slack thread** — visible in the mock UI.
|
|
|
|
## What is faked vs. real
|
|
|
|
Only the **LLM** and the **external SaaS HTTP boundaries** are faked. All agent
|
|
code runs for real.
|
|
|
|
| Piece | Real or fake |
|
|
| ---------------------------------------------------------------- | -------------------------------------------------------------------------- |
|
|
| Slack webhook → `process_slack_mention` → run dispatch | **real** (`agent.webapp`) |
|
|
| `get_agent`, deepagents loop, tools, middleware, prompt | **real** |
|
|
| `open_pull_request`, `slack_thread_reply` tools | **real** |
|
|
| Sandbox | **real** `local` provider, rooted in a throwaway temp dir |
|
|
| Git remote ("GitHub") | **real git**, a local bare repo the agent clones/pushes |
|
|
| The LLM | **fake** — a scripted model (`fake_llm.py`) emitting a fixed tool sequence |
|
|
| `api.github.com` REST (PR create) + dashboard GitHub OAuth login | **fake** (`/fake-gh/...`), state rendered at `/mock/github` |
|
|
| `slack.com/api` (post message, etc.) | **fake** (`/fake-slack/...`), thread rendered at `/mock/slack` |
|
|
| GitHub App token mint, `api.github.com/user` identity | stubbed (offline) |
|
|
|
|
The fake GitHub/Slack stores are the single source of truth the mock UIs render,
|
|
so what Playwright asserts on is exactly what the real agent produced.
|
|
|
|
## Files
|
|
|
|
- `e2e_env.py` — env + constants set before any `agent.*` import (sandbox=local,
|
|
fake API URLs, isolated `GIT_CONFIG_GLOBAL`, bot-token-only mode).
|
|
- `fake_llm.py` — the scripted `BaseChatModel` (the only faked agent piece).
|
|
- `patches.py` — monkeypatches the boundaries (LLM, GitHub/Slack URLs, token mint).
|
|
- `agent_entrypoint.py` — langgraph `agent` graph: applies patches, re-exports the
|
|
real `traced_agent`.
|
|
- `harness.py` — langgraph `http.app`: the real `agent.webapp` plus the fake
|
|
GitHub/Slack APIs, the mock UIs, and the control/compose endpoints.
|
|
- `fakes.py` — in-memory PR/Slack stores + git seeding of the bare remote.
|
|
- `langgraph.e2e.json` — dev-server config pointing at the two entrypoints above.
|
|
- `static/{slack,github}.html` — the mock Slack/GitHub UIs (external SaaS we can't
|
|
run locally). The dashboard is **not** mocked — it's the real `ui/` app.
|
|
- `global-setup.ts` — builds the real `ui/` SPA (once) so the harness can serve it.
|
|
|
|
## The dashboard — the real `ui/` app
|
|
|
|
The dashboard is **not** mocked. The bot's "Open in Web" link
|
|
(`DASHBOARD_BASE_URL/agents/{thread_id}`) loads the **actual built `ui/` React
|
|
app** — served same-origin from the harness so the session cookie and
|
|
`/dashboard/api/*` calls work without CORS. The signed session cookie is real
|
|
(minted via `/control/login`), so per-user authorization is genuine; the only
|
|
extra fake is the OAuth-token store (an external credential).
|
|
|
|
The UI is built by `global-setup.ts` with `VITE_DASHBOARD_API_BASE_URL` pointed at
|
|
the harness. It builds once; set `E2E_FORCE_UI_BUILD=1` to rebuild (e.g. after a
|
|
UI change or port change). Requires Corepack with `pnpm` enabled.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
cd tests/e2e
|
|
npm install
|
|
npx playwright install chromium
|
|
npx playwright test # boots langgraph dev automatically, then runs
|
|
```
|
|
|
|
Watch it in human time:
|
|
|
|
```bash
|
|
SLOW_MO=700 npx playwright test --headed
|
|
```
|
|
|
|
## Artifacts (replay a run)
|
|
|
|
Every test records a **trace** (DOM-snapshot timeline + network + console + source)
|
|
and a **video**; failures also get a screenshot. Locally they land in
|
|
`test-results/<test>/` and are embedded in `playwright-report/`:
|
|
|
|
```bash
|
|
npx playwright show-report # browse runs; each has a Trace tab
|
|
npx playwright show-trace test-results/<test>/trace.zip # open one trace directly
|
|
```
|
|
|
|
In CI the `Playwright E2E` job uploads both `playwright-report/` and
|
|
`test-results/` as the **playwright-report** artifact on the run. Download it,
|
|
then `npx playwright show-report <unzipped-dir>` (or drag a `trace.zip` onto
|
|
<https://trace.playwright.dev>) to replay.
|
|
|
|
Poke at it by hand (from the repo root):
|
|
|
|
```bash
|
|
uv run langgraph dev --config tests/e2e/langgraph.e2e.json --port 2024 \
|
|
--no-browser --allow-blocking --no-reload
|
|
# open http://127.0.0.1:2024/mock/slack and /mock/github
|
|
```
|