mirror of
https://github.com/Sea-Haven-Industries/open-swe.git
synced 2026-10-01 09:43:14 +00:00
a8c7af4192
* test(open-swe): add Playwright E2E for the Slack → PR → web handoff Local, secrets-free end-to-end suite that drives the full happy path through mock Slack/GitHub control panels and the real dashboard UI. Only the LLM and external SaaS HTTP boundaries (GitHub/Slack APIs, OAuth token mint) are faked — the real process_slack_mention, get_agent, deepagents loop, tools, middleware, and dashboard authorization all run under `langgraph dev` with a scripted fake chat model and a local temp-dir sandbox. - full_flow: a Slack mention runs the agent, which implements a change in the sandbox, opens a PR against a fake GitHub remote, and replies with the PR link in the same thread. - dashboard: clicking the bot's real "Open in Web" link loads the built ui/ app (served same-origin); the thread owner can continue the conversation, while a different user sees the same thread read-only (no composer). Wired into Agent CI as a `Playwright E2E` job that runs on pull requests. * fix(open-swe): serve E2E UI assets via explicit route; pin Playwright The dashboard E2E served the built ui/ SPA's /assets via app.mount(StaticFiles), but LangGraph's custom-app loader serves APIRoutes and drops sub-app Mounts, so /assets 404'd under `langgraph dev` in CI — the React app never booted and the composer/transcript never rendered. Serve assets via an explicit route instead. Also pin @playwright/test to the latest (1.61.0) for reproducible runs, and make the owner composer assertion tolerant of either hydration state. * test(open-swe): record Playwright trace + video on every E2E run Capture a replayable trace (DOM snapshots, network, console, source) and a screen recording for every test, not just retries, plus a screenshot on failure. The CI job already uploads playwright-report/ and test-results/, so each run now has a downloadable replay; documented how to open it.
53 lines
2.5 KiB
TypeScript
53 lines
2.5 KiB
TypeScript
import { test, expect } from "@playwright/test";
|
|
|
|
// Full happy path, driven entirely through the mock Slack + GitHub UIs:
|
|
// user asks in Slack -> real agent implements in a local sandbox -> opens a PR
|
|
// on the fake GitHub -> replies with the PR link in the SAME Slack thread.
|
|
// Only the LLM is faked; all agent code runs for real via langgraph dev.
|
|
test.describe("Open SWE full flow", () => {
|
|
test.beforeEach(async ({ page }) => {
|
|
await page.goto("/mock/slack");
|
|
await page.locator("#reset").click();
|
|
await expect(page.locator("#thread")).toContainText("No messages yet");
|
|
});
|
|
|
|
test("Slack request → implements → opens PR → links it back in the thread", async ({ page }) => {
|
|
await page.locator("#text").fill("<@U0BOT> please add a greet() helper and open a PR");
|
|
await page.locator("#send").click();
|
|
|
|
// The user's message lands in the thread.
|
|
await expect(page.locator(".msg").filter({ hasText: "add a greet() helper" })).toBeVisible();
|
|
|
|
// The agent replies in the SAME thread with a link to the PR it opened.
|
|
const reply = page.locator(".msg.bot").filter({ hasText: "Add greet() helper" });
|
|
await expect(reply).toBeVisible();
|
|
const prLink = reply.locator('a[href*="/pull/"]');
|
|
await expect(prLink).toBeVisible();
|
|
|
|
// Follow the link → the PR page shows what GitHub would show.
|
|
await prLink.click();
|
|
await expect(page.locator("#pr-title")).toContainText("Add greet() helper");
|
|
await expect(page.locator("#pr-state")).toHaveText("open");
|
|
await expect(page.locator("#pr-head")).toHaveText("add-greet");
|
|
await expect(page.locator('#pr-files li[data-file="greet.py"]')).toBeVisible();
|
|
|
|
// The PR list view shows it too.
|
|
await page.goto("/mock/github");
|
|
await expect(page.locator('.pr[data-pr="1"]')).toContainText("Add greet() helper");
|
|
await expect(page.locator('.pr[data-pr="1"]')).toContainText("greet.py");
|
|
});
|
|
|
|
test("a message that does not mention the bot produces no run and no PR", async ({ page }) => {
|
|
await page.locator("#mention").uncheck();
|
|
await page.locator("#text").fill("just chatting with the team, nothing for the bot");
|
|
await page.locator("#send").click();
|
|
|
|
await expect(page.locator(".msg").filter({ hasText: "just chatting" })).toBeVisible();
|
|
// No agent activity: give the (non-)run a moment, then assert nothing came back.
|
|
await page.waitForTimeout(3000);
|
|
await expect(page.locator(".msg.bot")).toHaveCount(0);
|
|
|
|
const prs = await (await page.request.get("/mock/github/data")).json();
|
|
expect(prs.length).toBe(0);
|
|
});
|
|
});
|