--- title: "Testing E2e" description: "End-to-end testing patterns with Playwright — page objects, AI agent testing, visual regression, accessibility testing with axe-core, and CI integration. Use when writing E2E tests, setting up Playwright, implementing visual regression, or testing accessibility." canonical: "https://orchestkit.yonyon.ai/docs/reference/skills/testing-e2e" --- # Testing E2e End-to-end testing patterns with Playwright — page objects, AI agent testing, visual regression, accessibility testing with axe-core, and CI integration. Use when writing E2E tests, setting up Playwright, implementing visual regression, or testing accessibility. Reference medium > **Auto-activated** — this skill loads automatically when Claude detects matching context. > **Testing E2e** End-to-end testing patterns with Playwright — page objects, AI agent testing, visual regression, accessibility testing with axe-core, and CI integration. Use when writing E2E tests, setting up Playwright, implementing visual regression, or testing accessibility. # E2E Testing Patterns End-to-end testing with Playwright 1.59+, visual regression, accessibility, and AI agent workflows. ## Quick Reference | Category | Rules | Impact | When to Use | |----------|-------|--------|-------------| | [**emulate Backends**](#emulate-backends) | `rules/emulate-e2e.md` | **HIGH** | **FIRST CHOICE — deterministic API backends for E2E** | | [Playwright Core](#playwright-core) | `rules/e2e-playwright.md` | HIGH | Semantic locators, auto-wait, flaky detection | | [Page Objects](#page-objects) | `rules/e2e-page-objects.md` | HIGH | Encapsulate page interactions, visual regression | | [AI Agents](#ai-agents) | `rules/e2e-ai-agents.md` | HIGH | Planner/Generator/Healer, init-agents | | [A11y Playwright](#accessibility-playwright) | `rules/a11y-playwright.md` | MEDIUM | Full-page axe-core scanning with WCAG 2.2 AA | | [A11y CI/CD](#accessibility-cicd) | `rules/a11y-testing.md` | MEDIUM | CI gates, jest-axe unit tests, PR blocking | | [End-to-End Types](#end-to-end-types) | `rules/validation-end-to-end.md` | HIGH | tRPC, Prisma, Pydantic type safety | **Total: 7 rules, 2 references, 3 checklists, 1 example, 1 script** ## Upstream coverage (do not restate) Playwright, axe-core and jest-axe document themselves. This skill carries only the OrchestKit delta (`references/ork-delta.md`) plus the house subsets in `rules/`. Fetch the vendor page for anything below instead of expecting it here. | Topic | Source | |-------|--------| | Screenshot comparison workflow, baseline files, `snapshotPathTemplate` | https://playwright.dev/docs/test-snapshots | | `toHaveScreenshot` options: `mask`, `maxDiffPixelRatio`, `stylePath`, `animations` | https://playwright.dev/docs/api/class-pageassertions | | Auth reuse: `storageState`, setup projects, IndexedDB | https://playwright.dev/docs/auth | | Network interception with `page.route` (the house default is still emulate first, see `rules/emulate-e2e.md`) | https://playwright.dev/docs/mock | | Removed and changed APIs per release (SKILL.md keeps only the short denylist below) | https://playwright.dev/docs/release-notes | | Full locator API surface (the house priority ladder stays in `rules/e2e-playwright.md`) | https://playwright.dev/docs/locators | | Playwright runner setup on CI (the house a11y gate workflow stays in `rules/a11y-testing.md`) | https://playwright.dev/docs/ci | | `init-agents` CLI flags and generated files (the Planner/Generator/Healer workflow stays in `rules/e2e-ai-agents.md`) | https://playwright.dev/docs/test-agents | | `jest-axe` matcher and `configureAxe` API (the house component-state subset stays in `rules/a11y-testing.md`) | https://github.com/NickColley/jest-axe | | Lighthouse CI configuration and score assertions | https://github.com/GoogleChrome/lighthouse-ci | WCAG 2.2 success criteria and the manual keyboard / screen-reader / contrast / zoom passes are NOT routed away: `checklists/a11y-testing-checklist.md` still carries them in full, with https://www.w3.org/WAI/WCAG22/quickref/ as the normative reference. ## emulate Backends For E2E tests that interact with external APIs (GitHub, Vercel, Google), **use emulate as the backend** instead of hitting real APIs. This eliminates flakiness from rate limits, network issues, and non-deterministic data. | Approach | Result | |----------|--------| | **emulate backends** (FIRST CHOICE) | Deterministic, fast, CI-friendly | | Real APIs | Flaky, rate-limited, slow | | MSW/Nock intercepts | No state machines, manual response management | Key features: seed config for reproducible data, per-worker port isolation for parallel Playwright, full state machine transitions. See `rules/emulate-e2e.md` for patterns, CI configuration, and per-worker isolation fixtures. --- ## Playwright Quick Start ```typescript import { test, expect } from '@playwright/test'; test('user can complete checkout', async ({ page }) => { await page.goto('/products'); await page.getByRole('button', { name: 'Add to cart' }).click(); await page.getByRole('link', { name: 'Checkout' }).click(); await page.getByLabel('Email').fill('test@example.com'); await page.getByRole('button', { name: 'Submit' }).click(); await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible(); }); ``` **Locator Priority:** `getByRole()` > `getByLabel()` > `getByPlaceholder()` > `getByTestId()` ## Playwright Core Semantic locator patterns and best practices for resilient tests. | Rule | File | Key Pattern | |------|------|-------------| | Playwright E2E | `rules/e2e-playwright.md` | Semantic locators, auto-wait, new 1.58+ features | Anti-patterns (FORBIDDEN): - Hardcoded waits: `await page.waitForTimeout(2000)` - CSS selectors for interactions: `await page.click('.submit-btn')` - XPath locators Removed in 1.58/1.59 — do NOT use: - `_react=ComponentName[prop=value]` and `_vue=...` component selector engines — **removed in 1.58** - `:light` selector suffix — **removed** - `launch(\{ devtools: true \})` option — **removed**; use `args: ['--auto-open-devtools-for-tabs']` ## Page Objects Encapsulate page interactions into reusable classes. | Rule | File | Key Pattern | |------|------|-------------| | Page Object Model | `rules/e2e-page-objects.md` | Locators in constructor, action methods, assertion methods | ```typescript const checkout = new CheckoutPage(page); await checkout.fillEmail('test@example.com'); await checkout.submit(); await checkout.expectConfirmation(); ``` ## AI Agents Playwright 1.59+ AI agent framework for test planning, generation, and self-healing. Includes a **token-efficient CLI mode** designed for coding agents — minimal output, structured responses, reduced context overhead. | Rule | File | Key Pattern | |------|------|-------------| | AI Agents | `rules/e2e-ai-agents.md` | Planner, Generator, Healer workflow | ```bash npx playwright init-agents --loop=claude # For Claude Code ``` **Token-efficient CLI mode** (1.58+): Playwright ships a SKILL-focused CLI mode that produces compact, agent-friendly output — use this when running Playwright from AI agents to minimize token consumption. Workflow: Planner (explores app, creates specs) -> Generator (reads spec, tests live app) -> Healer (fixes failures, updates selectors). **New in Playwright 1.59 (Apr 2026) — relevant for AI agents:** - `page.screencast(\{ start, stop, showActions \})` — unified video + real-time JPEG frame streaming. Lets a Healer agent read frames mid-run for visual assertion without writing video files. - `browser.bind()` / `npx playwright-cli attach` — attach to a running browser from an MCP client mid-test; useful for Healer to inspect a hung or failing CI run. - `locator.normalize()` — rewrites a brittle locator to best-practice equivalents. Pair with Healer to auto-upgrade `getByTestId` → `getByRole` where possible. ## Accessibility (Playwright) Full-page accessibility validation with axe-core in E2E tests. | Rule | File | Key Pattern | |------|------|-------------| | Playwright + axe | `rules/a11y-playwright.md` | WCAG 2.2 AA, interactive state testing | ```typescript import AxeBuilder from '@axe-core/playwright'; test('page meets WCAG 2.2 AA', async ({ page }) => { await page.goto('/'); const results = await new AxeBuilder({ page }) .withTags(['wcag2a', 'wcag2aa', 'wcag22aa']) .analyze(); expect(results.violations).toEqual([]); }); ``` ## Accessibility (CI/CD) CI pipeline integration and jest-axe unit-level component testing. | Rule | File | Key Pattern | |------|------|-------------| | CI Gates + jest-axe | `rules/a11y-testing.md` | PR blocking, component state testing | ## End-to-End Types Type safety across API layers to eliminate runtime type errors. | Rule | File | Key Pattern | |------|------|-------------| | Type Safety | `rules/validation-end-to-end.md` | tRPC, Zod, Pydantic, schema rejection tests | ## Visual Regression Native Playwright screenshot comparison without external services. ```typescript await expect(page).toHaveScreenshot('checkout-page.png', { maxDiffPixels: 100, mask: [page.locator('.dynamic-content')], }); ``` House rules that the vendor docs do not state (CI-only baselines, single snapshot project, mask over threshold): `references/ork-delta.md`. Option reference and the baseline workflow itself: see the Upstream coverage table above. ## Key Decisions | Decision | Recommendation | |----------|----------------| | E2E framework | Playwright 1.59+ with semantic locators | | Locator strategy | `getByRole` > `getByLabel` > `getByTestId` | | Browser | Chromium (Chrome for Testing in 1.59+) | | Page pattern | Page Object Model for complex pages | | Visual regression | Playwright native `toHaveScreenshot()` | | A11y testing | axe-core (E2E) + jest-axe (unit) | | CI retries | 2-3 in CI, 0 locally | | Flaky detection | `failOnFlakyTests: true` in CI | | AI agents | Planner/Generator/Healer via `init-agents` | | Type safety | tRPC for end-to-end, Zod for runtime validation | ## References | Resource | Description | |----------|-------------| | `references/ork-delta.md` | House rules the vendor docs do not state: jest-axe over vitest-axe, CLI-only agent init, CI-only baselines, single snapshot project, mask over threshold | | `references/playwright-setup.md` | Installation, MCP server, seed tests, agent initialization | ## Checklists | Checklist | Description | |-----------|-------------| | `checklists/e2e-checklist.md` | Locator strategy, page objects, CI/CD, visual regression | | `checklists/e2e-testing-checklist.md` | Comprehensive: planning, implementation, SSE, responsive, maintenance | | `checklists/a11y-testing-checklist.md` | Automated + manual: keyboard, screen reader, color contrast, WCAG | ## Examples | Example | Description | |---------|-------------| | `examples/orchestkit-e2e-tests.md` | OrchestKit analysis flow: page objects, SSE progress, error handling | Generic Playwright samples (user flows, auth fixtures, API mocking, multi-tab, file upload, axe scans) now come from the vendor pages in the Upstream coverage table. ## Scripts | Script | Description | |--------|-------------| | `scripts/create-page-object.md` | Generate Playwright page object with auto-detected patterns | ## Related Skills - `testing-unit` - Unit testing patterns with mocking, fixtures, and data factories - `testing-integration` - API boundary and contract testing - `cover` - Generates the E2E tier when the suite does not exist yet - `verify` - Grades an existing suite and returns a merge verdict - `expect` - Diff-aware browser verification via agent-browser - `emulate-seed` - Seed configuration authoring for emulate providers - `portless` (upstream) - Stable HTTPS `baseURL` for local E2E tests (`https://myapp.localhost` instead of port guessing; HTTPS-on-443 default since portless 0.10) --- ## Rules (7) ### Validate full-page accessibility compliance through Playwright E2E tests with axe-core — MEDIUM # Playwright + axe-core E2E ```typescript import { test, expect } from '@playwright/test'; import AxeBuilder from '@axe-core/playwright'; test('page has no a11y violations', async ({ page }) => { await page.goto('/'); const results = await new AxeBuilder({ page }) .withTags(['wcag2a', 'wcag2aa', 'wcag22aa']) .analyze(); expect(results.violations).toEqual([]); }); test('modal state has no violations', async ({ page }) => { await page.goto('/'); await page.click('[data-testid="open-modal"]'); await page.waitForSelector('[role="dialog"]'); const results = await new AxeBuilder({ page }) .include('[role="dialog"]') .withTags(['wcag2a', 'wcag2aa']) .analyze(); expect(results.violations).toEqual([]); }); ``` ## Key Decisions | Decision | Choice | Rationale | |----------|--------|-----------| | Test runner | Playwright + axe | Full page coverage | | WCAG level | AA (wcag2aa) | Industry standard | | State testing | Test all interactive states | Modal, error, loading | | Browser matrix | Chromium + Firefox | Cross-browser coverage | **Incorrect — Testing page without WCAG tags:** ```typescript test('page has no violations', async ({ page }) => { await page.goto('/'); const results = await new AxeBuilder({ page }).analyze(); expect(results.violations).toEqual([]); }); ``` **Correct — Testing with WCAG 2.2 AA compliance:** ```typescript test('page meets WCAG 2.2 AA', async ({ page }) => { await page.goto('/'); const results = await new AxeBuilder({ page }) .withTags(['wcag2a', 'wcag2aa', 'wcag22aa']) .analyze(); expect(results.violations).toEqual([]); }); ``` ### Enforce accessibility testing in CI pipelines and enable unit-level component testing with jest-axe — MEDIUM # CI/CD Accessibility Gates ```yaml # .github/workflows/accessibility.yml name: Accessibility on: [pull_request] jobs: a11y: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: { node-version: '20', cache: 'npm' } - run: npm ci - run: npm run test:a11y - run: npm run build - run: npx playwright install --with-deps chromium - run: npm start & npx wait-on http://localhost:3000 - run: npx playwright test e2e/accessibility ``` ## Anti-Patterns (FORBIDDEN) ```typescript // BAD: Excluding too much new AxeBuilder({ page }) .exclude('body') // Defeats the purpose .analyze(); // BAD: No CI enforcement // Accessibility tests exist but don't block PRs // BAD: Manual-only testing // Relying solely on human review ``` ## Key Decisions | Decision | Choice | Rationale | |----------|--------|-----------| | CI gate | Block on violations | Prevent regression | | Tags | wcag2a, wcag2aa, wcag22aa | Full WCAG 2.2 AA | | Exclusions | Third-party widgets only | Minimize blind spots | **Incorrect — Accessibility tests exist but don't enforce in CI:** ```yaml # .github/workflows/test.yml - run: npm run test:a11y # Runs but doesn't block on failures - run: npm run test:unit ``` **Correct — CI blocks PRs on accessibility violations:** ```yaml # .github/workflows/accessibility.yml on: [pull_request] jobs: a11y: runs-on: ubuntu-latest steps: - run: npm run test:a11y # Exits with code 1 on violations - run: npx playwright test e2e/accessibility # Blocks merge ``` --- # jest-axe Unit Testing ## Setup ```typescript // jest.setup.ts import { toHaveNoViolations } from 'jest-axe'; expect.extend(toHaveNoViolations); ``` ## Component Testing ```typescript import { render } from '@testing-library/react'; import { axe } from 'jest-axe'; it('has no a11y violations', async () => { const { container } = render(); expect(await axe(container)).toHaveNoViolations(); }); ``` ## Anti-Patterns (FORBIDDEN) ```typescript // BAD: Disabling rules globally const results = await axe(container, { rules: { 'color-contrast': { enabled: false } } // NEVER disable rules }); // BAD: Only testing happy path it('form is accessible', async () => { const { container } = render(
); expect(await axe(container)).toHaveNoViolations(); // Missing: error state, loading state, disabled state }); ``` ## Key Patterns - Test all component states (default, error, loading, disabled) - Never disable axe rules globally - Use for fast feedback in development **Incorrect — Only testing the default state:** ```typescript it('form is accessible', async () => { const { container } = render(); expect(await axe(container)).toHaveNoViolations(); // Missing: error, loading, disabled states }); ``` **Correct — Testing all component states:** ```typescript it('form is accessible in all states', async () => { const { container, rerender } = render(); expect(await axe(container)).toHaveNoViolations(); rerender(); expect(await axe(container)).toHaveNoViolations(); rerender(); expect(await axe(container)).toHaveNoViolations(); }); ``` ### Use Playwright AI agent framework for test planning, generation, and self-healing — HIGH # Playwright AI Agents (1.58+) ## Initialize AI Agents ```bash npx playwright init-agents --loop=claude # For Claude Code npx playwright init-agents --loop=vscode # For VS Code (v1.105+) npx playwright init-agents --loop=opencode # For OpenCode ``` ## Generated Structure | Directory/File | Purpose | |----------------|---------| | `.github/` | Agent definitions and configuration | | `specs/` | Test plans in Markdown format | | `tests/seed.spec.ts` | Seed file for AI agents to reference | ## Agent Workflow ``` 1. PLANNER --> Explores app --> Creates specs/checkout.md (uses seed.spec.ts) 2. GENERATOR --> Reads spec --> Tests live app --> Outputs tests/checkout.spec.ts (verifies selectors actually work) 3. HEALER --> Runs tests --> Fixes failures --> Updates selectors/waits (self-healing) ``` ## Key Concepts - **seed.spec.ts is required** — Planner executes this to learn environment, auth, UI elements - **Generator validates live** — Actually tests app to verify selectors work - **Healer auto-fixes** — When UI changes break tests, replays and patches ## Setup Requirements ```json // .mcp.json in project root { "mcpServers": { "playwright": { "command": "npx", "args": ["-y", "@playwright/mcp@latest"] } } } ``` **Incorrect — No seed file for AI agents to learn from:** ```typescript // Missing tests/seed.spec.ts // AI agents have no example to understand app structure npx playwright init-agents --loop=claude ``` **Correct — Seed file teaches agents app patterns:** ```typescript // tests/seed.spec.ts import { test } from '@playwright/test'; test('example checkout flow', async ({ page }) => { await page.goto('/'); await page.getByRole('button', { name: 'Add to cart' }).click(); await page.getByRole('link', { name: 'Checkout' }).click(); // Agents learn selectors and patterns from this }); ``` ### Encapsulate page interactions into reusable page object classes for maintainable E2E tests — HIGH # Page Object Model Extract page interactions into reusable classes for maintainable E2E tests. ## Pattern ```typescript // pages/CheckoutPage.ts import { Page, Locator } from '@playwright/test'; export class CheckoutPage { readonly page: Page; readonly emailInput: Locator; readonly submitButton: Locator; readonly confirmationHeading: Locator; constructor(page: Page) { this.page = page; this.emailInput = page.getByLabel('Email'); this.submitButton = page.getByRole('button', { name: 'Submit' }); this.confirmationHeading = page.getByRole('heading', { name: 'Order confirmed' }); } async fillEmail(email: string) { await this.emailInput.fill(email); } async submit() { await this.submitButton.click(); } async expectConfirmation() { await expect(this.confirmationHeading).toBeVisible(); } } ``` ## Visual Regression ```typescript // Capture and compare visual snapshots await expect(page).toHaveScreenshot('checkout-page.png', { maxDiffPixels: 100, mask: [page.locator('.dynamic-content')], }); ``` ## Critical User Journeys to Test 1. **Authentication:** Signup, login, password reset 2. **Core Transaction:** Purchase, booking, submission 3. **Data Operations:** Create, update, delete 4. **User Settings:** Profile update, preferences **Incorrect — Duplicating selectors across tests:** ```typescript test('checkout flow', async ({ page }) => { await page.getByLabel('Email').fill('test@example.com'); await page.getByRole('button', { name: 'Submit' }).click(); }); test('another checkout test', async ({ page }) => { await page.getByLabel('Email').fill('user@example.com'); // Duplicated await page.getByRole('button', { name: 'Submit' }).click(); // Duplicated }); ``` **Correct — Page Object encapsulates selectors:** ```typescript const checkout = new CheckoutPage(page); await checkout.fillEmail('test@example.com'); await checkout.submit(); await checkout.expectConfirmation(); ``` ### Apply semantic locator patterns and best practices for resilient Playwright E2E tests — HIGH # Playwright E2E Testing (1.58+) ## Semantic Locators ```typescript // PREFERRED: Role-based locators (most resilient) await page.getByRole('button', { name: 'Add to cart' }).click(); await page.getByRole('link', { name: 'Checkout' }).click(); // GOOD: Label-based for form controls await page.getByLabel('Email').fill('test@example.com'); // ACCEPTABLE: Test IDs for stable anchors await page.getByTestId('checkout-button').click(); // AVOID: CSS selectors and XPath (fragile) ``` **Locator Priority:** `getByRole()` > `getByLabel()` > `getByPlaceholder()` > `getByTestId()` ## Basic Test ```typescript import { test, expect } from '@playwright/test'; test('user can complete checkout', async ({ page }) => { await page.goto('/products'); await page.getByRole('button', { name: 'Add to cart' }).click(); await page.getByRole('link', { name: 'Checkout' }).click(); await page.getByLabel('Email').fill('test@example.com'); await page.getByRole('button', { name: 'Submit' }).click(); await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible(); }); ``` ## New Features (1.58+) ```typescript // Flaky test detection export default defineConfig({ failOnFlakyTests: true }); // Assert individual class names await expect(page.locator('.card')).toContainClass('highlighted'); // IndexedDB storage state await page.context().storageState({ path: 'auth.json', indexedDB: true }); ``` ## Anti-Patterns (FORBIDDEN) ```typescript // NEVER use hardcoded waits await page.waitForTimeout(2000); // NEVER use CSS selectors for user interactions await page.click('.submit-btn'); // ALWAYS use semantic locators + auto-wait await page.getByRole('button', { name: 'Submit' }).click(); await expect(page.getByRole('alert')).toBeVisible(); ``` ## Key Decisions | Decision | Recommendation | |----------|----------------| | Locators | `getByRole` > `getByLabel` > `getByTestId` | | Browser | Chromium (Chrome for Testing in 1.58+) | | Execution | 5-30s per test | | Retries | 2-3 in CI, 0 locally | **Incorrect — Using hardcoded waits and CSS selectors:** ```typescript await page.click('.submit-button'); await page.waitForTimeout(2000); await expect(page.locator('.success-message')).toBeVisible(); ``` **Correct — Semantic locators with auto-wait:** ```typescript await page.getByRole('button', { name: 'Submit' }).click(); await expect(page.getByRole('alert', { name: /success/i })).toBeVisible(); ``` ### E2E Testing with emulate Backends — HIGH # E2E Testing with emulate Backends Use emulate as the backend API layer for Playwright E2E tests. This eliminates flaky real-API calls, rate limiting, and non-deterministic data. ## Pattern: Seed -> Start -> Playwright -> Assert ```typescript import { test, expect } from '@playwright/test'; // Package is `emulate`, entry point createEmulator. // Verified 2026-07-31 against emulate@0.9.0 dist/api.d.ts. import { createEmulator, type Emulator } from 'emulate'; let emulator: Emulator; test.beforeAll(async () => { emulator = await createEmulator({ service: 'github', // ServiceName union, not `provider` seed: { repos: [{ owner: 'acme', name: 'app', issues: [{ number: 1, title: 'Bug fix', state: 'open' }] }], }, }); }); test.afterAll(async () => { await emulator.close(); }); test('dashboard shows seeded issues', async ({ page }) => { await page.goto('/dashboard'); await expect(page.getByRole('cell', { name: 'Bug fix' })).toBeVisible(); }); test('closing issue updates UI', async ({ page }) => { await page.goto('/issues/1'); await page.getByRole('button', { name: 'Close issue' }).click(); await expect(page.getByText('Closed')).toBeVisible(); }); ``` ## Per-Worker Port Isolation (Parallel Playwright) Playwright runs workers in parallel. Each worker needs its own emulate instance on a unique port: ```typescript // fixtures/emulate.fixture.ts import { test as base } from '@playwright/test'; // Package is `emulate`, entry point createEmulator. // Verified 2026-07-31 against emulate@0.9.0 dist/api.d.ts. import { createEmulator } from 'emulate'; type EmulateFixtures = { emulateUrl: string }; export const test = base.extend({ emulateUrl: [async ({}, use, workerInfo) => { // Per-worker port isolation: `port` is a real EmulatorOptions field. const emulator = await createEmulator({ service: 'github', port: 4100 + workerInfo.workerIndex, seed: { repos: [{ owner: 'acme', name: 'app' }] }, }); await use(emulator.url); // hand tests the url, not a bare port await emulator.close(); }, { scope: 'worker' }], }); // In tests: test('worker-isolated test', async ({ page, emulateUrl }) => { // App configured to use emulateUrl as its API base await page.goto(`/dashboard?api_base=${encodeURIComponent(emulateUrl)}`); }); ``` ## CI Configuration ```yaml # .github/workflows/e2e.yml jobs: e2e: steps: - uses: actions/checkout@v4 - run: npm ci - run: npx playwright install --with-deps chromium - run: npx playwright test env: GITHUB_API_BASE: http://localhost:4100 CI: true ``` **Incorrect -- E2E tests hitting real GitHub/Vercel APIs:** ```typescript test('shows repos from GitHub', async ({ page }) => { // Flaky: depends on real GitHub API, rate-limited, non-deterministic data await page.goto('/repos'); await expect(page.getByRole('cell')).toHaveCount(30); // Random count }); ``` **Correct -- emulate backends with seeded data:** ```typescript test('shows repos from emulate', async ({ page }) => { // Deterministic: emulate seeded with exactly 3 repos await page.goto('/repos'); await expect(page.getByRole('row')).toHaveCount(4); // 3 repos + header }); ``` ## Related Skills - `emulate-seed` — Seed configuration authoring for emulate providers - `testing-integration` — Integration tests also use emulate as first choice ### Validate end-to-end type safety across API layers to eliminate runtime type errors — HIGH ## End-to-End Type Safety Validation **Incorrect -- type gaps between API layers:** ```typescript // Manual type definitions that can drift from schema interface User { id: string name: string // Missing 'email' field that database has } // No type connection between client and server const response = await fetch('/api/users') const users = await response.json() // type: any ``` **Correct -- tRPC end-to-end type safety:** ```typescript import { initTRPC } from '@trpc/server' import { z } from 'zod' const t = initTRPC.create() export const appRouter = t.router({ getUser: t.procedure .input(z.object({ id: z.string() })) .query(async ({ input }) => { return await db.user.findUnique({ where: { id: input.id } }) }), createUser: t.procedure .input(z.object({ email: z.email(), name: z.string() })) .mutation(async ({ input }) => { return await db.user.create({ data: input }) }) }) export type AppRouter = typeof appRouter // Client gets full type inference from server without code generation ``` **Correct -- Python type safety with Pydantic and NewType:** ```python from typing import NewType from uuid import UUID from pydantic import BaseModel, EmailStr AnalysisID = NewType("AnalysisID", UUID) ArtifactID = NewType("ArtifactID", UUID) def delete_analysis(id: AnalysisID) -> None: ... delete_analysis(artifact_id) # Error with mypy/ty class CreateUserRequest(BaseModel): email: EmailStr name: str = Field(min_length=2, max_length=100) # Type-safe extraction from untyped dict result = {"findings": {...}, "confidence_score": 0.85} findings: dict[str, object] | None = ( cast("dict[str, object]", result.get("findings")) if isinstance(result.get("findings"), dict) else None ) ``` **Testing type safety:** ```typescript // Test that schema rejects invalid data describe('UserSchema', () => { test('rejects invalid email', () => { const result = UserSchema.safeParse({ email: 'not-email', name: 'Test' }) expect(result.success).toBe(false) }) test('rejects missing required fields', () => { const result = UserSchema.safeParse({}) expect(result.success).toBe(false) expect(result.error.issues).toHaveLength(2) }) }) ``` Key decisions: - Runtime validation: Zod (best DX, TypeScript inference) - API layer: tRPC for end-to-end type safety without codegen - Exhaustive checks: assertNever for compile-time union completeness - Python: Pydantic v2 + NewType for branded IDs - Always test validation schemas reject invalid data --- ## References (2) ### Ork Delta # OrchestKit delta for Playwright E2E House rules that are NOT in the vendor docs. Everything else about Playwright, axe-core and jest-axe is upstream: see the "Upstream coverage" table in `SKILL.md`. Provenance lines below name retired files. Those files were deleted in this change and are recorded only to say where a rule came from; they are not live pointers. ## Pin accessibility unit tests to jest-axe, never vitest-axe Why: the house a11y stack standardised on `jest-axe` because `vitest-axe` is frozen at its first release while `jest-axe` keeps shipping (npm registry, checked 2026-07-31: `vitest-axe` 0.1.0, `jest-axe` 11.0.0). `jest-axe` works under Vitest via `expect.extend(toHaveNoViolations)`, so the Vitest-named package buys nothing. Distilled from the retired references/a11y-testing-tools.md; no traced incident. Upstream: https://github.com/NickColley/jest-axe ## Initialize Playwright agents from the CLI, never from playwright.config.ts Why: there is no `aiAgents` key (or any equivalent) in Playwright `TestOptions`. An invented config key type-errors at best and silently does nothing at worst, and the retired reference carried this correction explicitly because the plausible-looking config block is a recurring model invention. Agents are CLI-only: `npx playwright init-agents --loop=claude`. Distilled from the retired references/playwright-1.59-api.md; no traced incident. Upstream: https://playwright.dev/docs/test-agents ## Generate screenshot baselines in CI only Why: a baseline captured on a macOS laptop fails on Linux CI from font hinting and anti-aliasing alone, so a locally-written baseline is a guaranteed red PR. The house default is `updateSnapshots: process.env.CI ? 'missing' : 'none'`, with refreshes done by a manual `workflow_dispatch` job that commits the new PNGs, never by a local run. Distilled from the retired references/visual-regression.md; no traced incident. Upstream: https://playwright.dev/docs/api/class-testconfig ## Snapshot exactly one browser project Why: the same rendering drift exists across engines, so cross-browser screenshots produce three baselines that disagree for reasons no reviewer can act on. Chromium owns the visual baselines; every other project keeps running the functional suite with `ignoreSnapshots: true`. Distilled from the retired references/visual-regression.md; no traced incident. Upstream: https://playwright.dev/docs/api/class-testproject ## Mask the unstable region instead of raising the global diff threshold Why: bumping `maxDiffPixelRatio` to silence one clock or avatar blinds every other screenshot in the suite, which is how a real regression ships green. Scope the tolerance to the element that actually moves: `mask`, `stylePath`, and `animations: 'disabled'`. Distilled from the retired references/visual-regression.md; no traced incident. Upstream: https://playwright.dev/docs/api/class-pageassertions ### Playwright Setup # Playwright Setup with Test Agents Install and configure Playwright with autonomous test agents for Claude Code. ## Prerequisites **Required:** VS Code v1.105+ (released Oct 9, 2025) for agent functionality ## Step 1: Install Playwright ```bash npm install --save-dev @playwright/test npx playwright install # Install browsers (Chromium, Firefox, WebKit) ``` ## Step 2: Add Playwright MCP Server (CC 2.1.6) Create or update `.mcp.json` in your project root: ```json { "mcpServers": { "playwright": { "command": "npx", "args": ["-y", "@playwright/mcp@latest"] } } } ``` Restart your Claude Code session to pick up the MCP configuration. > **Note:** The `claude mcp add` command is deprecated in CC 2.1.6. Configure MCPs directly via `.mcp.json`. ## Step 3: Initialize Test Agents ```bash # Initialize the three agents (planner, generator, healer) npx playwright init-agents --loop=claude # OR for VS Code: --loop=vscode # OR for OpenCode: --loop=opencode ``` **What this does:** - Creates agent definition files in your project - Agents are Markdown-based instruction files - Regenerate when Playwright updates to get latest tools ## Step 4: Create Seed Test Create `tests/seed.spec.ts` - the planner uses this to understand your setup: ```typescript // tests/seed.spec.ts import { test, expect } from '@playwright/test'; test.beforeEach(async ({ page }) => { // Your app initialization await page.goto('http://localhost:3000'); // Login if needed // await page.getByLabel('Email').fill('test@example.com'); // await page.getByLabel('Password').fill('password123'); // await page.getByRole('button', { name: 'Login' }).click(); }); test('seed test - app is accessible', async ({ page }) => { await expect(page).toHaveTitle(/MyApp/); await expect(page.getByRole('navigation')).toBeVisible(); }); ``` **Why seed.spec.ts?** - Planner executes this to learn: - Environment setup (fixtures, hooks) - Authentication flow - App initialization - Available selectors ## Directory Structure ``` your-project/ ├── specs/ <- Planner outputs test plans here (Markdown) ├── tests/ <- Generator outputs test code here (.spec.ts) │ └── seed.spec.ts <- Your initialization test (REQUIRED) ├── playwright.config.ts └── .mcp.json <- MCP server config ``` ## Basic Configuration ```typescript // playwright.config.ts import { defineConfig } from '@playwright/test'; export default defineConfig({ testDir: './tests', fullyParallel: true, retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 1 : undefined, use: { baseURL: 'http://localhost:3000', trace: 'on-first-retry', screenshot: 'only-on-failure', }, projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] } }, ], }); ``` ## Running Tests ```bash npx playwright test # Run all tests npx playwright test --ui # UI mode npx playwright test --debug # Debug mode npx playwright test --headed # See browser ``` ## Browser Automation For quick browser automation outside of Playwright tests, use agent-browser CLI: ```bash # Quick visual verification agent-browser open http://localhost:5173 agent-browser snapshot -i agent-browser screenshot /tmp/screenshot.png agent-browser close ``` Run `agent-browser --help` for full CLI docs. ## Next Steps 1. **Planner**: "Generate test plan for checkout flow" -> creates `specs/checkout.md` 2. **Generator**: "Generate tests from checkout spec" -> creates `tests/checkout.spec.ts` 3. **Healer**: Automatically fixes tests when selectors break See `references/planner-agent.md` for detailed workflow. --- ## Checklists (3) ### A11y Testing Checklist # Accessibility Testing Checklist Use this checklist to ensure comprehensive accessibility coverage. ## Automated Test Coverage ### Unit Tests (jest-axe) - [ ] All form components tested with axe - [ ] All interactive components (buttons, links, modals) tested - [ ] Custom UI widgets tested (date pickers, dropdowns, sliders) - [ ] Dynamic content updates tested - [ ] Error states tested for proper announcements - [ ] Loading states have appropriate ARIA attributes - [ ] Tests cover WCAG 2.1 Level AA tags minimum - [ ] No disabled rules without documented justification ### E2E Tests (Playwright + axe-core) - [ ] Homepage scanned for violations - [ ] All critical user journeys include a11y scan - [ ] Post-interaction states scanned (after form submit, modal open) - [ ] Multi-step flows tested (signup, checkout, settings) - [ ] Error pages and 404s tested - [ ] Third-party widgets excluded from scan if necessary - [ ] Tests run in CI/CD pipeline - [ ] Accessibility reports archived on failure ### CI/CD Integration - [ ] Accessibility tests run on every PR - [ ] Pre-commit hook runs a11y tests on changed files - [ ] Lighthouse CI monitors accessibility score (>95%) - [ ] Failed tests block deployment - [ ] Test results published to team (GitHub comments, Slack) ## Manual Testing Requirements ### Keyboard Navigation - [ ] **Tab Navigation** - [ ] All interactive elements reachable via Tab/Shift+Tab - [ ] Tab order follows visual layout (top to bottom, left to right) - [ ] Focus indicator visible on all focusable elements - [ ] No keyboard traps (can always Tab away) - [ ] **Action Keys** - [ ] Enter/Space activates buttons and links - [ ] Escape closes modals, dropdowns, menus - [ ] Arrow keys navigate within compound widgets (tabs, menus, sliders) - [ ] Home/End keys navigate to start/end where appropriate - [ ] **Form Controls** - [ ] All form fields accessible via keyboard - [ ] Enter submits forms - [ ] Error messages keyboard-navigable - [ ] Custom controls (date pickers, color pickers) keyboard-operable - [ ] **Skip Links** - [ ] "Skip to main content" link present and functional - [ ] Appears on first Tab press - [ ] Actually skips navigation when activated ### Screen Reader Testing Test with at least one screen reader: - macOS: VoiceOver (Cmd+F5) - Windows: NVDA (free) or JAWS - Linux: Orca #### Content Structure - [ ] **Headings** - [ ] Logical heading hierarchy (h1 → h2 → h3, no skips) - [ ] Page has exactly one h1 - [ ] Headings describe section content - [ ] Can navigate by heading (H key in screen reader) - [ ] **Landmarks** - [ ] `<header>`, `<nav>`, `<main>`, `<footer>` present - [ ] Multiple landmarks of same type have unique labels - [ ] Can navigate by landmark (D key in screen reader) - [ ] **Lists** - [ ] Navigation uses `
    ` or `<nav>` - [ ] Related items grouped in lists - [ ] Screen reader announces list with item count #### Interactive Elements - [ ] **Forms** - [ ] All inputs have associated `<label>` or `aria-label` - [ ] Required fields announced as required - [ ] Error messages announced when they appear - [ ] Field types announced (email, password, number) - [ ] Placeholder text not used as only label - [ ] **Buttons and Links** - [ ] Role announced ("button", "link") - [ ] Purpose clear from label alone - [ ] State announced (expanded/collapsed, selected) - [ ] Icon-only buttons have `aria-label` - [ ] **Images** - [ ] Informative images have meaningful `alt` text - [ ] Decorative images have `alt=""` or `role="presentation"` - [ ] Complex images have longer description (`aria-describedby` or caption) - [ ] **Dynamic Content** - [ ] Live regions announce updates (`aria-live="polite"` or `"assertive"`) - [ ] Loading states announced - [ ] Success/error messages announced - [ ] Content changes don't lose focus position #### Navigation - [ ] **Menus** - [ ] Menu buttons announce expanded/collapsed state - [ ] Arrow keys navigate menu items - [ ] First/last items wrap or stop appropriately - [ ] Escape closes menu - [ ] **Modals/Dialogs** - [ ] Focus moves to modal on open - [ ] Focus trapped within modal - [ ] Modal title announced - [ ] Escape closes modal - [ ] Focus returns to trigger on close - [ ] **Tabs** - [ ] Tab role announced - [ ] Active tab announced as selected - [ ] Arrow keys navigate tabs - [ ] Tab panel content announced ### Color and Contrast Use browser extensions (axe DevTools, WAVE) or online tools: - [ ] **Text Contrast** - [ ] Normal text (< 18pt): 4.5:1 minimum ratio - [ ] Large text (≥ 18pt or 14pt bold): 3:1 minimum ratio - [ ] Passes for all text (body, headings, labels, placeholders) - [ ] **UI Component Contrast** - [ ] Buttons, inputs, icons: 3:1 minimum against background - [ ] Focus indicators: 3:1 minimum - [ ] Error/success states: 3:1 minimum - [ ] **Color Independence** - [ ] Information not conveyed by color alone - [ ] Links distinguishable without color (underline, icon, etc.) - [ ] Form errors indicated by icon + text, not just red border - [ ] Charts/graphs have patterns or labels, not just colors ### Responsive and Zoom Testing - [ ] **Browser Zoom (200%)** - [ ] Test at 200% zoom level (WCAG 2.1 requirement) - [ ] No horizontal scrolling at 200% zoom - [ ] All content visible and readable - [ ] No overlapping or cut-off text - [ ] Interactive elements remain operable - [ ] **Mobile/Touch** - [ ] Touch targets ≥ 44×44 CSS pixels - [ ] Sufficient spacing between interactive elements (at least 8px) - [ ] No reliance on hover (all hover info accessible on tap) - [ ] Pinch-to-zoom enabled (no `user-scalable=no`) - [ ] Orientation works in both portrait and landscape ### Animation and Motion - [ ] **Respect Motion Preferences** - [ ] Check `prefers-reduced-motion` media query - [ ] Disable or reduce animations when preferred - [ ] Test with system setting enabled (macOS, Windows) - [ ] **No Seizure Triggers** - [ ] No flashing content faster than 3 times per second - [ ] Autoplay videos have controls (pause/stop) - [ ] Parallax effects can be disabled ## Documentation Review - [ ] **ARIA Usage** - [ ] ARIA only used when native HTML insufficient - [ ] ARIA roles match HTML semantics - [ ] All required ARIA properties present - [ ] No conflicting or redundant ARIA - [ ] **Code Comments** - [ ] Complex accessibility patterns documented - [ ] Keyboard shortcuts documented - [ ] Focus management documented ## Cross-Browser Testing Test in multiple browsers and assistive tech combinations: - [ ] Chrome + NVDA (Windows) - [ ] Firefox + NVDA (Windows) - [ ] Safari + VoiceOver (macOS) - [ ] Safari + VoiceOver (iOS) - [ ] Chrome + TalkBack (Android) ## Compliance Verification - [ ] **WCAG 2.1 Level AA** - [ ] Automated tests pass for wcag2a, wcag2aa, wcag22aa tags - [ ] Manual testing confirms keyboard accessibility - [ ] Manual testing confirms screen reader accessibility - [ ] Color contrast verified - [ ] **Legal Requirements** - [ ] Section 508 (US federal) - [ ] ADA (US) - [ ] EN 301 549 (EU) - [ ] Accessibility statement page present (if required) ## Continuous Monitoring - [ ] Lighthouse accessibility score tracked over time - [ ] Accessibility tests in regression suite - [ ] New features include a11y tests from day one - [ ] Team trained on accessibility best practices - [ ] Accessibility champion assigned - [ ] Regular audits scheduled (quarterly recommended) ## When to Seek Expert Help Engage an accessibility specialist if: - [ ] Building complex custom widgets (ARIA patterns) - [ ] Handling advanced screen reader interactions - [ ] Preparing for legal compliance audit - [ ] User feedback indicates accessibility issues - [ ] Automated tests show many violations - [ ] Team lacks accessibility expertise ## Quick Wins for Common Issues ### Missing Alt Text ```html Company Logo ``` ### Unlabeled Form Input ```html ``` ### Low Contrast Text ```css /* Before */ color: #999; /* 2.8:1 ratio */ /* After */ color: #767676; /* 4.5:1 ratio */ ``` ### Keyboard Trap ```jsx // Before
    Click me
    // After ``` ### Missing Focus Indicator ```css /* Before */ button:focus { outline: none; } /* After */ button:focus-visible { outline: 2px solid blue; outline-offset: 2px; } ``` ### E2e Checklist # E2E Testing Checklist ## Test Selection Checklist Focus E2E tests on business-critical paths: - [ ] **Authentication:** Signup, login, password reset, logout - [ ] **Core Transaction:** Purchase, booking, submission, payment - [ ] **Data Operations:** Create, update, delete critical entities - [ ] **User Settings:** Profile update, preferences, notifications - [ ] **Error Recovery:** Form validation, API errors, network issues ## Locator Strategy Checklist - [ ] Use `getByRole()` as primary locator strategy - [ ] Use `getByLabel()` for form inputs - [ ] Use `getByPlaceholder()` when no label available - [ ] Use `getByTestId()` only as last resort - [ ] **AVOID** CSS selectors for user interactions - [ ] **AVOID** XPath locators - [ ] **AVOID** `page.click('[data-testid=...]')` - use `getByTestId` instead ## Test Implementation Checklist For each test: - [ ] Clear, descriptive test name - [ ] Tests one user flow or scenario - [ ] Uses semantic locators (getByRole, getByLabel) - [ ] Waits for elements using Playwright's auto-wait - [ ] No hardcoded `sleep()` or `wait()` calls - [ ] Assertions use `expect()` with appropriate matchers - [ ] Test can run in isolation (no dependencies on other tests) ## Page Object Checklist For each page object: - [ ] Locators defined in constructor - [ ] Methods for user actions (login, submit, navigate) - [ ] Assertion methods (expectError, expectSuccess) - [ ] No direct `page.click()` calls - wrap in methods - [ ] TypeScript types for all methods ## Configuration Checklist - [ ] Set `baseURL` in config - [ ] Configure browser(s) for testing - [ ] Set up authentication state project - [ ] Configure retries for CI (2-3 retries) - [ ] Enable `failOnFlakyTests` in CI - [ ] Set appropriate timeouts - [ ] Configure screenshot on failure ## CI/CD Checklist - [ ] Tests run in CI pipeline - [ ] Artifacts (screenshots, traces) uploaded on failure - [ ] Tests parallelized with sharding - [ ] Auth state cached between runs - [ ] Web server waits for ready signal ## Visual Regression Checklist - [ ] Screenshots stored in version control - [ ] Different screenshots per browser/platform - [ ] Mobile viewports tested - [ ] Dark mode tested (if applicable) - [ ] Threshold set for acceptable diff ## Accessibility Checklist - [ ] axe-core integrated for a11y testing - [ ] Critical pages tested for violations - [ ] Forms have proper labels - [ ] Focus management tested - [ ] Keyboard navigation tested ## Review Checklist Before PR: - [ ] All tests pass locally - [ ] Tests are deterministic (no flakes) - [ ] Locators follow semantic strategy - [ ] No hardcoded waits - [ ] Test files organized logically - [ ] Page objects used for complex pages - [ ] CI configuration updated if needed ## Anti-Patterns to Avoid - [ ] Too many E2E tests (keep it focused) - [ ] Testing non-critical paths - [ ] Hard-coded waits (`await page.waitForTimeout()`) - [ ] CSS/XPath selectors for interactions - [ ] Tests that depend on each other - [ ] Tests that modify global state - [ ] Ignoring flaky test warnings ### E2e Testing Checklist # E2E Testing Checklist Comprehensive checklist for planning, implementing, and maintaining E2E tests with Playwright. ## Pre-Implementation ### Test Planning - [ ] Identify critical user journeys to test - [ ] Map out happy paths and error scenarios - [ ] Determine test data requirements - [ ] Decide on mocking strategy (API, SSE, external services) - [ ] Plan for visual regression testing needs - [ ] Identify accessibility requirements (WCAG 2.1 AA) - [ ] Estimate test execution time and CI impact ### Environment Setup - [ ] Install Playwright (`npm install -D @playwright/test`) - [ ] Install browser binaries (`npx playwright install`) - [ ] Create `playwright.config.ts` with base URL and timeouts - [ ] Configure test directory structure (`tests/e2e/`) - [ ] Set up Page Object pattern structure - [ ] Configure CI environment (GitHub Actions, GitLab CI, etc.) - [ ] Set up test database/backend for integration tests ### Test Data Strategy - [ ] Create fixtures for common test scenarios - [ ] Set up database seeding scripts - [ ] Plan API mocking approach (mock server vs route interception) - [ ] Create reusable test data generators - [ ] Handle authentication/authorization test cases - [ ] Plan for cleanup between tests ## Test Implementation ### Page Objects - [ ] Create base page class with common utilities - [ ] Implement page object for each major page/component - [ ] Use semantic locators (role, label, test-id) - [ ] Avoid brittle CSS/XPath selectors - [ ] Encapsulate complex interactions in helper methods - [ ] Add TypeScript types for type safety - [ ] Document page object APIs ### Test Structure - [ ] Follow Arrange-Act-Assert (AAA) pattern - [ ] Use descriptive test names (should/when/given format) - [ ] Group related tests with `test.describe()` - [ ] Set up common state in `beforeEach()` - [ ] Clean up resources in `afterEach()` - [ ] Use test fixtures for shared setup - [ ] Keep tests independent (no test interdependencies) ### Assertions - [ ] Use specific assertions (`toHaveText` vs `toBeTruthy`) - [ ] Assert on user-visible behavior, not implementation - [ ] Verify loading states appear and disappear - [ ] Check error messages and validation feedback - [ ] Validate success states and confirmations - [ ] Test navigation and URL changes - [ ] Verify data persistence across page loads ### API Interactions - [ ] Mock external API calls for reliability - [ ] Test real API endpoints in integration tests - [ ] Handle async operations properly (promises, awaits) - [ ] Test timeout scenarios - [ ] Verify retry logic - [ ] Test rate limiting behavior - [ ] Mock SSE/WebSocket streams ### SSE/Real-Time Features - [ ] Test SSE connection establishment - [ ] Verify progress updates stream correctly - [ ] Test reconnection on connection drop - [ ] Handle SSE error events - [ ] Test SSE completion and cleanup - [ ] Verify UI updates from SSE events - [ ] Test SSE with network throttling ### Error Handling - [ ] Test form validation errors - [ ] Test API error responses (400, 500, etc.) - [ ] Test network failures - [ ] Test timeout scenarios - [ ] Verify error messages shown to user - [ ] Test retry/recovery mechanisms - [ ] Test graceful degradation ### Loading States - [ ] Test loading spinners appear - [ ] Verify skeleton screens render - [ ] Test loading state timeouts - [ ] Check loading states disappear on completion - [ ] Test loading state cancellation - [ ] Verify loading indicators are accessible ### Responsive Design - [ ] Test on desktop viewports (1920x1080, 1366x768) - [ ] Test on tablet viewports (768x1024, 1024x768) - [ ] Test on mobile viewports (375x667, 414x896) - [ ] Verify touch interactions on mobile - [ ] Test responsive navigation menus - [ ] Verify content reflow on viewport changes - [ ] Test orientation changes (portrait/landscape) ### Accessibility - [ ] Test keyboard navigation (Tab, Enter, Escape, arrows) - [ ] Verify focus management (focus visible, focus traps) - [ ] Test screen reader announcements (aria-live, role=status) - [ ] Check ARIA labels and descriptions - [ ] Test color contrast (use automated tools) - [ ] Verify form labels and error associations - [ ] Test with browser accessibility extensions - [ ] Consider adding axe-core integration ### Visual Regression - [ ] Identify components/pages for screenshot testing - [ ] Set up baseline screenshots - [ ] Configure pixel diff thresholds - [ ] Test responsive breakpoints visually - [ ] Test theme variations (light/dark mode) - [ ] Test different locales (i18n) - [ ] Update baselines when designs change ## Code Quality ### Test Maintainability - [ ] Avoid test duplication (use helpers, fixtures) - [ ] Use constants for magic strings/numbers - [ ] Keep tests readable (avoid over-abstraction) - [ ] Add comments for complex test logic - [ ] Refactor brittle tests - [ ] Remove flaky tests or fix root cause - [ ] Review test coverage regularly ### Performance - [ ] Run tests in parallel where possible - [ ] Minimize test execution time (mock slow APIs) - [ ] Use `test.describe.configure(\{ mode: 'parallel' \})` - [ ] Avoid unnecessary waits (`waitForTimeout`) - [ ] Use strategic waits (`waitForSelector`, `waitForLoadState`) - [ ] Optimize page load times (disable unnecessary assets) - [ ] Profile slow tests and optimize ### Flakiness Prevention - [ ] Use deterministic waits (waitFor* methods) - [ ] Avoid race conditions (wait for element visibility) - [ ] Handle timing issues (debounce, throttle) - [ ] Retry flaky tests in CI (max 2 retries) - [ ] Investigate and fix root cause of flakiness - [ ] Use `test.slow()` for long-running tests - [ ] Increase timeouts for legitimate slow operations ## CI/CD Integration ### Pipeline Configuration - [ ] Add E2E test job to CI pipeline - [ ] Run tests on every PR - [ ] Block merge on test failures - [ ] Run tests against staging environment - [ ] Configure test parallelization in CI - [ ] Set up test result reporting - [ ] Archive test artifacts (videos, screenshots, traces) ### Environment Management - [ ] Use Docker Compose for backend services - [ ] Seed test database before test run - [ ] Run migrations before tests - [ ] Clean up test data after run - [ ] Use environment variables for config - [ ] Isolate test environments (per PR if possible) - [ ] Monitor test environment health ### Monitoring & Reporting - [ ] Generate HTML test reports - [ ] Upload test artifacts to CI - [ ] Send notifications on test failures - [ ] Track test execution time trends - [ ] Monitor test flakiness rates - [ ] Set up dashboard for test metrics - [ ] Alert on sustained test failures ## OrchestKit-Specific ### Analysis Flow Tests - [ ] Test URL submission with validation - [ ] Test analysis progress SSE stream - [ ] Verify agent status updates (8 agents) - [ ] Test progress bar updates (0% to 100%) - [ ] Test analysis completion detection - [ ] Test artifact generation - [ ] Test navigation to artifact view ### Agent Orchestration - [ ] Verify supervisor assigns tasks - [ ] Test worker agent execution - [ ] Verify quality gate checks - [ ] Test agent failure handling - [ ] Test partial completion scenarios - [ ] Verify agent status badges ### Artifact Display - [ ] Test artifact metadata display - [ ] Verify quality scores shown - [ ] Test findings/recommendations rendering - [ ] Test artifact search functionality - [ ] Test section navigation (tabs) - [ ] Test download artifact feature - [ ] Test share/copy link feature ### Error Scenarios - [ ] Test invalid URL submission - [ ] Test network timeout during analysis - [ ] Test SSE connection drop - [ ] Test analysis cancellation - [ ] Test concurrent analysis limit - [ ] Test backend service unavailable - [ ] Test rate limiting ### Performance Tests - [ ] Test with large artifact (many findings) - [ ] Test SSE with high event frequency - [ ] Test concurrent analyses (multiple tabs) - [ ] Test long-running analysis (timeout) - [ ] Monitor memory leaks during SSE stream ## Maintenance ### Regular Tasks - [ ] Review and update tests after feature changes - [ ] Update page objects when UI changes - [ ] Update test data when backend schema changes - [ ] Refactor duplicate test code - [ ] Remove obsolete tests - [ ] Update dependencies (Playwright, browsers) - [ ] Review test coverage and add missing tests ### When Tests Fail - [ ] Check if failure is legitimate regression - [ ] Review CI logs and screenshots - [ ] Download and analyze trace files - [ ] Reproduce locally with `--debug` flag - [ ] Fix root cause (not just update assertions) - [ ] Add regression test if bug found - [ ] Update documentation if expected behavior changed ### Optimization - [ ] Profile slow tests and optimize - [ ] Reduce unnecessary API calls - [ ] Optimize page object selectors - [ ] Minimize test data setup - [ ] Use test fixtures for common scenarios - [ ] Run critical tests first (fail fast) - [ ] Archive old test runs ## Documentation ### Test Documentation - [ ] Document test structure in README - [ ] Add comments for complex test logic - [ ] Document page object APIs - [ ] Create testing guide for contributors - [ ] Document CI pipeline configuration - [ ] Maintain test data documentation - [ ] Document mocking strategies ### Knowledge Sharing - [ ] Share test results in PR reviews - [ ] Conduct test review sessions - [ ] Create troubleshooting guide - [ ] Document common test patterns - [ ] Share CI optimization learnings - [ ] Create onboarding guide for new contributors ## Quality Gates ### Before Committing - [ ] All tests pass locally - [ ] New tests added for new features - [ ] No new flaky tests introduced - [ ] Test execution time acceptable - [ ] Code reviewed for maintainability - [ ] Accessibility tests pass - [ ] Visual regression tests updated ### Before Merging PR - [ ] All CI tests pass - [ ] No flaky test failures - [ ] Test coverage maintained or improved - [ ] Test artifacts reviewed (screenshots, videos) - [ ] Performance impact assessed - [ ] Breaking changes documented ### Before Production Deploy - [ ] Full E2E suite passes on staging - [ ] Performance tests pass - [ ] Accessibility tests pass - [ ] Visual regression tests reviewed - [ ] Smoke tests identified for post-deploy - [ ] Rollback plan documented ## Advanced Topics ### Cross-Browser Testing - [ ] Test on Chromium (Chrome/Edge) - [ ] Test on Firefox - [ ] Test on WebKit (Safari) - [ ] Handle browser-specific quirks - [ ] Test with different browser versions ### Internationalization (i18n) - [ ] Test with different locales - [ ] Verify RTL languages (Arabic, Hebrew) - [ ] Test date/time formatting - [ ] Test currency formatting - [ ] Verify translations loaded correctly ### Security Testing - [ ] Test authentication flows - [ ] Test authorization (role-based access) - [ ] Test XSS prevention - [ ] Test CSRF protection - [ ] Test input sanitization - [ ] Test secure headers (CSP, etc.) ### Performance Testing - [ ] Measure page load time - [ ] Test Core Web Vitals (LCP, FID, CLS) - [ ] Test with network throttling - [ ] Test with CPU throttling - [ ] Monitor memory usage - [ ] Test bundle size impact ## Success Metrics - [ ] Test coverage > 80% for critical paths - [ ] Test execution time < 10 minutes - [ ] Test flakiness rate < 2% - [ ] Zero P0 bugs in production from untested areas - [ ] All critical user journeys tested - [ ] 100% of new features have E2E tests - [ ] Test results visible in every PR - [ ] Tests block merge on failure --- **Note:** This checklist is comprehensive but should be adapted to your project's specific needs. Not all items apply to every project. Prioritize based on risk, criticality, and available resources. **OrchestKit Priority:** 1. Analysis flow (URL → Progress → Artifact) 2. SSE real-time updates 3. Error handling and recovery 4. Agent orchestration visibility 5. Accessibility and responsive design --- ## Examples (1) ### Orchestkit E2e Tests # OrchestKit E2E Test Examples Complete E2E test suite examples for OrchestKit's analysis workflow using Playwright + TypeScript. ## Test Configuration ### playwright.config.ts ```typescript import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ testDir: './tests/e2e', fullyParallel: true, forbidOnly: !!process.env.CI, retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 1 : undefined, reporter: 'html', use: { baseURL: 'http://localhost:5173', trace: 'on-first-retry', screenshot: 'only-on-failure', }, projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, }, { name: 'firefox', use: { ...devices['Desktop Firefox'] }, }, { name: 'mobile', use: { ...devices['iPhone 13'] }, }, ], webServer: { command: 'npm run dev', url: 'http://localhost:5173', reuseExistingServer: !process.env.CI, }, }); ``` ## Page Objects ### HomePage (URL Submission) ```typescript // tests/e2e/pages/HomePage.ts import { Page, Locator } from '@playwright/test'; import { BasePage } from '.claude/skills/webapp-testing/assets/playwright-test-template'; export class HomePage extends BasePage { readonly urlInput: Locator; readonly analyzeButton: Locator; readonly analysisTypeSelect: Locator; readonly recentAnalyses: Locator; constructor(page: Page) { super(page); this.urlInput = page.getByTestId('url-input'); this.analyzeButton = page.getByRole('button', { name: /analyze/i }); this.analysisTypeSelect = page.getByTestId('analysis-type-select'); this.recentAnalyses = page.getByTestId('recent-analyses-list'); } async goto(): Promise { await super.goto('/'); await this.waitForLoad(); } async submitUrl(url: string, analysisType = 'comprehensive'): Promise { await this.urlInput.fill(url); if (analysisType !== 'comprehensive') { await this.analysisTypeSelect.selectOption(analysisType); } await this.analyzeButton.click(); } async getRecentAnalysesCount(): Promise { return await this.recentAnalyses.locator('li').count(); } async clickRecentAnalysis(index: number): Promise { await this.recentAnalyses.locator('li').nth(index).click(); } } ``` ### AnalysisProgressPage (SSE Stream) ```typescript // tests/e2e/pages/AnalysisProgressPage.ts import { Page, Locator } from '@playwright/test'; import { BasePage, WaitHelpers } from '.claude/skills/webapp-testing/assets/playwright-test-template'; export class AnalysisProgressPage extends BasePage { readonly progressBar: Locator; readonly progressPercentage: Locator; readonly statusBadge: Locator; readonly agentCards: Locator; readonly errorMessage: Locator; readonly cancelButton: Locator; readonly viewArtifactButton: Locator; private waitHelpers: WaitHelpers; constructor(page: Page) { super(page); this.progressBar = page.getByTestId('analysis-progress-bar'); this.progressPercentage = page.getByTestId('progress-percentage'); this.statusBadge = page.getByTestId('status-badge'); this.agentCards = page.getByTestId('agent-card'); this.errorMessage = page.getByTestId('error-message'); this.cancelButton = page.getByRole('button', { name: /cancel/i }); this.viewArtifactButton = page.getByRole('button', { name: /view artifact/i }); this.waitHelpers = new WaitHelpers(page); } async waitForAnalysisComplete(timeout = 60000): Promise { await this.page.waitForFunction( () => { const badge = document.querySelector('[data-testid="status-badge"]'); return badge?.textContent?.toLowerCase().includes('complete'); }, { timeout } ); } async waitForProgress(percentage: number, timeout = 30000): Promise { await this.page.waitForFunction( (targetPercentage) => { const progressText = document.querySelector('[data-testid="progress-percentage"]')?.textContent; const currentPercentage = parseInt(progressText || '0', 10); return currentPercentage >= targetPercentage; }, percentage, { timeout } ); } async getAgentStatus(agentName: string): Promise<'pending' | 'running' | 'completed' | 'failed'> { const agentCard = this.agentCards.filter({ hasText: agentName }).first(); const statusElement = agentCard.getByTestId('agent-status'); const status = await statusElement.textContent(); return status?.toLowerCase() as any; } async getCompletedAgentsCount(): Promise { return await this.agentCards.filter({ has: this.page.getByText('completed') }).count(); } async cancelAnalysis(): Promise { await this.cancelButton.click(); } async goToArtifact(): Promise { await this.viewArtifactButton.click(); } async getErrorText(): Promise { if (await this.errorMessage.isVisible()) { return await this.errorMessage.textContent(); } return null; } } ``` ### ArtifactPage (View Results) ```typescript // tests/e2e/pages/ArtifactPage.ts import { Page, Locator } from '@playwright/test'; import { BasePage } from '.claude/skills/webapp-testing/assets/playwright-test-template'; export class ArtifactPage extends BasePage { readonly artifactTitle: Locator; readonly sourceUrl: Locator; readonly qualityScore: Locator; readonly findingsSection: Locator; readonly downloadButton: Locator; readonly shareButton: Locator; readonly searchInput: Locator; readonly sectionTabs: Locator; constructor(page: Page) { super(page); this.artifactTitle = page.getByTestId('artifact-title'); this.sourceUrl = page.getByTestId('source-url'); this.qualityScore = page.getByTestId('quality-score'); this.findingsSection = page.getByTestId('findings-section'); this.downloadButton = page.getByRole('button', { name: /download/i }); this.shareButton = page.getByRole('button', { name: /share/i }); this.searchInput = page.getByTestId('artifact-search'); this.sectionTabs = page.getByRole('tab'); } async getQualityScoreValue(): Promise { const scoreText = await this.qualityScore.textContent(); return parseFloat(scoreText || '0'); } async searchInArtifact(query: string): Promise { await this.searchInput.fill(query); await this.page.waitForTimeout(300); // Debounce } async switchToTab(tabName: string): Promise { await this.sectionTabs.filter({ hasText: tabName }).click(); } async downloadArtifact(): Promise { const downloadPromise = this.page.waitForEvent('download'); await this.downloadButton.click(); await downloadPromise; } async getFindingsCount(): Promise { return await this.findingsSection.locator('[data-testid="finding-item"]').count(); } } ``` ## Test Suites ### 1. Happy Path - Complete Analysis Flow ```typescript // tests/e2e/analysis-flow.spec.ts import { test, expect } from '@playwright/test'; import { HomePage } from './pages/HomePage'; import { AnalysisProgressPage } from './pages/AnalysisProgressPage'; import { ArtifactPage } from './pages/ArtifactPage'; import { ApiMocker, CustomAssertions } from '.claude/skills/webapp-testing/assets/playwright-test-template'; test.describe('Analysis Flow - Happy Path', () => { test('should complete full analysis flow from URL submission to artifact view', async ({ page }) => { // 1. Submit URL for analysis const homePage = new HomePage(page); await homePage.goto(); await expect(homePage.urlInput).toBeVisible(); await homePage.submitUrl('https://example.com/article', 'comprehensive'); // 2. Monitor progress with SSE const progressPage = new AnalysisProgressPage(page); await expect(progressPage.progressBar).toBeVisible(); // Wait for initial progress await progressPage.waitForProgress(10); // Check at least one agent is running const agentStatus = await progressPage.getAgentStatus('Tech Comparator'); expect(['running', 'completed']).toContain(agentStatus); // Wait for completion (with timeout for real API) await progressPage.waitForAnalysisComplete(90000); // 90s timeout // Verify all agents completed const completedCount = await progressPage.getCompletedAgentsCount(); expect(completedCount).toBeGreaterThan(0); // 3. Navigate to artifact await progressPage.goToArtifact(); // 4. Verify artifact content const artifactPage = new ArtifactPage(page); await expect(artifactPage.artifactTitle).toBeVisible(); const qualityScore = await artifactPage.getQualityScoreValue(); expect(qualityScore).toBeGreaterThan(0); expect(qualityScore).toBeLessThanOrEqual(10); const findingsCount = await artifactPage.getFindingsCount(); expect(findingsCount).toBeGreaterThan(0); }); }); ``` ### 2. SSE Progress Updates ```typescript // tests/e2e/sse-progress.spec.ts import { test, expect } from '@playwright/test'; import { HomePage } from './pages/HomePage'; import { AnalysisProgressPage } from './pages/AnalysisProgressPage'; import { ApiMocker } from '.claude/skills/webapp-testing/assets/playwright-test-template'; test.describe('SSE Progress Updates', () => { test('should show real-time progress updates via SSE', async ({ page }) => { // Mock SSE stream with progress events const apiMocker = new ApiMocker(page); const sseEvents = [ { data: { type: 'progress', percentage: 0, message: 'Starting analysis...' } }, { data: { type: 'agent_start', agent: 'Tech Comparator' }, delay: 500 }, { data: { type: 'progress', percentage: 25, message: 'Tech Comparator running...' } }, { data: { type: 'agent_complete', agent: 'Tech Comparator' }, delay: 1000 }, { data: { type: 'progress', percentage: 50, message: 'Security Auditor running...' } }, { data: { type: 'agent_complete', agent: 'Security Auditor' }, delay: 1000 }, { data: { type: 'progress', percentage: 100, message: 'Analysis complete!' } }, { data: { type: 'complete', artifact_id: 'test-artifact-123' } }, ]; await apiMocker.mockSSE(/api\/v1\/analyses\/\d+\/stream/, sseEvents); // Submit analysis const homePage = new HomePage(page); await homePage.goto(); await homePage.submitUrl('https://example.com/test'); // Monitor progress updates const progressPage = new AnalysisProgressPage(page); // Wait for 25% progress await progressPage.waitForProgress(25); expect(await progressPage.progressPercentage.textContent()).toContain('25'); // Wait for 50% progress await progressPage.waitForProgress(50); expect(await progressPage.progressPercentage.textContent()).toContain('50'); // Wait for completion await progressPage.waitForProgress(100); await expect(progressPage.statusBadge).toContainText('Complete'); }); test('should handle SSE connection errors gracefully', async ({ page }) => { // Mock SSE connection failure await page.route(/api\/v1\/analyses\/\d+\/stream/, (route) => { route.abort('failed'); }); const homePage = new HomePage(page); await homePage.goto(); await homePage.submitUrl('https://example.com/test'); const progressPage = new AnalysisProgressPage(page); // Should show error message await expect(progressPage.errorMessage).toBeVisible(); const errorText = await progressPage.getErrorText(); expect(errorText).toContain('connection'); }); }); ``` ### 3. Error Handling ```typescript // tests/e2e/error-handling.spec.ts import { test, expect } from '@playwright/test'; import { HomePage } from './pages/HomePage'; import { AnalysisProgressPage } from './pages/AnalysisProgressPage'; import { ApiMocker, CustomAssertions } from '.claude/skills/webapp-testing/assets/playwright-test-template'; test.describe('Error Handling', () => { test('should show validation error for invalid URL', async ({ page }) => { const homePage = new HomePage(page); await homePage.goto(); await homePage.submitUrl('not-a-valid-url'); const assertions = new CustomAssertions(page); await assertions.expectToast('Please enter a valid URL', 'error'); }); test('should handle API error during analysis submission', async ({ page }) => { const apiMocker = new ApiMocker(page); await apiMocker.mockError(/api\/v1\/analyses/, 500, 'Internal server error'); const homePage = new HomePage(page); await homePage.goto(); await homePage.submitUrl('https://example.com/test'); const assertions = new CustomAssertions(page); await assertions.expectToast('Failed to start analysis', 'error'); }); test('should handle analysis failure from backend', async ({ page }) => { const apiMocker = new ApiMocker(page); // Mock successful submission await apiMocker.mockSuccess(/api\/v1\/analyses$/, { id: 123, status: 'processing', url: 'https://example.com/test', }); // Mock SSE with failure event await apiMocker.mockSSE(/api\/v1\/analyses\/123\/stream/, [ { data: { type: 'progress', percentage: 10 } }, { data: { type: 'error', message: 'Failed to fetch content' } }, ]); const homePage = new HomePage(page); await homePage.goto(); await homePage.submitUrl('https://example.com/test'); const progressPage = new AnalysisProgressPage(page); await expect(progressPage.errorMessage).toBeVisible(); const errorText = await progressPage.getErrorText(); expect(errorText).toContain('Failed to fetch content'); }); test('should allow retry after failed analysis', async ({ page }) => { const homePage = new HomePage(page); const progressPage = new AnalysisProgressPage(page); await homePage.goto(); await homePage.submitUrl('https://example.com/test'); // Wait for error state await expect(progressPage.errorMessage).toBeVisible(); // Click retry button const retryButton = page.getByRole('button', { name: /retry/i }); await retryButton.click(); // Should restart analysis await expect(progressPage.progressBar).toBeVisible(); }); }); ``` ### 4. Cancellation & Cleanup ```typescript // tests/e2e/cancellation.spec.ts import { test, expect } from '@playwright/test'; import { HomePage } from './pages/HomePage'; import { AnalysisProgressPage } from './pages/AnalysisProgressPage'; test.describe('Analysis Cancellation', () => { test('should cancel in-progress analysis', async ({ page }) => { const homePage = new HomePage(page); await homePage.goto(); await homePage.submitUrl('https://example.com/long-analysis'); const progressPage = new AnalysisProgressPage(page); // Wait for analysis to start await progressPage.waitForProgress(10); // Cancel analysis await progressPage.cancelAnalysis(); // Confirm cancellation in dialog page.on('dialog', dialog => dialog.accept()); // Should redirect back to home await expect(page).toHaveURL('/'); // Should show cancellation toast const assertions = new CustomAssertions(page); await assertions.expectToast('Analysis cancelled', 'info'); }); test('should not allow cancellation of completed analysis', async ({ page }) => { // Navigate to completed analysis await page.goto('/analysis/completed-123'); const progressPage = new AnalysisProgressPage(page); // Cancel button should be disabled or hidden await expect(progressPage.cancelButton).not.toBeVisible(); }); }); ``` ### 5. Responsive & Mobile ```typescript // tests/e2e/responsive.spec.ts import { test, expect, devices } from '@playwright/test'; import { HomePage } from './pages/HomePage'; test.describe('Responsive Design', () => { test.use({ ...devices['iPhone 13'] }); test('should work on mobile viewport', async ({ page }) => { const homePage = new HomePage(page); await homePage.goto(); // URL input should be visible and usable await expect(homePage.urlInput).toBeVisible(); await homePage.urlInput.fill('https://example.com/mobile-test'); // Button should be tappable await homePage.analyzeButton.click(); // Progress page should be mobile-friendly const progressBar = page.getByTestId('analysis-progress-bar'); await expect(progressBar).toBeVisible(); // Agent cards should stack vertically const agentCards = page.getByTestId('agent-card'); const firstCard = agentCards.first(); const secondCard = agentCards.nth(1); const firstBox = await firstCard.boundingBox(); const secondBox = await secondCard.boundingBox(); // Second card should be below first (Y coordinate) expect(secondBox!.y).toBeGreaterThan(firstBox!.y + firstBox!.height); }); }); ``` ### 6. Accessibility ```typescript // tests/e2e/accessibility.spec.ts import { test, expect } from '@playwright/test'; import { HomePage } from './pages/HomePage'; import { AnalysisProgressPage } from './pages/AnalysisProgressPage'; test.describe('Accessibility', () => { test('should be keyboard navigable', async ({ page }) => { const homePage = new HomePage(page); await homePage.goto(); // Tab to URL input await page.keyboard.press('Tab'); await expect(homePage.urlInput).toBeFocused(); // Type URL await page.keyboard.type('https://example.com/test'); // Tab to analyze button await page.keyboard.press('Tab'); await expect(homePage.analyzeButton).toBeFocused(); // Press Enter to submit await page.keyboard.press('Enter'); // Should navigate to progress page const progressPage = new AnalysisProgressPage(page); await expect(progressPage.progressBar).toBeVisible(); }); test('should have proper ARIA labels', async ({ page }) => { const homePage = new HomePage(page); await homePage.goto(); // URL input should have aria-label await expect(homePage.urlInput).toHaveAttribute('aria-label'); // Submit button should have accessible name const buttonName = await homePage.analyzeButton.getAttribute('aria-label'); expect(buttonName).toBeTruthy(); }); test('should announce progress updates to screen readers', async ({ page }) => { await page.goto('/analysis/123'); const progressPage = new AnalysisProgressPage(page); // Progress region should have aria-live await expect(progressPage.progressBar).toHaveAttribute('aria-live', 'polite'); // Status updates should have role="status" const statusRegion = page.getByTestId('status-updates'); await expect(statusRegion).toHaveAttribute('role', 'status'); }); }); ``` ## Running Tests ```bash # Install Playwright npm install -D @playwright/test npx playwright install # Run all tests npx playwright test # Run specific suite npx playwright test tests/e2e/analysis-flow.spec.ts # Run in UI mode (interactive) npx playwright test --ui # Run in headed mode (see browser) npx playwright test --headed # Run on specific browser npx playwright test --project=chromium # Debug mode npx playwright test --debug # Generate test report npx playwright show-report ``` ## CI Integration ```yaml # .github/workflows/e2e-tests.yml name: E2E Tests on: [push, pull_request] jobs: test: runs-on: ubuntu-latest services: postgres: image: postgres:15 env: POSTGRES_PASSWORD: postgres options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 redis: image: redis:7 options: >- --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5 steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '22' - name: Install dependencies run: npm ci - name: Install Playwright browsers run: npx playwright install --with-deps - name: Start backend run: | cd backend poetry install poetry run uvicorn app.main:app --host 0.0.0.0 --port 8500 & sleep 5 - name: Start frontend run: | npm run build npm run preview & sleep 3 - name: Run E2E tests run: npx playwright test - name: Upload test results if: always() uses: actions/upload-artifact@v3 with: name: playwright-report path: playwright-report/ retention-days: 30 ``` ## Best Practices 1. **Use Page Objects** - Encapsulate page logic, improve maintainability 2. **Mock External APIs** - Fast, reliable tests without network dependencies 3. **Wait Strategically** - Use `waitForSelector`, avoid arbitrary timeouts 4. **Test Real Flows** - Mirror actual user journeys 5. **Handle Async** - SSE streams, debounced inputs, loading states 6. **Accessibility First** - Test keyboard nav, ARIA, screen reader announcements 7. **Visual Regression** - Screenshot testing for UI consistency 8. **CI Integration** - Run tests on every PR, block merges on failures