---
title: "Expect"
description: "Diff-aware AI browser testing, reads the git diff, maps changes to affected pages via the route map, generates a targeted test plan, and executes it via agent-browser (Rust daemon + CDP, ARIA-tree-first) with pass/fail reporting. Use when testing UI changes, verifying PRs before merge, or running regression checks on changed components."
canonical: "https://orchestkit.yonyon.ai/docs/reference/skills/expect"
---

# Expect

Diff-aware AI browser testing, reads the git diff, maps changes to affected pages via the route map, generates a targeted test plan, and executes it via agent-browser (Rust daemon + CDP, ARIA-tree-first) with pass/fail reporting. Use when testing UI changes, verifying PRs before merge, or running regression checks on changed components.

<span className="badge badge-blue">Command</span> <span className="badge badge-orange">high</span>

```bash title="Invoke"
/ork:expect
```

<ContextualSkillSidebar slug="expect" />

> **Expect** Diff-aware AI browser testing, reads the git diff, maps changes to affected pages via the route map, generates a targeted test plan, and executes it via agent-browser (Rust daemon + CDP, ARIA-tree-first) with pass/fail reporting. Use when testing UI changes, verifying PRs before merge, or running regression checks on changed components.


# Expect: Diff-Aware AI Browser Testing

Analyze git changes, generate targeted test plans, and execute them via AI-driven browser automation.

> **Note:** If `disableSkillShellExecution` is enabled (CC 2.1.91), the agent-browser install check won't run. Verify it's installed: `npx agent-browser --version`.

```bash
expect                              # Auto-detect changes, test affected pages
expect -m "test the checkout flow"  # Specific instruction
expect --flow login                 # Replay a saved test flow
expect --target branch              # Test all changes on current branch vs main
expect -y                           # Skip plan review, run immediately
```

**Core principle:** Only test what changed. Git diff drives scope, no wasted cycles on unaffected pages.


## Argument Resolution

```python
ARGS = "[-m <instruction>] [--target unstaged|branch|commit] [--flow <slug>] [-y]"

# Parse from full argument string
import re
raw = ""  # Full argument string from CC

INSTRUCTION = None
TARGET = "unstaged"  # Default: test unstaged changes
FLOW = None
SKIP_REVIEW = False

# Extract -m "instruction"
m_match = re.search(r'-m\s+["\']([^"\']+)["\']|-m\s+(\S+)', raw)
if m_match:
    INSTRUCTION = m_match.group(1) or m_match.group(2)

# Extract --target
t_match = re.search(r'--target\s+(unstaged|branch|commit)', raw)
if t_match:
    TARGET = t_match.group(1)

# Extract --flow
f_match = re.search(r'--flow\s+(\S+)', raw)
if f_match:
    FLOW = f_match.group(1)

# Extract -y
if '-y' in raw.split():
    SKIP_REVIEW = True
```


## STEP 0: MCP Probe + Prerequisite Check

```python
# memory is alwaysLoad in .mcp.json (CC 2.1.121+, #1541) — probe below kept as fallback for older CC:
ToolSearch(query="select:mcp__memory__search_nodes")

# Verify agent-browser is available (Rust-native, no Playwright)
Bash("command -v agent-browser || npx agent-browser --version")
# If missing: "Install agent-browser: npm i -g agent-browser"

# Load agent-browser's version-matched workflow guide (ships with the CLI).
# NOT `skills get agent-browser` — that resolves but returns only the thin
# top-level router; `core --full` is the actual 2,800+ line command reference.
Bash("agent-browser skills get core --full")
```


## CRITICAL: Task Management

```python
# 1. Create main task IMMEDIATELY
TaskCreate(
  subject="Expect: test changed code",
  description="Diff-aware browser testing pipeline",
  activeForm="Running diff-aware browser tests"
)

# 2. Create subtasks for each pipeline phase
TaskCreate(subject="Check fingerprint (skip if unchanged)", activeForm="Checking fingerprint")  # id=2
TaskCreate(subject="Scan git diff and classify changes", activeForm="Scanning diff")            # id=3
TaskCreate(subject="Map changes to routes/URLs", activeForm="Mapping routes")                   # id=4
TaskCreate(subject="Generate AI test plan", activeForm="Generating test plan")                   # id=5
TaskCreate(subject="Execute tests via agent-browser", activeForm="Executing browser tests")     # id=6
TaskCreate(subject="Compile test report", activeForm="Compiling report")                        # id=7

# 3. Set dependencies for sequential phases
TaskUpdate(taskId="3", addBlockedBy=["2"])  # Diff scan needs fingerprint check
TaskUpdate(taskId="4", addBlockedBy=["3"])  # Route map needs diff results
TaskUpdate(taskId="5", addBlockedBy=["4"])  # Test plan needs route map
TaskUpdate(taskId="6", addBlockedBy=["5"])  # Execution needs test plan
TaskUpdate(taskId="7", addBlockedBy=["6"])  # Report needs execution results

# 4. Update status as you progress
TaskUpdate(taskId="2", status="in_progress")  # When starting
TaskUpdate(taskId="2", status="completed")    # When done — repeat for each subtask
```


## Pipeline Overview

```
Git Diff → Route Map → Fingerprint Check → Test Plan → Execute → Report
```

| Phase | What | Output | Reference |
|-------|------|--------|-----------|
| **1. Fingerprint** | SHA-256 hash of changed files | Skip if unchanged since last run | `references/fingerprint.md` |
| **2. Diff Scan** | Parse git diff, classify changes | ChangesFor data (files, components, routes) | `references/diff-scanner.md` |
| **3. Route Map** | Map changed files to affected pages/URLs | Scoped page list | `references/route-map.md` |
| **4. Test Plan** | Generate AI test plan from diff + route map | Markdown test plan with steps | `references/test-plan.md` |
| **5. Execute** | Run test plan via agent-browser | Pass/fail per step, screenshots | `references/execution.md` |
| **6. Report** | Aggregate results, artifacts, exit code | Structured report + artifacts | `references/report.md` |


## Phase 1: Fingerprint Check

Check if the current changes have already been tested:

```python
Read(".expect/fingerprints.json")  # Previous run hashes
# Compare SHA-256 of changed files against stored fingerprints
# If match: "No changes since last test run. Use --force to re-run."
# If no match or --force: continue to Phase 2
```

Load: `Read("references/fingerprint.md")`


## Phase 2: Diff Scan

Analyze git changes based on `--target`:

```python
if TARGET == "unstaged":
    diff = Bash("git diff")
    files = Bash("git diff --name-only")
elif TARGET == "branch":
    diff = Bash("git diff main...HEAD")
    files = Bash("git diff main...HEAD --name-only")
elif TARGET == "commit":
    diff = Bash("git diff HEAD~1")
    files = Bash("git diff HEAD~1 --name-only")
```

Classify each changed file into 3 levels:
1. **Direct**: the file itself changed
2. **Imported**: a file that imports the changed file
3. **Routed**: the page/route that renders the changed component

Load: `Read("references/diff-scanner.md")`


## Phase 3: Route Map

Map changed files to testable URLs using `.expect/config.yaml`:

```yaml
# .expect/config.yaml
base_url: http://localhost:3000
route_map:
  "src/components/Header.tsx": ["/", "/about", "/pricing"]
  "src/app/auth/**": ["/login", "/signup", "/forgot-password"]
  "src/app/dashboard/**": ["/dashboard"]
```

If no route map exists, infer from Next.js App Router / Pages Router conventions.

Load: `Read("references/route-map.md")`


## Phase 4: Test Plan Generation

Build an AI test plan scoped to the diff, using the scope strategy for the current target:

```python
scope_strategy = get_scope_strategy(TARGET)  # See references/scope-strategy.md

prompt = f"""
{scope_strategy}

Changes: {diff_summary}
Affected pages: {affected_urls}
Instruction: {INSTRUCTION or "Test that the changes work correctly"}

Generate a test plan with:
1. Page-level checks (loads, no console errors, correct content)
2. Interaction tests (forms, buttons, navigation affected by the diff)
3. Visual regression (compare ARIA snapshots if saved)
4. Accessibility (axe-core scan on affected pages)
"""
```

If `--flow` specified, load saved flow from `.expect/flows/{slug}.yaml` instead of generating.

If NOT `--y`, present plan to user via `AskUserQuestion` for review before executing.

Load: `Read("references/test-plan.md")`


## Phase 5: Execution

### agent-browser Quick Primer

> Floor is `>= 0.31.1` (0.27.1 is documented broken on prod pages); current tested release is **0.38.1** (see `upstream-version-tested`). Commands below hold across this range. 0.30+ adds `agent-browser read` (agent-readable text extraction) and the `--restore` / `--namespace` session-restore workflow for stable, isolated browser state across agent runs. 0.33.0 adds `agent-browser a11y [url]`, an embedded axe-core audit (WCAG tag filtering, selector scoping, iframe-aware text/JSON output) available as both a CLI command and an MCP tool. 0.34.0 adds persistent session-to-tab binding for shared Chrome sessions, with `--pin-tab` making the binding strict so an externally closed tab yields a stable `tab_gone` error. 0.35.0 adds `--ca-cert <path>` for a trusted local CA behind an SSL-inspecting proxy (Linux-only; the targeted alternative to `--ignore-https-errors`). 0.35.1 changes behaviour the Diff row below relies on: `diff snapshot` now resets ref numbering per diff and invalidates refs across navigations, so refs captured before a navigation must be re-read rather than reused. 0.36.0 adds experimental WebMCP tool discovery (`webmcp list/invoke/result/cancel`, brief untrusted catalog summaries on `open`). 0.37.0 makes `record` capture the active page at 30 fps and inherits session setup into new tabs. 0.38.0 adds `snapshot --delta` (full baseline, then `unchanged` or a compact structural change), `screenshot --if-changed --threshold`, persistent refs that survive same-document DOM changes, `--human` / `--input-mode` pointer realism, and `auth login --no-navigate`. Full command surface: `references/upstream.md` and `agent-browser skills get core --full`.


| Area | Command | Notes |
|------|---------|-------|
| Snapshot | `agent-browser snapshot -i` | ARIA tree w/ `@eN` refs. `-C`/`--cursor` was removed in 0.22 |
| Snapshot delta *(0.38+)* | `agent-browser snapshot --delta` | Full tree once, then `unchanged` or a compact structural diff; refs persist across same-document changes |
| Semantic locator | `agent-browser find role button click --name "Continue"` | Grammar: `find <locator> <value> [action]`; stable alternative to `@eN` refs |
| Interaction | `fill @e1 "..."`, `click @e2`, `press Enter`, `drag @e1 @e2`, `upload @e1 file.pdf` | All take ARIA refs. Add `--human` for a curved approach instead of an instant jump |
| Waits | `wait --load networkidle`, `wait --text "Success"`, `wait --fn "window.ready"` | Event-driven, never sleep-based |
| Network | `network route "*analytics*" --abort`, `network route "https://api/*" --body '{...}'` | Intercept + stub |
| State | `state save/load auth.json`, `--session-name <name>` | Persist auth across runs |
| Vault | `vault store github_pat`, `vault load github_pat` | Encrypted credential store |
| Diff | `diff snapshot`, `diff screenshot --baseline /tmp/x.png` | ARIA + pixel diffing across separate runs; prefer `snapshot --delta` for within-run before/after on the same page |
| Capture | `screenshot --annotate`, `screenshot --if-changed --threshold <n>` *(0.38+)*, `pdf`, `record start/stop --fps <n>` | `--if-changed` skips re-encoding an unchanged screenshot; `record` defaults to 30 fps (0.37+) |
| Dashboard | `agent-browser dashboard start` *(0.25+)* | Browser-side runtime inspector on :4848 |

### Run the test plan

```python
expect_task = Agent(
  subagent_type="ork:expect-agent",
  prompt=f"""Execute this test plan:
  {test_plan}

  For each step:
  1. Navigate to the URL
  2. Execute the test action
  3. Take a screenshot on failure
  4. Report PASS/FAIL with evidence
  """,
  run_in_background=True,
  model="sonnet",
  max_turns=50
)

# Stream agent-browser progress line-by-line instead of polling (CC 2.1.98+)
# Each stdout line from agent-browser arrives as a notification — useful for
# catching a failing step early rather than waiting for the full plan.
# Full pattern: Read("${CLAUDE_PLUGIN_ROOT}/skills/chain-patterns/references/monitor-patterns.md")
Monitor(pid=expect_task.agent_id)

# For long test plans (>3 min typical), notify on completion — requires
# Remote Control + "Push when Claude decides" config (CC 2.1.110+).
# Skip silently if the user doesn't have Remote Control enabled.
if test_plan_duration_estimate > 180:
    PushNotification(
        message=f"ork:expect complete — {passed}/{total} steps passed on {len(affected_urls)} pages",
        status="proactive"
    )
```

Load: `Read("references/execution.md")`


## Phase 6: Report

```
expect Report
═══════════════════════════════════════
Target: unstaged (3 files changed)
Pages tested: 4
Duration: 45s

Results:
  ✓ /login — form renders, submit works
  ✓ /signup — validation triggers on empty fields
  ✗ /dashboard — chart component crashes (TypeError)
  ✓ /settings — preferences save correctly

3 passed, 1 failed

Artifacts:
  .expect/reports/2026-03-26T16-30-00.json
  .expect/screenshots/dashboard-error.png
```

Load: `Read("references/report.md")`


## Saved Flows

Reusable test sequences stored in `.expect/flows/`:

```yaml
# .expect/flows/login.yaml
name: Login Flow
steps:
  - navigate: /login
  - fill: { selector: "#email", value: "test@example.com" }
  - fill: { selector: "#password", value: "password123" }
  - click: button[type="submit"]
  - assert: { url: "/dashboard" }
  - assert: { text: "Welcome back" }
```

Run with: `expect --flow login`


## Auto-trigger after UI edits (M125 #2)

When the dev stack is live (`dev`), saving any `.tsx`, `.jsx`, `.css`, or `.scss` file (and Next.js route files like `app/**/page.tsx`, `pages/**/*.tsx`) emits a nudge to run `expect <route>`. The hook (`posttool/ui-change-detector`) is **default-on** and:

- skips silently if `dev` hasn't booted (no agent-browser session to attach to);
- enforces a 30-second cooldown per route to prevent spam on rapid saves;
- honors `.claude/state/expect-skip.<sessionId>` as a per-session opt-out (write any content);
- honors `ORK_EXPECT_AUTO=0` for an env-level kill switch.

Route resolution: `app/dashboard/page.tsx` → `/dashboard`, `pages/settings.tsx` → `/settings`, component / global-style edits → `/` (home as proxy). Route groups like `app/(marketing)/pricing/page.tsx` strip to `/pricing`.

## ARIA snapshot recording (M125 #6)

After a passing run, the `posttool/expect/snapshot-recorder` hook persists the captured ARIA tree to `.claude/state/expect-snapshots/<route-slug>/<parent-commit>.json`. Subsequent `expect <route> --diff` runs compare against the most recent prior snapshot for that route, surfaces structural regressions (added/removed buttons, label changes, hierarchy shifts) without needing a baseline screenshot.

For the snapshot recorder to fire, the expect run output must contain `RUN_COMPLETED|passed`, `ROUTE|<route>`, and `ARIA|<json-summary>` tags. The agent-browser-driven flow already emits these.

## Jev step judge (opt-in, three modes)

`ORK_EXPECT_JEV` selects the mode. Unset or falsey is off, and off means zero network. `shadow` (also selected by a truthy `ORK_EXPECT_JEV_SHADOW` for compatibility) logs one Jev request per step beside the agent's pick: a Choice over the bounded legal-action set built from the interactive ARIA elements, plus three verify Nouls, with probabilities and per-step agreement in the report. `1` or `act` lets the Jev pick drive the step instead, failing closed to the agent's own pick on any error, timeout, empty or malformed answer, or a confidence below `act_confidence_floor`. Tunables live in `.expect/config.yaml` `jev_shadow:`. Load: `Read("references/jev-shadow.md")`


## When NOT to Use

- **Unit tests**: use `cover` instead
- **API-only changes**: no browser UI to test
- **Generated files**: skip build artifacts, lock files
- **Docs-only changes**: unless you want to verify docs site rendering


## Quality Bar

Done means all of these hold:
- Test-plan scope is derived from the git diff for the chosen `--target`, no unaffected page appears in the plan.
- Every changed file maps to at least one tested route (via `.expect/config.yaml` or inferred convention) OR is explicitly excluded as API-only, generated, or docs-only.
- Fingerprint check runs first; a diff unchanged since the last run skips execution instead of re-testing.
- Unless `-y` is passed, the plan is presented for review before any browser action runs.
- Each executed step reports PASS or FAIL with evidence (screenshot on failure), and the report's pass/fail totals match the steps actually run.
- The report's exit code is non-zero whenever any step failed.

Budget: at most 30 minutes total and 5 minutes per page (a slower page is skipped, per `references/test-plan.md`; defaults, `.expect/config.yaml` overrides), 50 executor turns (`max_turns=50`), one sequential `ork:expect-agent`, never parallel browsers (`rules/no-parallel-browsers.md`); stop and report partial results at the finish line or the first cap, whichever comes first.

## Related Skills

- `agent-browser`: Browser automation engine (required dependency)
- `ork:cover`: Test suite generation (unit/integration/e2e)
- `ork:verify`: Grade existing test quality
- `testing-e2e`: Playwright patterns and best practices


## References

Load on demand with `Read("references/<file>")`:

| File | Content |
|------|---------|
| `fingerprint.md` | SHA-256 gating logic |
| `diff-scanner.md` | Git diff parsing + 3-level classification |
| `route-map.md` | File-to-URL mapping conventions |
| `test-plan.md` | AI test plan generation prompt templates |
| `execution.md` | agent-browser orchestration patterns |
| `report.md` | Report format + artifact storage |
| `config-schema.md` | .expect/config.yaml full schema |
| `aria-diffing.md` | ARIA snapshot comparison for semantic diffing |
| `scope-strategy.md` | Test depth strategy per target mode |
| `saved-flows.md` | Markdown+YAML flow format, adaptive replay |
| `rrweb-recording.md` | rrweb DOM replay integration |
| `human-review.md` | AskUserQuestion plan review gate |
| `ci-integration.md` | GitHub Actions workflow + pre-push hooks |
| `research.md` | millionco/expect architecture analysis |
| `jev-shadow.md` | Opt-in Jev step judge per step: bounded-action Choice + verify Nouls; shadow logs, act drives with fail-closed fallback |


**Version:** 1.0.0 (March 2026), Initial scaffold, M99 milestone


---

## Rules (5)

### Artifact storage conventions for reports, screenshots, and fingerprints: MEDIUM


## Artifact Storage

All expect artifacts live under `.expect/` with a consistent directory structure.

**Incorrect, scattered artifact locations:**
```bash
# Wrong: artifacts in random locations
/tmp/test-screenshot-1.png
~/Desktop/test-report.json
./screenshots/login-fail.png
```

**Correct, structured under .expect/:**
```
.expect/
├── config.yaml              # Project config (committed)
├── flows/                   # Saved test flows (committed)
│   ├── login.yaml
│   └── checkout.yaml
├── fingerprints.json         # SHA-256 hashes (gitignored)
├── reports/                  # Test run reports (gitignored)
│   ├── 2026-03-26T16-30-00.json
│   └── 2026-03-26T17-00-00.json
├── screenshots/              # Failure screenshots (gitignored)
│   ├── dashboard-step2-fail.png
│   └── login-step5-fail.png
└── snapshots/                # ARIA snapshots (committed)
    ├── login.json
    └── dashboard.json
```

**Key rules:**
- Reports use ISO timestamp filenames (UTC, replace `:` with `-`)
- Keep last N reports (default 10, configurable in config.yaml)
- Screenshots only on failure (`on_fail` default)
- ARIA snapshots and flows are committed (they're baseline references)
- Fingerprints, reports, and screenshots are gitignored (ephemeral)


### Scope test runs to changed code only: HIGH


## Diff Scope Boundaries

Only test pages that are connected to the changed files via the 3-level classification.

**Incorrect, testing all pages regardless of diff:**
```python
# Wrong: testing entire site when only Button.tsx changed
pages_to_test = ["/", "/about", "/pricing", "/dashboard", "/settings", "/login"]
```

**Correct, scoped to affected routes:**
```python
# Right: only test pages that render the changed component
changed = ["src/components/Button.tsx"]
direct = changed                                    # Level 1
imported = find_importers("Button", "src/")         # Level 2
routed = route_map.resolve(direct + imported)       # Level 3
pages_to_test = routed  # ["/", "/dashboard"] — only pages using Button
```

**Key rules:**
- Always run diff scan before route mapping, never assume scope
- If route map is empty (no `.expect/config.yaml`, no framework detected), test only `base_url` root
- Log which level triggered each page test for debugging
- Respect `ignore_patterns` from config, skip test files, docs, lockfiles


### When to invalidate fingerprints and force re-run: HIGH


## Fingerprint Invalidation

Fingerprints must be invalidated when file contents change outside the normal edit flow.

**Incorrect, trusting fingerprints after git operations:**
```python
# Wrong: fingerprints match but code is completely different branch
git checkout feature-branch  # Different code
/ork:expect                  # "No changes since last run" — WRONG
```

**Correct, invalidate on state-changing git operations:**
```python
# Right: clear fingerprints when git state changes
INVALIDATION_TRIGGERS = [
    "git checkout",    # Different branch = different code
    "git stash pop",   # Restored changes
    "git merge",       # Merged code from another branch
    "git rebase",      # Rebased commits
    "git reset",       # Reset to different state
    "git pull",        # Pulled upstream changes
]
# After any of these: delete .expect/fingerprints.json
```

**Key rules:**
- Hash file **contents** (sha256sum), not metadata (mtime)
- Store fingerprints per target (unstaged/branch/commit), don't mix
- Always re-run if last result was `fail` (even if fingerprints match)
- `--force` flag bypasses fingerprint check entirely
- `.expect/fingerprints.json` should be in `.gitignore`


### Sequential browser testing: no parallel page visits: CRITICAL


## No Parallel Browsers

Always test pages sequentially in a single browser session.

**Incorrect, parallel browser sessions:**
```python
# Wrong: multiple agents hitting the same app simultaneously
Agent(prompt="Test /login", run_in_background=True)
Agent(prompt="Test /dashboard", run_in_background=True)
Agent(prompt="Test /settings", run_in_background=True)
# Risk: shared cookies, race conditions, port conflicts
```

**Correct, single agent, sequential navigation:**
```python
# Right: one agent tests all pages in sequence
Agent(prompt="""Test these pages in order:
  1. /login
  2. /dashboard
  3. /settings
Navigate between them sequentially. Do not open multiple tabs.""")
```

**Key rules:**
- One browser session per test run
- Navigate sequentially between pages
- Clear cookies/state between unrelated page groups if needed
- If app requires auth, login once and reuse the session
- Never spawn parallel browser agents for the same base_url


### Timeout and retry conventions for browser test execution: CRITICAL


## Timeout and Retry

Set explicit timeouts for every browser operation and retry transient failures exactly once.

**Incorrect, no timeout, no retry:**
```python
# Wrong: waits forever if element doesn't exist
await page.click("#submit-button")
# Wrong: fails immediately on slow network
assert page.url == "/dashboard"
```

**Correct, explicit timeouts with single retry:**
```python
# Right: 10s timeout for element interaction
await page.click("#submit-button", timeout=10000)

# Right: wait for navigation with timeout
await page.wait_for_url("/dashboard", timeout=15000)

# Right: retry once on element-not-found
try:
    await page.click("#submit-button", timeout=5000)
except ElementNotFound:
    await page.wait_for_timeout(2000)  # Wait 2s
    await page.click("#submit-button", timeout=5000)  # One retry
```

**Timeout defaults:**

| Operation | Timeout | Retry |
|-----------|---------|-------|
| Page navigation | 15s | 1x |
| Element click/fill | 10s | 1x after 2s wait |
| Assertion | 5s | No retry |
| Page crash (5xx) | - | Skip remaining steps on page |
| Network timeout | 15s | 1x |



---

## References (15)

### Aria Diffing

# ARIA Snapshot Diffing

Semantic UI change detection using ARIA tree snapshots instead of pixel-based visual regression.

## Why ARIA Over Screenshots

| Approach | Pros | Cons |
|----------|------|------|
| Screenshot diff | Catches visual regressions | Brittle (font rendering, anti-aliasing, viewport), large files |
| ARIA snapshot | Semantic, tiny diffs, framework-agnostic | Misses purely visual changes (colors, spacing) |

ARIA diffing catches **structural and semantic** changes, missing labels, changed hierarchy, removed interactive elements, which are the changes most likely to break user experience.

## Snapshot Format

```json
{
  "page": "/login",
  "timestamp": "2026-03-26T16:30:00Z",
  "tree": {
    "role": "main",
    "name": "Login",
    "children": [
      {
        "role": "heading",
        "name": "Sign In",
        "level": 1
      },
      {
        "role": "form",
        "name": "Login form",
        "children": [
          { "role": "textbox", "name": "Email" },
          { "role": "textbox", "name": "Password" },
          { "role": "button", "name": "Sign In" }
        ]
      }
    ]
  }
}
```

## Capturing Snapshots

Via agent-browser:
```
Navigate to /login
Run: document.querySelector('main').computedRole  // or use axe-core
Extract ARIA tree as JSON
Save to .expect/snapshots/login.json
```

## Diffing Algorithm

1. Load previous snapshot from `.expect/snapshots/{page-slug}.json`
2. Capture current ARIA tree
3. Compute structural diff:
   - Added nodes (new elements)
   - Removed nodes (deleted elements)
   - Changed names/roles (label changes)
   - Reordered children (layout changes)
4. Score the diff as a percentage of total nodes changed
5. Flag if above `diff_threshold` (default 10%)

## Diff Output

```
ARIA Diff: /login
  + Added: textbox "Confirm Password" (new field)
  - Removed: link "Forgot Password?" (was in form)
  ~ Changed: button "Sign In" → "Log In" (label changed)

Change score: 15% (threshold: 10%) — FLAGGED
```


### Ci Integration

# CI Integration (#1180)

Run /ork:expect in GitHub Actions and pre-push hooks.

## GitHub Actions Workflow

```yaml
# .github/workflows/expect.yml
name: Browser Tests (expect)
on:
  pull_request:
    branches: [main]

jobs:
  expect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for git diff

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies
        run: npm ci

      - name: Start dev server
        run: npm run dev &
        env:
          PORT: 3000

      - name: Wait for server
        run: npx wait-on http://localhost:3000 --timeout 30000

      - name: Install Claude Code + OrchestKit
        run: |
          npm install -g @anthropic-ai/claude-code@latest
          claude plugin install orchestkit/ork

      - name: Run expect
        run: |
          claude "/ork:expect --target branch -y"
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Upload artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: expect-results
          path: |
            .expect/reports/
            .expect/screenshots/
            .expect/recordings/
```

## Pre-Push Hook

```bash
# .git/hooks/pre-push (or via husky/lefthook)
#!/usr/bin/env bash
set -euo pipefail

# Quick fingerprint check — skip if no changes
if bash scripts/expect/fingerprint.sh check >/dev/null 2>&1; then
  echo "expect: No changes since last test run — skipping"
  exit 0
fi

# Run expect with branch target, skip review
claude "/ork:expect --target branch -y"
```

## Exit Code Mapping

| /ork:expect Exit | CI Behavior |
|-----------------|-------------|
| `0` (all pass) | CI passes |
| `0` (skip, fingerprint) | CI passes (zero-cost) |
| `1` (test failure) | CI fails, artifacts uploaded |
| `0` + warning (env issue) | CI passes with warning annotation |

## Environment Variables

| Variable | Required | Purpose |
|----------|----------|---------|
| `ANTHROPIC_API_KEY` | Yes | Claude API access |
| `CI` | Auto-set | Detected by expect, enables CI output mode |
| `GITHUB_ACTIONS` | Auto-set | Enables GitHub annotations format |

## Cost Optimization

- Fingerprint gating: zero-cost when nothing changed
- Scope strategy: `branch` target in CI limits test count
- `-y` flag: skip human review in automated pipelines
- `--target branch`: only test branch changes, not full site


### Config Schema

# .expect/config.yaml Schema

Project-level configuration for /ork:expect.

## Full Schema

```yaml
# .expect/config.yaml

# Base URL for the application under test
base_url: http://localhost:3000

# Dev server start command (optional — expect can start it for you)
dev_command: npm run dev
dev_ready_pattern: "ready on"  # Pattern in stdout that signals server is ready
dev_timeout: 30                # Seconds to wait for dev server

# File-to-URL route mapping
route_map:
  "src/components/Header.tsx": ["/", "/about", "/pricing"]
  "src/app/auth/**": ["/login", "/signup", "/forgot-password"]
  "src/app/dashboard/**": ["/dashboard"]
  "src/app/settings/**": ["/settings"]

# Test parameters for dynamic routes
test_params:
  slug: "test-post"
  id: "1"
  username: "testuser"

# Auth configuration for protected pages
auth:
  strategy: cookie          # cookie | bearer | basic
  login_url: /login
  credentials:
    email: test@example.com
    password: from_env:TEST_PASSWORD  # Read from environment variable

# ARIA snapshot settings
aria_snapshots:
  enabled: true
  storage: .expect/snapshots/
  diff_threshold: 0.1  # 10% change tolerance before flagging

# Accessibility settings
accessibility:
  enabled: true
  standard: wcag2aa     # wcag2a | wcag2aa | wcag2aaa
  ignore_rules: []      # axe-core rule IDs to skip

# Report settings
reports:
  storage: .expect/reports/
  keep_last: 10         # Number of reports to retain
  screenshots: on_fail  # always | on_fail | never

# Jev step judge (see references/jev-shadow.md)
# Runs one Jev request per step when ORK_EXPECT_JEV selects a mode:
# `shadow` logs beside the agent pick, `1`/`act` runs the Jev pick with a
# fail-closed fallback to the agent pick. Unset or falsey is off;
# ORK_EXPECT_JEV_SHADOW truthy still selects shadow.
# Every key below has a shipped fallback in the skill's
# jev-shadow.defaults.yaml; set this section to override it wholesale.
jev_shadow:
  endpoint: "https://api.typesafe.ai/v1/systemone"
  model: "jev-1.13.0"       # pinned: a shadow comparison needs a fixed model
  element_cap: 60           # max interactive elements per request (ref+role+name)
  name_max_chars: 80        # per-name truncation after redaction
  context_max_chars: 300    # per-field truncation for goal and last verify
  latency_budget_ms: 400    # abort budget for the one request per step
  act_confidence_floor: 0.5 # act mode: Choice confidence below this falls back
  thresholds:
    low_confidence: 0.5     # Choice confidence below this flags the pick
    goal_reached: 0.7       # noul >= threshold raises the matching flag
    page_changed_as_expected: 0.7
    blocked: 0.7

# Files to ignore in diff scanning
ignore_patterns:
  - "**/*.test.*"
  - "**/*.spec.*"
  - "*.md"
  - "*.json"
  - "package-lock.json"
  - ".env*"
```

## Minimal Config

```yaml
base_url: http://localhost:3000
```

Everything else has sensible defaults or is inferred from the framework.

## Environment Variable Injection

Use `from_env:VAR_NAME` syntax for sensitive values:

```yaml
auth:
  credentials:
    password: from_env:TEST_PASSWORD
    api_key: from_env:TEST_API_KEY
```


### Diff Scanner

# Diff Scanner

Parse git diff output into 3 concurrent data levels for test targeting.

## Target Modes (ChangesFor)

| Mode | Git Command | Use Case |
|------|-------------|----------|
| `changes` (default) | `git diff $(merge-base)` | All changes, committed + uncommitted |
| `unstaged` | `git diff` | Only uncommitted working tree changes |
| `branch` | `git diff main...HEAD` | Full branch diff vs main |
| `commit [hash]` | `git diff {hash}^..{hash}` | Single commit |

## 3 Data Levels (Gathered Concurrently)

### Level 1: Changed Files
```bash
git diff --name-only --diff-filter=AMDRC
```
Returns file paths with status: Added, Modified, Deleted, Renamed, Copied.

Each file is typed: `component`, `logic`, `style`, `docs`, `config`, `test`, `script`, `python`, `other`.

### Level 2: File Stats
```bash
git diff --numstat
```
Returns lines added/removed per file + computed `magnitude` (added + removed) for prioritization.

### Level 3: Diff Preview
Full unified diff, truncated to 12K chars. Files are prioritized by magnitude (most changed first), limited to 12 files max.

## Usage

```bash
bash scripts/diff-scan.sh                    # Default: changes mode
bash scripts/diff-scan.sh unstaged           # Uncommitted only
bash scripts/diff-scan.sh branch             # Branch vs main
bash scripts/diff-scan.sh commit abc123f     # Specific commit
```

## Output Format

```json
{
  "target": "branch",
  "files": [
    {"path": "src/components/Button.tsx", "status": "modified", "type": "component"},
    {"path": "src/app/login/page.tsx", "status": "added", "type": "component"}
  ],
  "stats": [
    {"path": "src/components/Button.tsx", "added": 15, "removed": 3, "magnitude": 18},
    {"path": "src/app/login/page.tsx", "added": 45, "removed": 0, "magnitude": 45}
  ],
  "preview": "--- src/app/login/page.tsx ---\n+export default function Login()...",
  "context": [
    "abc123f feat: add login page",
    "def456a fix: button hover state"
  ],
  "summary": {
    "total": 2,
    "top_files_in_preview": 12,
    "preview_chars": 1234,
    "max_preview_chars": 12000
  }
}
```

## 3-Level Classification (Import Graph)

After the diff scan, the expect pipeline classifies each changed file:

| Level | Name | How to Find | Test Depth |
|-------|------|-------------|------------|
| 1 | **Direct** | `git diff --name-only` output | Full interaction tests |
| 2 | **Imported** | `grep -rl "from.*{module}" src/` | Render check + basic interaction |
| 3 | **Routed** | Route map lookup (config or inference) | Page load + smoke test |

## Filtering

Non-source files are automatically skipped:
- Lock files (`.lock`, `.log`, `.map`)
- `node_modules/`, `.git/`, `dist/`, `build/`
- Configure additional patterns in `.expect/config.yaml` `ignore_patterns`

## Magnitude Prioritization

When more than 12 files changed, the preview includes only the top 12 by magnitude (lines added + removed). This ensures the AI test plan focuses on the most impactful changes.


### Execution

# Execution Engine (#1175)

Run test plans via agent-browser with session management, auth profiles, and failure handling.

## Execution Flow

```
1. Load auth profile (if configured)
2. For each page in test plan:
   a. Open URL via agent-browser
   b. Take pre-test ARIA snapshot
   c. Execute test steps with status protocol
   d. Take post-test ARIA snapshot (for diffing)
   e. On failure: categorize → retry/skip/fail
3. Close session, collect artifacts
```

## Agent Spawn

```python
Agent(
    subagent_type="general-purpose",
    prompt=build_execution_prompt(diff_data, scope_strategy, coverage_context),
    run_in_background=True,
    name="expect-runner"
)
```

## Agent-Browser Commands

| Command | Use | Example |
|---------|-----|---------|
| `open <url>` | Navigate to page | `open http://localhost:3000/login` |
| `snapshot` | Full ARIA accessibility tree | Capture page structure |
| `snapshot -i` | Interactive elements only | Find clickable/fillable elements |
| `snapshot --delta` (0.38+) | Full tree once, `unchanged` or a compact diff after | Cheap re-check after an in-page action, no navigation |
| `screenshot` | Capture viewport | Auto on failure |
| `screenshot --if-changed` (0.38+) | Skip re-encoding an unchanged capture | Repeated evidence shots of the same route across a run |
| `screenshot --annotate` | Labeled screenshot | Vision fallback for complex UIs |
| `click @ref` | Click by ARIA ref | `click @e15` (from snapshot refs) |
| `fill @ref <text>` | Type into input | `fill @e8 "test@example.com"` |
| `select @ref <option>` | Dropdown selection | `select @e12 "United States"` |
| `eval <js>` | Execute JavaScript | `eval document.title` |

## Auth Profiles

If `.expect/config.yaml` specifies an `auth_profile`:

```python
# Load auth before testing protected pages
Bash(f"agent-browser auth login {auth_profile}")
```

Auth profiles are managed by agent-browser's vault system, credentials are never stored in `.expect/`.

## Session Management

- **One named session per run**: mint one literal session name (`expect-<run-id>`, for example `expect-1759500000-4821`) and write `--session <that name>` on EVERY agent-browser command, `open` and `close` included, so sequential page visits share auth state inside a session only this run owns. Do not export `AGENT_BROWSER_SESSION` once and rely on it: Bash tool calls run in separate shells, the export does not survive, and later commands fall back to `default`, the session every other agent-browser caller on the machine shares.
- **Session timeout**: 5 minutes per page (configurable)
- **Cleanup**: agent-browser auto-closes on agent completion
- **Never pass `--all` to `agent-browser close`**: on a shared machine it kills sessions owned by other agents (#4520). If cleanup is needed, close only this run's session (`agent-browser --session expect-<run-id> close`)

## Failure Decision Tree

```
Step fails
  ├── Is it a retry-able failure? (element-not-found, timeout)
  │   ├── First attempt → wait 2s, retry once
  │   └── Second attempt → categorize and continue
  ├── Is it a page-level failure? (5xx, crash)
  │   └── Skip remaining steps on this page
  ├── Is it auth-related? (401, redirect to login)
  │   └── Skip page, mark as auth-blocked
  └── Is it an app bug? (assertion fails with evidence)
      └── Log as app-bug, screenshot, continue
```

## ARIA Snapshot Diffing Integration

Two different comparisons need two different tools. Within one page visit
(no navigation between the two captures), agent-browser 0.38+ already
computes the structural diff natively, so use it instead of hand-rolling
one:

```python
# Before test steps: establish the delta baseline for this page visit
pre_snapshot = agent_browser("snapshot --delta --full")

# After test steps: same page, no navigation in between
post_snapshot = agent_browser("snapshot --delta")
# post_snapshot is "unchanged", or a compact structural change (added/removed
# nodes, changed names/roles, an exact ref/tree-change splice). Read it
# directly, no local diffing needed.
```

Across separate runs (baseline saved to `.expect/snapshots/{page-slug}.json`
on a prior run, compared against today's fresh capture after a fresh
navigation), `--delta` does not apply, since its baseline resets per session
and per navigation. Use the persisted-file comparison instead (see
`aria-diffing.md`):

```python
# Cross-run comparison: fresh navigation, so take a full snapshot and diff
# it against the saved baseline file, not against --delta's in-session state.
current_snapshot = agent_browser("snapshot")
diff = compute_aria_diff(load_saved_baseline(page), current_snapshot)
if diff.change_score > config.aria_snapshots.diff_threshold:
    report.add_aria_diff(page, diff)
```

## Concurrency Rules

- **Sequential pages**: no parallel browser sessions (see rules/no-parallel-browsers.md)
- **Background agent**: the runner agent runs in background, lead monitors via status protocol
- **Timeout per page**: 5 min default, configurable in config.yaml
- **Total run timeout**: 30 min default


### Fingerprint

# Fingerprint Gating

SHA-256 fingerprint system to skip redundant test runs when files haven't changed.

## How It Works

```
Changed files → SHA-256 each → Compare against .expect/fingerprints.json → Skip or Run
```

## Fingerprint Storage

```json
// .expect/fingerprints.json
{
  "lastRun": "2026-03-26T16:30:00Z",
  "target": "unstaged",
  "hashes": {
    "src/components/Button.tsx": "a1b2c3d4...",
    "src/app/login/page.tsx": "e5f6g7h8..."
  },
  "result": "pass"
}
```

## Computing Fingerprints

```bash
# Hash each changed file
sha256sum $(git diff --name-only) | sort
```

## Decision Logic

```python
def should_run(current_hashes: dict, stored: dict) -> bool:
    if not stored:
        return True  # First run — no fingerprints
    if current_hashes != stored["hashes"]:
        return True  # Files changed since last run
    if stored["result"] == "fail":
        return True  # Last run failed — re-run even if unchanged
    return False     # Same hashes, last run passed — skip
```

## Force Re-Run

Use `--force` flag to bypass fingerprint check:
```bash
/ork:expect --force  # Re-run even if fingerprints match
```

## Implementation Notes

- Hash file **contents**, not metadata (mtime changes shouldn't trigger re-runs)
- Store fingerprints per target (unstaged vs branch vs commit)
- Clear fingerprints on `git checkout` or `git stash` (contents changed)
- `.expect/fingerprints.json` should be gitignored


### Human Review

# Human-in-the-Loop Plan Review (#1179)

Present the generated test plan to the user for review before execution.

## Flow

```
Diff Scan → Plan Generated → [REVIEW GATE] → Execute → Report
                                   ↓
                        AskUserQuestion:
                        "Run this plan?"
                        ├── Run (proceed)
                        ├── Edit (modify)
                        └── Skip (cancel)
```

## Implementation

```python
if not SKIP_REVIEW:  # -y flag bypasses
    AskUserQuestion(questions=[{
        "question": f"Run this test plan? ({step_count} steps across {page_count} pages)",
        "header": "Plan",
        "options": [
            {
                "label": "Run (Recommended)",
                "description": f"{step_count} steps, ~{estimated_time}s"
            },
            {
                "label": "Edit plan",
                "description": "Modify steps before running"
            },
            {
                "label": "Skip",
                "description": "Cancel without running"
            }
        ],
        "multiSelect": False
    }])
```

## Edit Mode

When "Edit plan" is selected:
1. Present the full test plan as editable text
2. User modifies (add/remove/reorder steps)
3. Re-validate step count against scope strategy limits
4. Proceed to execution with modified plan

## Skip Scenarios

The review is automatically skipped when:
- `-y` flag is passed
- Running in CI (`CI=true`)
- Fingerprint matched (no test to run)
- Saved flow replay (`--flow` flag, flow is pre-approved)

## Progressive Feedback

After the user approves, show incremental progress:
```
Executing test plan...
  ✓ /login — 3/3 steps passed (2.1s)
  ◌ /dashboard — running step 2/4...
  ○ /settings — pending
```


### Jev Shadow

# Jev Step Judge for /ork:expect

One Jev judgment per expect step, in one of three modes selected by
`ORK_EXPECT_JEV`: unset or falsey is off (the script exits before python
or the network); `shadow` logs a second opinion beside the agent's pick
and never drives the browser; `1` or `act` lets the Jev pick drive the
step, failing closed to the agent's own pick on any error, timeout, empty
or malformed answer, or a confidence below `act_confidence_floor`. The
legacy `ORK_EXPECT_JEV_SHADOW` flag, truthy with `ORK_EXPECT_JEV` unset,
still selects `shadow`.

## Shape

Same pattern the browser-use/Jev integrations use, rebuilt on our own
stack (agent-browser ARIA refs, no third-party driver):

```
step goal + last verify + interactive elements
        |
        v
ONE Jev request (POST /v1/systemone)
  questions:
    next_action              Choice over the bounded legal-action set
    goal_reached             Noul   (loopback verify)
    page_changed_as_expected Noul   (loopback verify)
    blocked                  Noul   (dialog / login wall / captcha)
        |
        v
JEV_SHADOW|<step-id>|<json>  ->  report.sh  ->  run report (per step + totals)
```

The bounded action set is derived, not freeform: `click`/`fill`/`select`
per interactive element (role-dependent, keyed `verb:@eN` on the same
`@eN` refs agent-browser uses) plus the fixed meta moves `done`, `wait`,
`scroll`, `press_enter`. Jev cannot pick an action the stack cannot take.

## Gate

`ORK_EXPECT_JEV` selects the mode: `shadow` for log-only, `1`/`act` (or
any truthy word) to execute the Jev pick. Any other set value, or unset
with `ORK_EXPECT_JEV_SHADOW` unset or falsey, is off, and off means off:
`scripts/jev-shadow.sh` exits before python or the network are touched.
The API key comes from `ORK_TYPESAFE_API_KEY`; with no key the step logs
`error: no_api_key` and no request is made (in act mode the record falls
back to the agent's pick).

## What leaves while it is on (egress)

Per enabled step, exactly one HTTPS POST to the configured endpoint
carrying `state` = the step goal, the last verify result, and up to
`element_cap` interactive elements as `{ref, role, name}` triples. Only
the fields the four questions need leave; the only ids are the snapshot's
ephemeral `@eN` refs (positional, per-capture, not real identifiers, so
nothing is worth hashing). No page text beyond accessible names, no
cookies, no tokens: every outbound string passes a secret redactor
(`[redacted]`) and a length cap before the request is built. The guard
runs on the decoded form, so a secret smuggled raw (`token=<v>`) or
JSON-escaped (`token\u003d<v>`) is caught the same. Question text and
option descriptions are static strings built locally.

## Tunables

All in `.expect/config.yaml` under `jev_shadow:` (see
references/config-schema.md), falling back to the shipped
`jev-shadow.defaults.yaml`. The script refuses to run with no resolved
config; nothing is a code constant.

| Key | Meaning |
|-----|---------|
| `element_cap` | Max interactive elements per request (default 60) |
| `name_max_chars` | Per-name truncation after redaction |
| `context_max_chars` | Per-field truncation for goal / last verify |
| `latency_budget_ms` | Abort budget for the one request (default 400) |
| `act_confidence_floor` | Act mode only: Choice confidence below this falls back to the agent's pick (default 0.5) |
| `act_list_confidence_floor` | Raised floor for list fields (checkboxes); the two-step list answer and the state answer must agree and clear this floor or the field defers (default 0.7) |
| `model` | Pinned Jev model id |
| `endpoint` | POST target (`ORK_EXPECT_JEV_ENDPOINT` overrides, for mocks) |
| `thresholds.low_confidence` | Choice confidence below this flags the pick |
| `thresholds.{goal_reached,page_changed_as_expected,blocked}` | Noul at or above raises the flag |

## The record

Every active invocation appends its row to `.expect/jev-shadow.jsonl` before
emitting stdout. This journal does not depend on `report.sh --save`.
Use `--log-file` or `ORK_EXPECT_JEV_LOG` to choose another journal. New files
use mode 0600; write failures surface on stderr without changing the fallback.
All rows include `jev_pick`, `jev_confidence`, `incumbent_pick`, `agree`,
`floor`, and `decided_by`, including configuration and transport failures.
Pass the actual choice with `--model-action`. If the incumbent truly did not
run, shadow mode accepts `--incumbent-not-run REASON` instead. Missing evidence
is rejected before writing; act mode always requires an incumbent fallback.
An unrecognized supplied action stays recorded with unknown agreement.

Export the review bucket with
`node scripts/jev-shadow-report.mjs --confident-wrong .expect/jev-shadow.jsonl`.
Its predicate is `agree=false AND jev_confidence >= floor`. The result is a
candidate for adjudication, not an assertion that the incumbent was correct.

```json
{
  "step_id": "login-2",
  "model_action": "click @e15",
  "model_action_key": "click:@e15",
  "jev_action": "click:@e15",
  "jev_action_probability": 0.81,
  "jev_confidence": 0.79,
  "probabilities": {"click:@e15": 0.81, "done": 0.05, "...": 0.0},
  "nouls": {"goal_reached": 0.12, "page_changed_as_expected": 0.9, "blocked": 0.03},
  "flags": {"low_confidence": false, "goal_reached": false,
            "page_changed_as_expected": true, "blocked": false},
  "agree": true,
  "latency_ms": 210,
  "latency_budget_ms": 400,
  "model": "jev-1.13.0",
  "error": null
}
```

`model_action_key` is the agent's pick normalized onto the candidate
keyspace: `click @e15` maps direct, `click "Sign in"` resolves the
accessible name to its ref. Actions outside the keyspace (`press Tab`,
`navigate`, `drag`) log their raw text with `agree: false`; they are
visible in the report but not comparable.

`report.sh` folds each record into the step's `jev_shadow` field, adds a
run-level `jev_shadow: \{recorded, agreed, disagreed, steps, picks_taken,
fallbacks, agree_rate\}` summary to the JSON report, and prints one
`JEV_RUN|steps=N|picks=N|fallbacks=N|agree_rate=R` line at the end of the
run. Failures (`http N`, `unreachable_or_timeout`, `malformed_answer`,
`empty_answer`, `no_api_key`, `no jev_shadow config`) land as error
records with `agree: null`. With valid CLI arguments, the script exits 0: a dead Jev lane
must never fail a run.

In act mode every record additionally carries `mode: "act"`, `path`, and
`executed_action`. `path` is `"jev"` when the Jev pick ran and
`"fallback:<reason>"` when the agent's own pick ran instead
(`fallback:low_confidence` still logs the full Jev answer; the floor
rejected it, not the network). `executed_action` is the canonical
candidate key (`verb:@eN` or a meta verb) the agent translates back to an
agent-browser command; on fallback it is the agent's normalized pick, or
its raw action text when the pick sits outside the keyspace.

Every record also carries the decision provenance fields: `jev_pick`,
`jev_confidence`, `incumbent_pick` (the agent's normalized pick),
`agree`, `floor` (the resolved `act_confidence_floor`), `decided_by`
(`"jev"` or `"incumbent"`), `prompt_id` (config `prompt_id`, override
with `--prompt-id`), and `session_id` (`--session-id`, else
`ORK_EXPECT_SESSION`, else `AB_SESSION`).

## Form path (`--task-doc`)

When `--task-doc` carries the step's task document, the request adds
field questions per form element (up to `field_cap`): a value Choice over
candidates extracted from the document for textbox/searchbox/spinbutton/
combobox/listbox elements, and two questions per checkbox-family element:
`list::<ref>` asks which document list names the item (keep/do/build/add
vs drop/remove/skip/hold/not-do vs unlisted) and `field::<ref>` asks the
final state (check/uncheck/skip). The decision is match-then-map: the
list answer derives the target (do=check, drop=uncheck, none=skip), the
state answer cross-checks it, and disagreement marks the field
`inconsistent` and forces deferral. List fields gate on the raised
`act_list_confidence_floor` (default 0.7); other fields use
`act_confidence_floor`. Per-field answers land in `field_decisions` keyed
by ref, each with `jev_pick`, `jev_confidence` (the weaker of the two
answers for list fields), `list_pick`, `list_conf`, `floor`,
`decided_by`, and the resolved `value` or `target` when Jev decided. In
act mode, when the step's executed action targets a decided field,
`executed_action` carries the decision through: `fill:@eN="value"`,
`check:@eN`, `uncheck:@eN`, or `done` when Jev's target is skip. A field
Jev answers below the floor keeps `decided_by: "incumbent"` and the
agent's own handling stands.

## Invocation (expect-agent, per step)

After picking the action for a step and before executing it:

```bash
bash $CLAUDE_PLUGIN_ROOT/skills/expect/scripts/jev-shadow.sh \
  --step-id login-2 \
  --goal "Submit the login form" \
  --last-verify "Fields filled with test credentials" \
  --model-action "click @e15" \
  --snapshot-file .expect/jev-snap.txt
```

`--snapshot-file` is the `agent-browser snapshot -i` output the agent
already captured (text or `--json` form; only ref/role/name are read).
`--config` or `ORK_EXPECT_CONFIG` points at an alternate config file;
`.expect/config.yaml` is used when present.

Deliberately NOT `snapshot --delta` (0.38+): the bounded legal-action set
needs the full list of interactive elements on every step, and a delta call
answers `unchanged` whenever the page has not moved, which would starve Jev
of candidates on exactly the steps where nothing visibly changed. Use
`snapshot -i` here even though `--delta` is the better choice for the
retry loop in `expect-agent.md`.


### Report

# Report Generator (#1176)

Aggregate execution results into structured reports with CI-compatible exit codes.

## Report Sections

### 1. Summary
```
/ork:expect Report
═══════════════════════════════════════
Target: branch (5 files changed)
Pages tested: 4
Duration: 45s
Result: 13 passed, 2 failed (86.7%)
```

### 2. Step Details
```
/login (Direct — auth form changed)
  ✓ Step 1: Page loads (0.8s)
  ✓ Step 2: Form renders with email + password (0.3s)
  ✗ Step 3: Submit empty form → validation [app-bug]
    Expected: validation errors shown
    Actual: form submitted with no validation
    Screenshot: .expect/screenshots/login-step3.png
  ✓ Step 4: Fill valid credentials → redirect (1.2s)

/dashboard (Routed — renders auth-dependent header)
  ✓ Step 1: Page loads (0.5s)
  ✓ Step 2: User name in header (0.2s)
```

### 3. ARIA Diff (if snapshots exist)
```
ARIA Changes: /login
  + Added: textbox "Confirm Password"
  - Removed: link "Forgot Password?"
  ~ Changed: button "Sign In" → "Log In"
  Change score: 15% (threshold: 10%) — FLAGGED
```

### 4. Artifacts
```
Artifacts:
  .expect/reports/2026-03-26T16-30-00.json
  .expect/screenshots/login-step3.png
```

### 5. Fingerprint
Updated on success, unchanged on failure.

## Output Formats

### Terminal (Default)
Colored output with pass/fail symbols, failure details, and artifact paths.

### CI Mode (GitHub Actions)
When running in CI (`CI=true` or `GITHUB_ACTIONS=true`):

```
::error file=src/components/LoginForm.tsx,line=1::Login form validation missing — expected error messages on empty submit
::warning file=src/app/login/page.tsx::ARIA snapshot changed by 15%% (threshold 10%%)
```

### JSON Report
```json
{
  "version": 1,
  "timestamp": "2026-03-26T16:30:00Z",
  "target": "branch",
  "duration_ms": 45000,
  "files_changed": 5,
  "pages_tested": 4,
  "results": [
    {
      "page": "/login",
      "level": "direct",
      "steps": [
        {"id": "login-1", "title": "Page loads", "status": "passed", "duration_ms": 800},
        {"id": "login-3", "title": "Submit empty form", "status": "failed",
         "category": "app-bug", "error": "No validation errors shown",
         "screenshot": ".expect/screenshots/login-step3.png"}
      ]
    }
  ],
  "aria_diffs": [
    {"page": "/login", "change_score": 0.15, "changes": ["+textbox 'Confirm Password'", "-link 'Forgot Password?'"]}
  ],
  "summary": {
    "total_steps": 15,
    "passed": 13,
    "failed": 2,
    "pass_rate": 0.867
  }
}
```

## Exit Codes

| Code | Meaning | When |
|------|---------|------|
| `0` | All passed | Every step passed, or fingerprint matched (skip) |
| `1` | Tests failed | At least one `app-bug` or `selector-drift` failure |
| `0` + warning | Skipped | `env-issue`, `auth-blocked`, or `missing-test-data` |

## Report Retention

- Keep last N reports (default 10, configurable in config.yaml)
- Auto-delete oldest when limit exceeded
- Reports are gitignored (`.expect/reports/` in `.gitignore`)
- Screenshots are gitignored (`.expect/screenshots/`)

## Post-Report Actions

1. Update fingerprint if all passed (`scripts/fingerprint.sh save`)
2. Persist critical failures to memory graph (if MCP available)
3. Suggest next steps:
   - All passed → "Safe to push."
   - Failed → "Fix \{N\} failures before pushing."
   - Skipped → "Resolve environment issues and re-run."


### Research

# Research Reference (#1181)

Architecture analysis of millionco/expect and related tools.

## millionco/expect

[GitHub: millionco/expect](https://github.com/millionco/expect): AI-powered browser testing tool.

### Key Architecture Decisions

1. **Diff-first**: Uses git diff to determine test scope, doesn't test unchanged code
2. **ARIA over pixels**: Accessibility tree snapshots for semantic UI diffing
3. **Natural language steps**: Test plans written in plain English, executed by AI
4. **Fingerprint gating**: SHA-256 hash of file state, zero-cost skip when unchanged
5. **Failure taxonomy**: 6 categories (app-bug, env-issue, auth-blocked, missing-test-data, selector-drift, agent-misread)

### What We Adopted

| Feature | millionco/expect | /ork:expect |
|---------|-----------------|-------------|
| Diff scanning | 3-level (direct/imported/routed) | Same, plus `changes` target mode |
| Fingerprinting | SHA-256 of HEAD+staged+unstaged | Same |
| Status protocol | STEP_START/STEP_DONE/etc. | Same format |
| Failure categories | 6 types | Same 6 types |
| ARIA snapshots | Line-based diffing | Same |
| Saved flows | YAML format | Markdown+YAML for human readability |
| Config | .expect/config.yaml | Same convention |

### What We Added

| Feature | /ork:expect Only |
|---------|-----------------|
| Scope strategy | Test depth varies by target (commit=narrow, branch=thorough) |
| Coverage context | Cross-ref changed files with existing test files |
| rrweb recording | DOM event replay (not in millionco/expect) |
| Anti-rabbit-hole | Max retry limits, stall detection |
| Agent Teams | Can use mesh orchestration for parallel analysis |
| MCP integration | Memory graph persistence of findings |
| fal.ai integration | Could generate test thumbnails/reports via fal MCP |

## Related Tools

| Tool | Approach | Difference |
|------|----------|------------|
| Playwright | Code-first E2E tests | Manual test authoring, no AI |
| Cypress | Code-first E2E tests | Same as Playwright |
| agent-browser | AI browser automation | Generic, expect adds diff-awareness |
| Meticulous | Visual regression | Pixel-based, not semantic |
| Chromatic | Storybook visual testing | Component-level, not page-level |
| testmon | Python test selection | Unit test scope, not browser |


### Route Map

# Route Map

Map changed files to testable URLs. The route map is the bridge between "what files changed" and "what pages to test."

## Config-Based Route Map

The primary source is `.expect/config.yaml`:

```yaml
base_url: http://localhost:3000
route_map:
  # Component → pages that use it
  "src/components/Header.tsx": ["/", "/about", "/pricing", "/dashboard"]
  "src/components/auth/**": ["/login", "/signup", "/forgot-password"]

  # Page directory → URL pattern
  "src/app/dashboard/**": ["/dashboard"]
  "src/app/settings/**": ["/settings", "/settings/profile", "/settings/billing"]

  # API routes → pages that call them
  "src/app/api/auth/**": ["/login", "/signup"]
```

## Framework Inference (No Config)

When `.expect/config.yaml` doesn't exist, infer from the framework:

### Next.js App Router
```
src/app/page.tsx          → /
src/app/about/page.tsx    → /about
src/app/[slug]/page.tsx   → /{slug} (use a test slug)
src/app/api/auth/route.ts → /login (infer from API name)
```

### Next.js Pages Router
```
pages/index.tsx           → /
pages/about.tsx           → /about
pages/[id].tsx            → /{id}
```

### Generic SPA
```
src/routes/*.tsx           → /{filename}
src/views/*.vue            → /{filename}
```

## Route Resolution Priority

1. `.expect/config.yaml` explicit mapping (highest priority)
2. Framework-specific inference (Next.js, Remix, SvelteKit)
3. Grep for `<Link href=` or `router.push` patterns
4. Fall back to `base_url` root only

## Dynamic Routes

For dynamic routes (`[slug]`, `[id]`), use test values from:
1. `.expect/config.yaml` `test_params` section
2. First entry from a seed/fixture file
3. Default: `test-1`, `1`, `example`


### Rrweb Recording

# rrweb Session Recording (#1178)

Full session replay without video encoding, captures DOM mutations and events as lightweight JSON.

## Why rrweb Over Video

| Approach | Size | Quality | Interaction |
|----------|------|---------|-------------|
| Video (mp4) | ~5MB/min | Lossy | Watch only |
| rrweb JSON | ~100KB/min | Lossless DOM | Replay, inspect, debug |

## Integration Points

### Injection via agent-browser eval
```javascript
// Inject rrweb recorder at test start
eval(`
  const script = document.createElement('script');
  script.src = 'https://cdn.jsdelivr.net/npm/rrweb@2.0.0-alpha.4/dist/rrweb-all.min.js';
  script.onload = () => {
    window.__rrweb_events = [];
    rrweb.record({ emit: (e) => window.__rrweb_events.push(e) });
  };
  document.head.appendChild(script);
`);
```

### Collect events at test end
```javascript
// Extract recorded events
const events = eval("JSON.stringify(window.__rrweb_events)");
```

### Storage
```
.expect/recordings/
├── 2026-03-26T16-30-00-login.json    # rrweb events
└── 2026-03-26T16-30-00-dashboard.json
```

## Replay

rrweb recordings can be replayed in any browser:

```html
<script src="https://cdn.jsdelivr.net/npm/rrweb-player@2.0.0-alpha.4/dist/index.js"></script>
<div id="player"></div>
<script>
  fetch('.expect/recordings/login.json')
    .then(r => r.json())
    .then(events => new rrwebPlayer({ target: document.getElementById('player'), events }));
</script>
```

## Config

```yaml
# .expect/config.yaml
rrweb:
  enabled: false          # Opt-in (adds ~100KB overhead per page)
  storage: .expect/recordings/
  keep_last: 5            # Retain last 5 recordings
  record_on: fail         # always | fail | never
```

## Notes

- rrweb is injected via `eval`, works with any framework, no build step needed
- Recordings are gitignored (ephemeral, large-ish)
- Only record on failure by default to minimize storage
- Future: integrate with report.md to embed replay links in failure details


### Saved Flows

# Saved Test Flows (#1173)

Reusable test sequences stored as Markdown+YAML files in `.expect/flows/`.

## Flow Format

```markdown
---
format_version: 1
title: "Login flow test"
slug: "login-flow-test"
target_scope: "branch"
created: "2026-03-26T12:00:00Z"
last_run: "2026-03-26T14:30:00Z"
last_result: "passed"
steps:
  - instruction: "Navigate to /login"
    expected: "Login form visible with email and password fields"
  - instruction: "Fill email with test@example.com and password with test123"
    expected: "Fields populated"
  - instruction: "Click Login button"
    expected: "Redirect to /dashboard"
  - instruction: "Verify welcome message"
    expected: "Text 'Welcome back' visible on page"
---

# Login Flow Test

Tests the standard login flow with valid credentials.

## Notes
- Requires test user: test@example.com / test123
- Dashboard should show welcome message after redirect
- Auth cookie should be set (verify via eval document.cookie)
```

## Directory Structure

```
.expect/flows/
├── login.md           # Login flow
├── checkout.md        # Checkout flow
└── signup.md          # Signup flow
```

## Running a Flow

```bash
/ork:expect --flow login          # Replay the login flow
/ork:expect --flow checkout -y    # Replay checkout, skip review
```

## Adaptive Replay

When replaying a saved flow, the agent adapts to UI changes:

1. Load flow steps from YAML frontmatter
2. For each step:
   a. Take ARIA snapshot of current page
   b. Match instruction to current UI state
   c. If element exists → execute as-is
   d. If element missing → use ARIA snapshot to find equivalent
   e. If no equivalent found → mark step as `selector-drift` failure
3. After all steps, compare results with `last_result`

## Creating Flows

Flows are created manually by the developer:

```bash
# Create a new flow file
cat > .expect/flows/login.md << 'EOF'
---
format_version: 1
title: "Login flow"
slug: "login"
steps:
  - instruction: "Navigate to /login"
    expected: "Login form visible"
  - instruction: "Fill email and password, click submit"
    expected: "Redirect to /dashboard"
---
# Login Flow
Standard login test with valid credentials.
EOF
```

Future: auto-generate flows from successful test runs by recording the steps the agent executed.

## Flow Metadata

| Field | Required | Description |
|-------|----------|-------------|
| `format_version` | Yes | Always `1` for now |
| `title` | Yes | Human-readable flow name |
| `slug` | Yes | URL-safe identifier, matches filename |
| `target_scope` | No | Recommended target mode (branch, commit, etc.) |
| `created` | No | ISO timestamp of creation |
| `last_run` | No | ISO timestamp of last execution |
| `last_result` | No | `passed` or `failed` |
| `steps` | Yes | Array of instruction+expected pairs |


### Scope Strategy

# Scope-Aware Test Depth Strategy

Adjust test plan depth based on the change target scope.

## Strategy Matrix

| Target | Depth | Flow Count | Strategy | Edge Cases |
|--------|-------|------------|----------|------------|
| `commit` | Narrow | 2-4 | Prove the commit works + 2-3 adjacent flows | Minimal |
| `unstaged` | Exact | 2-3 | Test exact changed flow, watch for partial features | None |
| `changes` | Combined | 3-5 | Treat committed+uncommitted as one body | Light |
| `branch` | Thorough | 5-8 | Full coverage including negative/edge-case flows | Full |

## Strategy Definitions

### commit: Narrow Focus
```
Test depth: NARROW
Focus: Prove this specific commit works correctly.
Flow count: 2-4 flows max.
Strategy: Test the primary flow the commit modifies, then 2-3 adjacent
flows that could be affected. Don't test unrelated pages.
Edge cases: Only test edge cases if the commit explicitly handles them.
Style: Quick validation — this is a single logical change.
```

### unstaged: Exact Match
```
Test depth: EXACT
Focus: Test exactly what's been modified in the working tree.
Flow count: 2-3 flows max.
Strategy: The developer is mid-work. Test the exact flow being changed.
Watch for partial implementations (half-finished features).
Edge cases: Skip — the code may be incomplete.
Style: Development feedback loop — fast, targeted, forgiving of WIP.
```

### changes: Combined (Default)
```
Test depth: COMBINED
Focus: Treat committed branch changes + uncommitted edits as one body.
Flow count: 3-5 flows.
Strategy: Test the overall feature being developed. Include the primary
flow and its dependencies. Check that committed work still integrates
with uncommitted changes.
Edge cases: Light — test obvious boundary conditions.
Style: Pre-push validation — comprehensive but not exhaustive.
```

### branch: Thorough Coverage
```
Test depth: THOROUGH
Focus: Full coverage of all changes on this branch vs main.
Flow count: 5-8 flows.
Strategy: This is the final check before merge. Test all affected pages
thoroughly. Include negative flows (invalid input, error states).
Cover accessibility on key pages. Verify no regressions.
Edge cases: Full — test boundary conditions, empty states, error handling.
Style: PR readiness — the branch should be merge-ready after this passes.
```

## Integration with Test Plan

The scope strategy is injected into the AI test plan generation prompt:

```python
def get_scope_strategy(target: str) -> str:
    strategies = {
        "commit": COMMIT_STRATEGY,
        "unstaged": UNSTAGED_STRATEGY,
        "changes": CHANGES_STRATEGY,
        "branch": BRANCH_STRATEGY,
    }
    return strategies.get(target, CHANGES_STRATEGY)

# In test-plan generation:
prompt = f"""
{scope_strategy}

Based on the above testing strategy, generate a test plan for:
{diff_summary}
"""
```

## Flow Count Enforcement

The test plan generator should respect the flow count range:
- If the plan exceeds the max, trim to highest-magnitude pages
- If the plan is under the min, expand to include imported (Level 2) pages
- Log which flows were trimmed/added and why


### Test Plan

# AI Test Plan Generation: buildExecutionPrompt (#1169)

Core prompt template that generates test plans from diff context using AI agents.

## Prompt Template (8 Sections)

```python
def build_execution_prompt(
    diff_data: dict,
    scope_strategy: str,
    coverage_context: str,
    saved_flow: str | None = None,
    instruction: str | None = None,
) -> str:
    return f"""
You are a QA engineer executing browser tests via agent-browser.

═══════════════════════════════════════════════════════════════
SECTION 1: DIFF CONTEXT
═══════════════════════════════════════════════════════════════

Changed files ({diff_data['summary']['total']} total):
{format_files(diff_data['files'])}

File stats (by magnitude):
{format_stats(diff_data['stats'])}

Diff preview:
{diff_data['preview']}

Recent commits:
{format_context(diff_data['context'])}

═══════════════════════════════════════════════════════════════
SECTION 2: SCOPE STRATEGY
═══════════════════════════════════════════════════════════════

{scope_strategy}

═══════════════════════════════════════════════════════════════
SECTION 3: COVERAGE CONTEXT
═══════════════════════════════════════════════════════════════

{coverage_context}

Files WITH existing tests are lower priority — focus on files WITHOUT test coverage.

═══════════════════════════════════════════════════════════════
SECTION 4: AGENT-BROWSER TOOL DOCS
═══════════════════════════════════════════════════════════════

Available commands (use via agent-browser skill):
- open <url>: Go to URL
- snapshot: Capture current page accessibility tree
- click <selector>: Click an element
- fill <selector> <value>: Type into an input
- select <selector> <option>: Select dropdown option
- screenshot [filename]: Take screenshot (auto on failure)
- eval <js>: Run JavaScript in page context; assert page text or URL by returning a check from here
- wait <selector>: Wait for an element to appear (prefer this over a fixed delay)
- wait --url <pattern>: Wait for the URL to match
- wait --load networkidle: Wait for the page to settle

═══════════════════════════════════════════════════════════════
SECTION 5: INTERACTION PATTERN
═══════════════════════════════════════════════════════════════

Follow this pattern for every page:

1. Navigate to URL
2. Take ARIA snapshot (accessibility tree)
3. Use ARIA roles/names as selectors — NOT CSS selectors
   Prefer: click "Submit" (by accessible name)
   Avoid: click "#btn-submit-form-1" (brittle CSS)
4. Batch related assertions together
5. Screenshot only on failure (not every step)

When interacting with forms:
- Fill all fields before submitting
- Check validation messages after submit
- Verify redirect/state change after success

═══════════════════════════════════════════════════════════════
SECTION 6: STATUS PROTOCOL
═══════════════════════════════════════════════════════════════

Report every step using this exact format:

  STEP_START|<step-id>|<step-title>
  STEP_DONE|<step-id>|<short-summary>

On failure:
  ASSERTION_FAILED|<step-id>|<why-it-failed>

At the end:
  RUN_COMPLETED|passed|<summary>
  RUN_COMPLETED|failed|<summary>

Example:
  STEP_START|login-1|Navigate to /login
  STEP_DONE|login-1|Page loaded, form visible
  STEP_START|login-2|Fill email and password
  STEP_DONE|login-2|Fields filled
  STEP_START|login-3|Submit form
  ASSERTION_FAILED|login-3|Expected redirect to /dashboard, got /login with error "Invalid credentials"
  RUN_COMPLETED|failed|2 passed, 1 failed — login form validation error

═══════════════════════════════════════════════════════════════
SECTION 7: ANTI-RABBIT-HOLE HEURISTICS
═══════════════════════════════════════════════════════════════

CRITICAL — follow these rules to avoid wasting time:

1. Do NOT repeat the same failing action more than ONCE without new evidence.
   If click "Submit" fails, do not try clicking it again. Investigate why.

2. If 4 consecutive actions fail, STOP and report.
   Output: RUN_COMPLETED|failed|Stopped after 4 consecutive failures

3. Categorize every failure into one of these types:
   - app-bug: The application has a real bug (test found something!)
   - env-issue: Server not running, wrong URL, network error
   - auth-blocked: Need login but no credentials available
   - missing-test-data: Form requires data that doesn't exist
   - selector-drift: UI changed, saved selectors don't match
   - agent-misread: AI misinterpreted the page structure

4. If you detect env-issue or auth-blocked, skip remaining steps
   on that page and move to the next page.

5. Total time limit: 5 minutes per page. If a page takes longer, skip.

═══════════════════════════════════════════════════════════════
SECTION 8: USER INSTRUCTION / SAVED FLOW
═══════════════════════════════════════════════════════════════

{format_instruction_or_flow(instruction, saved_flow)}
"""
```

## Helper Functions

```python
def format_files(files: list) -> str:
    return "\n".join(
        f"  [{f['status'].upper()[0]}] {f['path']} ({f['type']})"
        for f in files
    )

def format_stats(stats: list) -> str:
    sorted_stats = sorted(stats, key=lambda s: s['magnitude'], reverse=True)
    return "\n".join(
        f"  +{s['added']} -{s['removed']} ({s['magnitude']} lines) {s['path']}"
        for s in sorted_stats[:12]
    )

def format_context(commits: list) -> str:
    return "\n".join(f"  {c}" for c in commits)

def format_instruction_or_flow(instruction, saved_flow):
    if saved_flow:
        return f"""REPLAYING SAVED FLOW:
{saved_flow}

Adapt if UI has changed since the flow was saved. If a step no longer
matches the page structure, use the ARIA snapshot to find the equivalent
element and continue."""

    if instruction:
        return f"""USER INSTRUCTION:
{instruction}

Generate a test plan that addresses this instruction, scoped to the
changed files from Section 1."""

    return """No specific instruction. Generate a test plan that verifies
the changed code works correctly and doesn't break existing functionality.
Focus on the most impactful changes (highest magnitude from Section 1)."""
```

## Coverage Context Generation

Cross-reference changed files with existing test files:

```python
def generate_coverage_context(changed_files: list, project_dir: str) -> str:
    covered = []
    uncovered = []

    for f in changed_files:
        # Check for co-located test
        test_patterns = [
            f.replace('.tsx', '.test.tsx'),
            f.replace('.ts', '.test.ts'),
            f.replace('.ts', '.spec.ts'),
            f.replace('src/', 'src/__tests__/'),
        ]
        has_test = any(os.path.exists(os.path.join(project_dir, t)) for t in test_patterns)

        if has_test:
            covered.append(f)
        else:
            uncovered.append(f)

    lines = []
    if uncovered:
        lines.append(f"Files WITHOUT test coverage ({len(uncovered)}) — HIGH PRIORITY:")
        lines.extend(f"  ⚠ {f}" for f in uncovered)
    if covered:
        lines.append(f"\nFiles WITH existing tests ({len(covered)}) — lower priority:")
        lines.extend(f"  ✓ {f}" for f in covered)

    return "\n".join(lines)
```

## Status Protocol Parsing

Parse agent output to extract structured results:

```python
import re

def parse_status_lines(output: str) -> dict:
    steps = []
    final_status = None

    for line in output.split('\n'):
        line = line.strip()

        if line.startswith('STEP_START|'):
            parts = line.split('|', 2)
            steps.append({"id": parts[1], "title": parts[2], "status": "running"})

        elif line.startswith('STEP_DONE|'):
            parts = line.split('|', 2)
            step = next((s for s in steps if s['id'] == parts[1]), None)
            if step:
                step['status'] = 'passed'
                step['summary'] = parts[2]

        elif line.startswith('ASSERTION_FAILED|'):
            parts = line.split('|', 2)
            step = next((s for s in steps if s['id'] == parts[1]), None)
            if step:
                step['status'] = 'failed'
                step['error'] = parts[2]

        elif line.startswith('RUN_COMPLETED|'):
            parts = line.split('|', 2)
            final_status = {"result": parts[1], "summary": parts[2]}

    passed = sum(1 for s in steps if s['status'] == 'passed')
    failed = sum(1 for s in steps if s['status'] == 'failed')

    return {
        "steps": steps,
        "passed": passed,
        "failed": failed,
        "final": final_status,
    }
```