​

CLI Benchmark: 4-Way Comparison

​

Overview

​

Methodology

• Model: Claude Sonnet 4.6 on AWS Bedrock (Converse API), us-west-1
• Tool: Single generic Bash tool for all 4 approaches (identical tool-definition overhead)
• System prompts: Per-approach optimized prompts with tool-specific commands and optimization tips
• Fairness: Both approach order AND task order randomized per run (eliminates OS/DNS caching bias)
• Browser: Persistent daemon per approach across all 6 tasks, headless mode, browser cleanup between approaches
• Statistics: N=3 runs, 10,000-sample bootstrap for 95% confidence intervals
• Tasks: Same 6 tasks as the MCP benchmark (fact_lookup, form_fill, multi_page_extract, search_navigate, deep_navigation, content_analysis) against live websites
• Benchmark script: benchmarks/e2e_4way_cli_benchmark.py
• Results data: benchmarks/e2e_4way_cli_results.json

​

Tasks

​

Fairness Design

• Randomized approach order: Each run shuffles which CLI goes first, preventing later approaches from benefiting from OS/DNS caching
• Randomized task order: Each approach sees tasks in a different order per run
• Persistent daemon: All 4 tools keep a browser session alive across 6 tasks (no cold-start advantage)
• Browser cleanup: Stale browser processes killed between approaches
• Headless mode: Eliminates rendering overhead differences

​

Results: Overall

​

Results: Per-Task Token Usage

​

Results: Cost Per Benchmark Run (6 Tasks)

​

Why openbrowser-ai Wins

​

1. Python Code Batching

openbrowser-ai -c '
await navigate("https://en.wikipedia.org/wiki/Python_(programming_language)")
info = await evaluate("document.querySelector(\".infobox\")?.innerText")
print(info)
'

​

2. Compact DOM Representation

[i_1]<input name="custname"/>  [i_2]<input name="tel"/>
[i_3]<radio name="size" value="medium"/>  [i_4]<checkbox name="topping" value="mushroom"/>
[i_5]<button>Submit order</button>

​

3. Server-Side Processing

​

4. Variable Persistence

openbrowser-ai -c 'await navigate("https://example.com"); title = await evaluate("document.title")'
openbrowser-ai -c 'print(f"Title was: {title}")'  # title still available

​

Variance Analysis

• openbrowser-ai: Moderate variance — consistent enough for reliable cost estimation
• browser-use: High token variance (43%) driven by search_navigate task where the LLM sometimes takes extra exploration turns
• playwright-cli: High token variance (38%) driven by form_fill where accessibility tree snapshots vary in size
• agent-browser: Lowest token variance (4%) but at 2.5x the absolute token cost

​

How Each CLI Works

​

openbrowser-ai

# Python code batching -- multiple operations per call
openbrowser-ai -c 'await navigate("url"); data = await evaluate("js"); print(data)'

• Persistent daemon over Unix socket
• -c flag executes async Python in a persistent namespace
• All browser functions available: navigate(), click(), input_text(), evaluate(), scroll(), etc.
• Variables persist across calls

​

browser-use

# Individual CLI commands
uvx --from "browser-use[cli]" browser-use open https://example.com
uvx --from "browser-use[cli]" browser-use state
uvx --from "browser-use[cli]" browser-use input 5 "text"

• Individual commands per operation
• input <index> "text" combines click + type (optimization)
• DOM with [N] indices
• uvx isolation due to dependency conflicts

​

playwright-cli

# JS batching via run-code
playwright-cli run-code "async page => { await page.goto('url'); return await page.title(); }"
# Snapshots save to .yml files
playwright-cli snapshot && cat .playwright-cli/page-*.yml

• run-code enables JS batching (similar to openbrowser-ai’s Python batching)
• Snapshots save to .yml files, requiring extra cat to read
• Accessibility tree format (~1,420 chars per page)

​

agent-browser

# Individual commands with && chaining
agent-browser open https://example.com && agent-browser snapshot -i
agent-browser click @e5
agent-browser eval "document.title"

• Individual commands, chainable with &&
• snapshot -i flag for compact output (85-95% smaller than full snapshot)
• Accessibility tree with @eN refs
• eval for JavaScript execution