> ## Documentation Index > Fetch the complete documentation index at: https://docs.openbrowser.me/llms.txt > Use this file to discover all available pages before exploring further. # CLI Benchmark: 4-Way Comparison > Benchmark comparison of openbrowser-ai vs browser-use vs playwright-cli vs agent-browser CLIs # CLI Benchmark: 4-Way Comparison Benchmark date: 2026-03-15 | Claude Sonnet 4.6 on AWS Bedrock | N=3 runs | 6 tasks | Single Bash tool ## Overview Four CLI browser automation tools compared head-to-head. Each tool gets a single generic Bash tool (identical overhead) with an optimized system prompt. The LLM drives each tool autonomously to complete 6 real-world browser tasks. | | openbrowser-ai | browser-use | playwright-cli | agent-browser | | ------------------------ | --------------------------------- | ------------------------ | ------------------------- | ------------------------- | | **Maintainer** | OpenBrowser | browser-use | Playwright | agent-browser | | **Engine** | Raw CDP (direct) | Playwright (CDP) | Playwright (CDP) | Playwright (CDP) | | **Interface** | `openbrowser-ai -c 'code'` | `uvx browser-use ` | `playwright-cli ` | `agent-browser ` | | **Code batching** | Python (multi-operation per call) | No (individual commands) | JS via `run-code` | No (`&&` chaining only) | | **Page state format** | DOM with `[i_N]` indices | DOM with `[N]` indices | A11y tree in `.yml` files | A11y tree with `@eN` refs | | **Page state size** | \~450 chars | \~880 chars | \~1,420 chars | \~590 chars (with `-i`) | | **Variable persistence** | Yes (daemon) | Yes (daemon) | Yes (background process) | Yes (background process) | ## Methodology * **Model**: Claude Sonnet 4.6 on AWS Bedrock (Converse API), us-west-1 * **Tool**: Single generic Bash tool for all 4 approaches (identical tool-definition overhead) * **System prompts**: Per-approach optimized prompts with tool-specific commands and optimization tips * **Fairness**: Both approach order AND task order randomized per run (eliminates OS/DNS caching bias) * **Browser**: Persistent daemon per approach across all 6 tasks, headless mode, browser cleanup between approaches * **Statistics**: N=3 runs, 10,000-sample bootstrap for 95% confidence intervals * **Tasks**: Same 6 tasks as the [MCP benchmark](/comparison) (fact\_lookup, form\_fill, multi\_page\_extract, search\_navigate, deep\_navigation, content\_analysis) against live websites * **Benchmark script**: `benchmarks/e2e_4way_cli_benchmark.py` * **Results data**: `benchmarks/e2e_4way_cli_results.json` ### Tasks | # | Task | Description | Target Site | | :-: | ------------------------ | ----------------------------------------------------------------------------- | ---------------------- | | 1 | **fact\_lookup** | Navigate to a Wikipedia article and extract specific facts (creator and year) | en.wikipedia.org | | 2 | **form\_fill** | Fill out a multi-field form (text input, radio button, checkbox) and submit | httpbin.org/forms/post | | 3 | **multi\_page\_extract** | Extract the titles of the top 5 stories from a dynamic page | news.ycombinator.com | | 4 | **search\_navigate** | Search Wikipedia, click a result, and extract specific information | en.wikipedia.org | | 5 | **deep\_navigation** | Navigate to a GitHub repo and find the latest release version number | github.com | | 6 | **content\_analysis** | Analyze page structure: count headings, links, and paragraphs | example.com | ### Fairness Design Unlike MCP benchmarks where each server defines its own tools (different counts, different schemas, different token overhead), this CLI benchmark uses a **single Bash tool** for all 4 approaches. This eliminates the tool-definition advantage -- the only difference is the system prompt telling the LLM how to use each tool. Additional fairness measures: * **Randomized approach order**: Each run shuffles which CLI goes first, preventing later approaches from benefiting from OS/DNS caching * **Randomized task order**: Each approach sees tasks in a different order per run * **Persistent daemon**: All 4 tools keep a browser session alive across 6 tasks (no cold-start advantage) * **Browser cleanup**: Stale browser processes killed between approaches * **Headless mode**: Eliminates rendering overhead differences ## Results: Overall All 4 tools achieve **100% accuracy** (18/18 task executions across 3 runs). CLI Benchmark: Token Usage vs Duration | Metric | openbrowser-ai | browser-use | playwright-cli | agent-browser | | ------------------------------------- | -------------------: | ----------------: | ----------------: | ----------------: | | **Duration (mean +/- std)** | **84.8 +/- 10.9s** | 106.0 +/- 9.5s | 118.3 +/- 21.4s | 99.0 +/- 6.8s | | **Tool Calls (mean +/- std)** | **15.3 +/- 2.3** | 20.7 +/- 6.4 | 25.7 +/- 8.1 | 25.0 +/- 4.0 | | **Bedrock API Tokens (mean +/- std)** | **36,010 +/- 6,063** | 77,123 +/- 33,354 | 94,130 +/- 35,982 | 90,107 +/- 3,698 | | **Response Chars (mean +/- std)** | **9,452 +/- 472** | 36,241 +/- 12,940 | 84,065 +/- 49,713 | 56,009 +/- 39,733 | | **Token ratio vs openbrowser-ai** | 1x (baseline) | 2.1x more | 2.6x more | 2.5x more | CLI Benchmark: 4-Way Comparison Overview ## Results: Per-Task Token Usage | Task | openbrowser-ai | browser-use | playwright-cli | agent-browser | | -------------------- | -------------: | ----------: | -------------: | ------------: | | fact\_lookup | **2,504** | 4,710 | 16,857 | 9,676 | | form\_fill | **7,887** | 15,811 | 31,757 | 19,226 | | multi\_page\_extract | **2,354** | 2,405 | 8,886 | 8,117 | | search\_navigate | **16,539** | 47,936 | 27,779 | 44,367 | | deep\_navigation | **2,178** | 3,747 | 4,705 | 5,534 | | content\_analysis | 4,548 | **2,515** | 4,147 | 3,189 | openbrowser-ai wins 5 of 6 tasks on tokens. The advantage is largest on complex pages (search\_navigate: 2.9x fewer tokens than browser-use) where code batching avoids repeated page state dumps. browser-use edges ahead on content\_analysis -- a trivial task where all approaches use minimal tokens. CLI Benchmark: Per-Task Token Usage ## Results: Cost Per Benchmark Run (6 Tasks) Based on Bedrock API token usage (input + output tokens at respective rates). | Model | openbrowser-ai | browser-use | playwright-cli | agent-browser | | -------------------------------- | -------------: | ----------: | -------------: | ------------: | | Claude Sonnet 4.6 ($3/$15 per M) | **\$0.12** | \$0.24 | \$0.29 | \$0.27 | | Claude Opus 4.6 ($5/$25 per M) | **\$0.24** | \$0.45 | \$0.56 | \$0.51 | ## Why openbrowser-ai Wins ### 1. Python Code Batching Multiple browser operations in a single `openbrowser-ai -c '...'` call: ```bash theme={null} openbrowser-ai -c ' await navigate("https://en.wikipedia.org/wiki/Python_(programming_language)") info = await evaluate("document.querySelector(\".infobox\")?.innerText") print(info) ' ``` One tool invocation does what competitors need 3-5 tool calls for. Each tool call incurs LLM inference overhead (reading full conversation history), so fewer calls = fewer tokens. ### 2. Compact DOM Representation Page state uses DOM with `[i_N]` indices at \~450 chars: ``` [i_1] [i_2] [i_3] [i_4] [i_5] ``` vs \~880 chars (browser-use DOM), \~590 chars (agent-browser a11y tree with `-i`), or \~1,420 chars (playwright-cli a11y tree in `.yml` file). ### 3. Server-Side Processing The LLM writes Python code that processes data server-side and returns only extracted results via `print()`. Competitors return full page state that the LLM must parse in its context window. ### 4. Variable Persistence The daemon maintains a Python namespace across `-c` calls. Intermediate results (selectors, extracted data, computed values) persist without re-extracting: ```bash theme={null} openbrowser-ai -c 'await navigate("https://example.com"); title = await evaluate("document.title")' openbrowser-ai -c 'print(f"Title was: {title}")' # title still available ``` ## Variance Analysis | CLI Tool | Token std / mean | Duration std / mean | | -------------- | :--------------: | :-----------------: | | openbrowser-ai | 17% | 13% | | browser-use | 43% | 9% | | playwright-cli | 38% | 18% | | agent-browser | 4% | 7% | * **openbrowser-ai**: Moderate variance -- consistent enough for reliable cost estimation * **browser-use**: High token variance (43%) driven by search\_navigate task where the LLM sometimes takes extra exploration turns * **playwright-cli**: High token variance (38%) driven by form\_fill where accessibility tree snapshots vary in size * **agent-browser**: Lowest token variance (4%) but at 2.5x the absolute token cost ## How Each CLI Works ### openbrowser-ai ```bash theme={null} # Python code batching -- multiple operations per call openbrowser-ai -c 'await navigate("url"); data = await evaluate("js"); print(data)' ``` * Persistent daemon over Unix socket * `-c` flag executes async Python in a persistent namespace * All browser functions available: `navigate()`, `click()`, `input_text()`, `evaluate()`, `scroll()`, etc. * Variables persist across calls ### browser-use ```bash theme={null} # Individual CLI commands uvx --from "browser-use[cli]" browser-use open https://example.com uvx --from "browser-use[cli]" browser-use state uvx --from "browser-use[cli]" browser-use input 5 "text" ``` * Individual commands per operation * `input "text"` combines click + type (optimization) * DOM with `[N]` indices * `uvx` isolation due to dependency conflicts ### playwright-cli ```bash theme={null} # JS batching via run-code playwright-cli run-code "async page => { await page.goto('url'); return await page.title(); }" # Snapshots save to .yml files playwright-cli snapshot && cat .playwright-cli/page-*.yml ``` * `run-code` enables JS batching (similar to openbrowser-ai's Python batching) * Snapshots save to `.yml` files, requiring extra `cat` to read * Accessibility tree format (\~1,420 chars per page) ### agent-browser ```bash theme={null} # Individual commands with && chaining agent-browser open https://example.com && agent-browser snapshot -i agent-browser click @e5 agent-browser eval "document.title" ``` * Individual commands, chainable with `&&` * `snapshot -i` flag for compact output (85-95% smaller than full snapshot) * Accessibility tree with `@eN` refs * `eval` for JavaScript execution