Skip to main content

Browser Automation Comparison

Two benchmark suites measuring OpenBrowser against competitors:
  1. CLI Benchmark (2026-03-15) — 4-way comparison of CLI tools driven by an LLM via a single Bash tool
  2. MCP Benchmark (below) — 3-way comparison of MCP servers as tool providers
Historical benchmark, not a current-version ranking. Both runs on this page were recorded on the date shown. The 6-task LLM run installed Playwright MCP and Chrome DevTools MCP from @latest at run time, but their resolved versions were not written to the retained artifacts. The 5-step run’s install method was not retained. The 5-step token benchmark’s raw per-response files are also not retained in this repository. Treat these numbers as historical observations for the versions installed on the run date, not as a comparison against today’s releases. No universal winner follows from them. See Provenance and limits.

MCP Server Comparison

Benchmark date: 2026-02-21 | OpenBrowser MCP v0.1.26 (CodeAgent, 1 tool) | Playwright MCP (resolved version not recorded) | Chrome DevTools MCP (resolved version not recorded)

Overview

Three approaches to browser automation via MCP, measured on identical tasks.

Token Usage Benchmark

Methodology

All three MCP servers were started as subprocesses and tested via JSON-RPC stdio transport. Same 5-step workflow, same pages, same measurement method. All numbers are real measurements, not estimates. Workflow: Navigate to Wikipedia Python page -> get page state -> click link -> go back -> get state again.

Results: 5-Step Workflow on Wikipedia (Complex Page)

Results: Small Page (httpbin.org/forms/post)

Per-Operation Token Breakdown (Wikipedia)

Why the Difference

Playwright MCP: Every navigation returns the full page accessibility snapshot (~124K tokens for Wikipedia). Consistent, but forces the LLM to process the entire page with every action. Chrome DevTools MCP: Returns minimal confirmations for navigation. The agent must explicitly call take_snapshot to see the page (~135K tokens for Wikipedia). One snapshot is comparable in size to Playwright’s, but actions don’t auto-return snapshots. OpenBrowser MCP: Actions return minimal confirmations. The agent explicitly requests the level of detail it needs — from 105 tokens (compact state) to 3,981 tokens (targeted search) to 25,164 tokens (full page text). More tool calls, dramatically fewer tokens.

What Each Server Returns (Verbatim)

All responses below are real output captured from each MCP server on the httpbin.org/forms/post page. Playwright MCP — returns full a11y snapshot with every navigation (~2,150 chars on httpbin, ~496K chars on Wikipedia):
Chrome DevTools MCP — returns URL confirmation only (~136 chars):
OpenBrowser MCP — returns URL confirmation only (~105 chars):

Get Page State / Snapshot

Playwright MCP browser_snapshot — full a11y tree again (~1,896 chars on httpbin, ~495K chars on Wikipedia):
Chrome DevTools MCP take_snapshot — full a11y tree (~1,214 chars on httpbin, ~538K chars on Wikipedia):
OpenBrowser MCP — the LLM writes Python code to query exactly what it needs:
The agent requests only the data it needs via Python code — page title, specific element attributes, or targeted JS evaluation. No full-page dumps.

Click / Type / Go Back

All three servers return short confirmations for actions:

Targeted Extraction (OpenBrowser only)

No equivalent in Playwright or Chrome DevTools MCP — both require dumping the full snapshot or running JavaScript from the client side. Element search — the LLM writes Python to find specific elements:
Data extraction — extract specific data via JS evaluation, not full-page dumps:
Comparison for finding “Guido van Rossum” on Wikipedia:

Cost Comparison

MCP tool response costs per 5-step workflow on a complex page (Wikipedia). These tokens are added to the LLM’s context window, charged at input token rates. All numbers based on real measurements. Per 1,000 workflows:

E2E LLM Benchmark

Methodology

Six real-world browser tasks run through Claude Sonnet 4.6 on AWS Bedrock (Converse API). Each task uses a single MCP server as tool provider. The LLM decides which tools to call and when the task is complete. All tasks run against live websites. Tasks: Wikipedia fact lookup, httpbin form fill, Hacker News data extraction, Wikipedia search + navigation, GitHub release lookup, example.com content analysis.

Results: Task Success

All three servers pass all 6 tasks.

Results: Tool Calls and Duration

Playwright completes most tasks in 1 tool call because every navigation returns the full accessibility snapshot — the LLM sees the entire page immediately and can answer without follow-up queries. OpenBrowser takes more turns because the LLM writes code to navigate, extract, and verify data step by step. MCP Server Benchmark: Duration vs Token Usage

Per-Task Duration

Per-Task Duration by MCP Server

Per-Task Tool Calls

Tool Calls Per Task by MCP Server

Results: Token Efficiency

Bedrock API token usage measured from the Converse API usage field (mean across 5 runs, 10,000-sample bootstrap CIs). Total Bedrock API Token Usage (Input vs Output)

Per-Task MCP Response Size

The pattern: any task involving a complex page (Wikipedia, GitHub releases) produces massive response payloads for Playwright and Chrome DevTools because they dump the full accessibility snapshot. OpenBrowser returns only the data the code explicitly extracts. MCP Response Size: Full Page Dumps vs Server-Side Processing

Per-Task Input Token Usage

Per-Task Input Token Usage by MCP Server

Results: Cost Per Benchmark Run (6 Tasks)

Cost per benchmark run based on Bedrock API token usage (input + output tokens at respective rates). Cost Per Benchmark Run by Model

Why OpenBrowser Uses More Tool Calls but Fewer Tokens

Playwright sends the full page with every response, so the LLM gets the answer immediately but pays for ~120K tokens per Wikipedia page load. OpenBrowser returns compact results (~30-800 chars per call), so the LLM needs more round-trips to navigate and extract but pays far fewer tokens overall. For a single simple task, Playwright’s approach is fast. In the 2026-02-21 six-task run, the ratio of mean total MCP response characters per six-task run, averaged over five runs per server, was 144x. Complex pages drove most of the difference. That ratio is a historical observation, not a current-version guarantee.

Tool Surface Comparison

Playwright MCP (22 core tools, 34 total)

Navigation, interaction, form filling, file upload, drag and drop, hover, key press, select option, screenshots, snapshots, console messages, dialog handling, network requests, tab management, code execution, PDF export, wait conditions, resize, vision-mode coordinate tools, test assertions.

Chrome DevTools MCP (26 tools)

Input automation (click, drag, fill, fill_form, hover, press_key, handle_dialog, upload_file), navigation (navigate_page, new_page, close_page, list_pages, select_page, wait_for), emulation (emulate, resize_page), performance tracing (start/stop/analyze), network debugging (list/get requests), JS execution, console messages, screenshots, DOM snapshots.

OpenBrowser MCP (CodeAgent, execute_code in v0.1.26)

The benchmark run used v0.1.26 with execute_code. get_status and reset_browser were added after v0.1.26 and were not part of the benchmarked server.

Unique to OpenBrowser

This section describes the benchmarked OpenBrowser v0.1.26 server. The benchmark itself used execute_code only. The get_status and reset_browser lifecycle tools were added after v0.1.26 and were not part of the comparison. The feature bullets below describe the v0.1.26 server. The run measured response size and tool counts; it did not benchmark each feature:
  • CodeAgent architecture — the core execute_code tool runs Python in a persistent namespace. The LLM writes code to navigate, extract, and process data rather than calling individual tools. Variables and state persist between calls.
  • 144x smaller mean total MCP response characters per six-task run in 2026-02-21 — returns only the data the code explicitly extracts, not full page dumps
  • JS evaluation with Python processing — await evaluate("JS expression") returns Python objects directly (dicts, lists, strings), enabling pandas/regex/json processing in the same code block
  • Built-in libraries — json, pandas, numpy, matplotlib, csv, re, datetime, requests, BeautifulSoup available in the execution namespace
  • File system access — file_system object for reading/writing local files from browser automation code
  • Dropdown support — select_dropdown() and dropdown_options() for native <select> elements
  • Task completion signal — done(text, success) to explicitly mark task completion with a result

Gaps vs Competitors

Feature lists below are from the versions tested on 2026-02-21. Competitor resolved versions were not recorded. Check current documentation before relying on a gap.

Provenance and limits

  • The 5-step token benchmark was recorded on 2026-02-21 with OpenBrowser MCP v0.1.26. The Playwright MCP and Chrome DevTools MCP resolved versions were not recorded.
  • The 6-task LLM benchmark was recorded on the same date with Claude Sonnet 4.6 on AWS Bedrock, 5 runs per server, and a 10,000-sample bootstrap. The retained aggregate is benchmarks/e2e_llm_stats_results.json. It records the model, run count, and server aggregates, but not the competitor tool versions or a timestamp. The page date is a page record, not an artifact field.
  • The per-response files cited by earlier revisions of this page are not present in the repository. Their numbers cannot be recomputed from the retained files. The tables on this page remain a historical summary, not a reproducible artifact.
  • Token volume is not task success, latency, or cost for every workload. The six tasks are a fixed local sample against public sites. Treat differences in tool count and response shape as design tradeoffs, not a universal winner.
  • Do not compare these numbers with a later release without a new pinned run. The native navigation harness is a separate lane and is not a source for this page.

Benchmark Methodology Notes

Token Usage Benchmark (5-step workflow)

  • All three servers benchmarked via JSON-RPC stdio subprocess — no estimates
  • Response sizes measured as total JSON-RPC response character count
  • Estimated tokens = characters / 4 (standard approximation for mixed English/JSON)
  • 5-step workflow uses matched operations: navigate, get state/snapshot, click, go back, get state/snapshot
  • Raw per-response data for the 5-step run: not retained in this repository. The numbers above are the retained summary.

E2E LLM Benchmark (6 real-world tasks, N=5 runs)

  • Model: Claude Sonnet 4.6 on AWS Bedrock (Converse API)
  • Each task uses a single MCP server as tool provider via JSON-RPC stdio
  • The LLM autonomously decides which tools to call and when the task is complete
  • 5 runs per server, 10,000-sample bootstrap for 95% confidence intervals
  • 6 tasks: fact_lookup, form_fill, multi_page_extract, search_navigate, deep_navigation, content_analysis
  • All tasks run against live websites (Wikipedia, httpbin.org, news.ycombinator.com, github.com, example.com)
  • Bedrock API token usage measured from the Converse API usage field (actual billed tokens)
  • MCP response sizes measured from tool response character counts
  • Playwright MCP installed from npx @playwright/mcp@latest on the run date; the resolved version was not recorded
  • Chrome DevTools MCP installed from npx -y chrome-devtools-mcp@latest on the run date; the resolved version was not recorded
  • OpenBrowser MCP v0.1.26 tested via uvx openbrowser-ai[mcp]==0.1.26 --mcp
  • All tests run on macOS, M-series Apple Silicon
  • Benchmark scripts: benchmarks/e2e_llm_benchmark.py, benchmarks/e2e_llm_stats.py
  • Results: benchmarks/e2e_llm_stats_results.json