Timothy Olaleke 85739d3f6a fix(#19): serialize V8 work across pages with process-wide async lock (#36)
* fix(#19): serialize V8 work across pages with process-wide async lock

Concurrent CDP clients (≥5) trip V8's
`heap->isolate() == Isolate::TryGetCurrent()` invariant and abort the
process via V8_Fatal — no Rust panic, just abort(3). Each Page owns its
own JsRuntime/Isolate and they all run on a single OS thread via
LocalSet+spawn_local; when two pages' V8-touching futures interleave
across an .await, V8 trips the check.

This adds a process-wide tokio::sync::Mutex (obscura_js::v8_lock) that
is acquired:

  - At the top of obscura_cdp::dispatch::dispatch (covers every CDP
    handler that may call into a JsRuntime).
  - In server.rs's process_with_interception spawn_local that runs
    page.navigate_with_wait + execute_preload_script (this path
    bypasses dispatch).

The guard keeps each V8-touching block contiguous on the thread: V8
fully exits one Isolate before another page is allowed in. Trade-off:
non-V8 handlers (Browser.*, Storage.*, Emulation.*) also serialize,
but those are cheap and the safety win dominates.

The properly concurrent fix is to pin each JsRuntime to its own OS
thread and message-pass; tracked as the larger #19 follow-up.

Adds an opt-in smoke test (`--ignored`) that boots the CDP server and
runs 5 parallel clients through createTarget+navigate. Note: the
deterministic abort needs subresource-fetch yields (page.rs:285
join_all) which data: URLs skip — full repro requires JS-heavy real
traffic (issue author used scrapegraphai.com).

* fix(#19): suspend other Isolates and defer foreign CDP during interception nav

Address maintainer feedback on PR #36 (comment 4328238087): the V8 fatal
still reproduces with `Fetch.enable patterns:["*"]` + 5 concurrent
`Page.navigate`s. The `obscura_js::v8_lock` mutex was insufficient
because V8's actual invariant is "only one Isolate may be entered per
OS thread", and the mutex can't enforce that across `.await` points
inside V8 ops.

Two distinct paths violated the invariant:

1. Foreign CDP messages dispatched while a nav task was running.
   `process_with_interception`'s `select!` routed every non-Fetch
   message through `process_cdp_message → dispatch → page::handle::*`.
   `get_session_page_mut` then `suspend_js`'es other pages (drops
   their `JsRuntime` → `Isolate::Exit`) while the spawned nav task is
   parked at `op_fetch_url`'s `resolve_rx.await` with its Isolate
   still entered. Drop trips
   `heap->isolate() == Isolate::TryGetCurrent()`.

2. Subsequent navs through `process_with_interception` did not
   suspend prior pages. After nav-1 completes (page-1's JsRuntime
   stays alive in `ctx.pages`), nav-2's `init_js` constructs
   Isolate-2 while Isolate-1 is still current. The next V8 scope
   unwind aborts via `Context::Exit`.

Fix (3 changes in `crates/obscura-cdp/src/server.rs`):

- `cdp_processor`: add a `VecDeque<ServerMessage> deferred` drained
  before each new `rx.recv()`, so deferred messages are processed
  with no nav task in flight.
- `process_with_interception` `select!`: only handle `NewConnection`
  and `Fetch.{continue,fulfill,fail}Request` inline (safe — no V8
  enter on this side); push everything else into `deferred`.
- `process_with_interception` entry: explicitly `suspend_js()` every
  other page in `ctx.pages` before spawning the nav task, mirroring
  the invariant the regular dispatch path already enforces via
  `get_session_page_mut`.

Verification:

- New test `crates/obscura-cdp/tests/concurrent_navigations_with_fetch.rs`
  is the maintainer's exact repro (5 clients, `Fetch.enable patterns:["*"]`,
  concurrent `Page.navigate` to https://example.com, auto-reply
  `Fetch.continueRequest`). Aborts on PR #36 head, passes after fix
  (5/5 stable runs, ~1.2s each).
- Existing data-URL smoke test (`concurrent_navigations`) still passes.
- Full workspace test suite (90 tests) still passes.

Run with:

  cargo test -p obscura-cdp --test concurrent_navigations_with_fetch \
    -- --nocapture --ignored

Honest caveat. This is still the lock/serialize approach — it converts
the abort into latency. The properly concurrent fix (Option 2 in
issue #19: pin each `JsRuntime` to its own OS thread and message-pass)
remains the right long-term direction.

* fix(cdp): bound deferred message queue to prevent OOM during stalled navigation

PR #36 comment 4341743194: process_with_interception deferred foreign
CDP messages into an unbounded VecDeque while a navigation task was in
flight. A stalled navigation (e.g. waiting for Fetch interception
resolution) could accumulate arbitrary queued messages and OOM the
process.

Add MAX_DEFERRED_MESSAGES = 256. When the cap is reached, return an
explicit CDP error (-32000) to the client instead of silently dropping
or growing without bound.

All 90 workspace tests pass.
2026-05-09 18:26:38 +02:00
2026-04-13 12:30:28 +02:00
2026-04-13 12:30:28 +02:00
2026-04-13 11:20:11 +02:00
2026-04-13 11:20:11 +02:00
2026-05-09 17:30:24 +02:00

Obscura

Obscura

The open-source headless browser for AI agents and web scraping.
Lightweight, stealthy, and built in Rust.


Obscura is a headless browser engine written in Rust, built for web scraping and AI agent automation. It runs real JavaScript via V8, supports the Chrome DevTools Protocol, and acts as a drop-in replacement for headless Chrome with Puppeteer and Playwright.

Why Obscura over headless Chrome?

Designed for automation at scale, not desktop browsing.

Metric Obscura Headless Chrome
Memory 30 MB 200+ MB
Binary size 70 MB 300+ MB
Anti-detect Built-in None
Page load 85 ms ~500 ms
Startup Instant ~2s
Puppeteer Yes Yes
Playwright Yes Yes

🎉 10,000 stars and what's next

I'm working on Obscura Cloud the hosted version, with managed infrastructure, residential proxies, and dedicated support. For people who want the engine without operating it themselves.

The open-source engine stays Apache-2.0, fully featured. No feature gating, ever.

Get on the waitlist →

Install

Download

Grab the latest binary from Releases:

# Linux x86_64
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
tar xzf obscura-x86_64-linux.tar.gz
./obscura fetch https://example.com --eval "document.title"

# macOS Apple Silicon
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-aarch64-macos.tar.gz
tar xzf obscura-aarch64-macos.tar.gz

# macOS Intel
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-macos.tar.gz
tar xzf obscura-x86_64-macos.tar.gz

# Windows
Download the `.zip` from the releases page and extract it manually.

No Chrome, no Node.js, no dependencies. Release archives include both obscura and obscura-worker; keep them in the same directory for the parallel scrape command.

Linux release builds target Ubuntu 22.04 so the downloaded binary remains usable on common LTS servers with glibc 2.35+.

Build from source

git clone https://github.com/h4ckf0r0day/obscura.git
cd obscura
cargo build --release

# With stealth mode (anti-detection + tracker blocking)
cargo build --release --features stealth

Requires Rust 1.75+ (rustup.rs). First build takes ~5 min (V8 compiles from source, cached after).

Quick Start

Fetch a page

# Get the page title
obscura fetch https://example.com --eval "document.title"

# Extract all links
obscura fetch https://example.com --dump links

# Render JavaScript and dump HTML
obscura fetch https://news.ycombinator.com --dump html

# Write dump or eval output to a file
obscura fetch https://example.com --dump text --output page.txt

# Wait for dynamic content
obscura fetch https://example.com --wait-until networkidle0

# Bound navigation time for slow or broken pages
obscura fetch https://example.com --timeout 10

Start the CDP server

obscura serve --port 9222

# With stealth mode (anti-detection + tracker blocking)
obscura serve --port 9222 --stealth

Scrape in parallel

obscura scrape url1 url2 url3 ... \
  --concurrency 25 \
  --eval "document.querySelector('h1').textContent" \
  --format json

# Suppress scrape progress on stderr for script-friendly output
obscura scrape https://example.com --quiet --format json

Puppeteer / Playwright

Puppeteer

npm install puppeteer-core
import puppeteer from 'puppeteer-core';

const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser',
});

const page = await browser.newPage();
await page.goto('https://news.ycombinator.com');

const stories = await page.evaluate(() =>
  Array.from(document.querySelectorAll('.titleline > a'))
    .map(a => ({ title: a.textContent, url: a.href }))
);
console.log(stories);

await browser.disconnect();

Playwright

npm install playwright-core
import { chromium } from 'playwright-core';

const browser = await chromium.connectOverCDP({
  endpointURL: 'ws://127.0.0.1:9222',
});

const page = await browser.newContext().then(ctx => ctx.newPage());
await page.goto('https://en.wikipedia.org/wiki/Web_scraping');
console.log(await page.title());

await browser.close();

Form submission & login

await page.goto('https://quotes.toscrape.com/login');
await page.evaluate(() => {
  document.querySelector('#username').value = 'admin';
  document.querySelector('#password').value = 'admin';
  document.querySelector('form').submit();
});
// Obscura handles the POST, follows the 302 redirect, maintains cookies

Benchmarks

Page load:

Page Obscura Chrome
Static HTML 51 ms ~500 ms
JS + XHR + fetch 84 ms ~800 ms
Dynamic scripts 78 ms ~700 ms

Stealth Mode

Enable with --features stealth.

Anti-fingerprinting

  • Per-session fingerprint randomization (GPU, screen, canvas, audio, battery)
  • Realistic navigator.userAgentData (Chrome 145, high-entropy values)
  • event.isTrusted = true for dispatched events
  • Hidden internal properties (Object.keys(window) safe)
  • Native function masking (Function.prototype.toString() → [native code])
  • navigator.webdriver = undefined (matches real Chrome)

Tracker Blocking

  • 3,520 domains blocked
  • Blocks analytics, ads, telemetry, and fingerprinting scripts
  • Prevents trackers from loading entirely
  • Enabled automatically with --stealth

CDP API

Obscura implements the Chrome DevTools Protocol for Puppeteer/Playwright compatibility.

Domain Methods
Target createTarget, closeTarget, attachToTarget, createBrowserContext, disposeBrowserContext
Page navigate, getFrameTree, addScriptToEvaluateOnNewDocument, lifecycleEvents
Runtime evaluate, callFunctionOn, getProperties, addBinding
DOM getDocument, querySelector, querySelectorAll, getOuterHTML, resolveNode
Network enable, setCookies, getCookies, setExtraHTTPHeaders, setUserAgentOverride
Fetch enable, continueRequest, fulfillRequest, failRequest (live interception)
Storage getCookies, setCookies, deleteCookies
Input dispatchMouseEvent, dispatchKeyEvent
LP getMarkdown (DOM-to-Markdown conversion)

CLI Reference

obscura serve

Start a CDP WebSocket server.

Flag Default Description
--port 9222 WebSocket port
--proxy — HTTP/SOCKS5 proxy URL
--stealth off Enable anti-detection + tracker blocking
--workers 1 Number of parallel worker processes
--obey-robots off Respect robots.txt

obscura fetch <URL>

Fetch and render a single page.

Flag Default Description
--dump html Output: html, text, or links
--eval — JavaScript expression to evaluate
--wait-until load Wait: load, domcontentloaded, networkidle0
--timeout 30 Maximum navigation time in seconds
--selector — Wait for CSS selector
--stealth off Anti-detection mode
--output — Write dump or eval output to a file
--quiet off Suppress banner

obscura scrape <URL...>

Scrape multiple URLs in parallel with worker processes.

Flag Default Description
--concurrency 10 Parallel workers
--eval — JS expression per page
--format json Output: json or text
--quiet off Suppress scrape progress on stderr

License

Apache 2.0


Description
Obscura browser engine source used by Naori, based on h4ckf0r0day/obscura
Readme 24 MiB
Languages
Rust 94.7%
Python 3.6%
HTML 1.4%
JavaScript 0.2%