headless_chrome (and older Puppeteer) wrap every CDP call inside
Target.sendMessageToTarget. Obscura was returning "Unknown Target
method" for the wrapper, which broke navigation through the high-level
client API even though the WebSocket itself connected.
Add a wrapper-level handler that:
1. Parses the inner JSON-encoded CDP request from params.message
2. Recursively dispatches it (boxed future to sidestep async-fn
recursion limits) using the wrapper's sessionId so routing works
3. Re-emits the inner response as a Target.receivedMessageFromTarget
event — the legacy contract these clients listen on instead of
correlating responses by id
4. Returns {} for the wrapper itself
This addresses the headless_chrome leg of #26. The chromiumoxide
"data did not match any variant of untagged enum Message" decode
failure is a separate event/response shape issue — needs a focused
investigation against a chromiumoxide repro that's not in the issue,
so it's left out of this PR.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
* fix(#19): serialize V8 work across pages with process-wide async lock
Concurrent CDP clients (≥5) trip V8's
`heap->isolate() == Isolate::TryGetCurrent()` invariant and abort the
process via V8_Fatal — no Rust panic, just abort(3). Each Page owns its
own JsRuntime/Isolate and they all run on a single OS thread via
LocalSet+spawn_local; when two pages' V8-touching futures interleave
across an .await, V8 trips the check.
This adds a process-wide tokio::sync::Mutex (obscura_js::v8_lock) that
is acquired:
- At the top of obscura_cdp::dispatch::dispatch (covers every CDP
handler that may call into a JsRuntime).
- In server.rs's process_with_interception spawn_local that runs
page.navigate_with_wait + execute_preload_script (this path
bypasses dispatch).
The guard keeps each V8-touching block contiguous on the thread: V8
fully exits one Isolate before another page is allowed in. Trade-off:
non-V8 handlers (Browser.*, Storage.*, Emulation.*) also serialize,
but those are cheap and the safety win dominates.
The properly concurrent fix is to pin each JsRuntime to its own OS
thread and message-pass; tracked as the larger #19 follow-up.
Adds an opt-in smoke test (`--ignored`) that boots the CDP server and
runs 5 parallel clients through createTarget+navigate. Note: the
deterministic abort needs subresource-fetch yields (page.rs:285
join_all) which data: URLs skip — full repro requires JS-heavy real
traffic (issue author used scrapegraphai.com).
* fix(#19): suspend other Isolates and defer foreign CDP during interception nav
Address maintainer feedback on PR #36 (comment 4328238087): the V8 fatal
still reproduces with `Fetch.enable patterns:["*"]` + 5 concurrent
`Page.navigate`s. The `obscura_js::v8_lock` mutex was insufficient
because V8's actual invariant is "only one Isolate may be entered per
OS thread", and the mutex can't enforce that across `.await` points
inside V8 ops.
Two distinct paths violated the invariant:
1. Foreign CDP messages dispatched while a nav task was running.
`process_with_interception`'s `select!` routed every non-Fetch
message through `process_cdp_message → dispatch → page::handle::*`.
`get_session_page_mut` then `suspend_js`'es other pages (drops
their `JsRuntime` → `Isolate::Exit`) while the spawned nav task is
parked at `op_fetch_url`'s `resolve_rx.await` with its Isolate
still entered. Drop trips
`heap->isolate() == Isolate::TryGetCurrent()`.
2. Subsequent navs through `process_with_interception` did not
suspend prior pages. After nav-1 completes (page-1's JsRuntime
stays alive in `ctx.pages`), nav-2's `init_js` constructs
Isolate-2 while Isolate-1 is still current. The next V8 scope
unwind aborts via `Context::Exit`.
Fix (3 changes in `crates/obscura-cdp/src/server.rs`):
- `cdp_processor`: add a `VecDeque<ServerMessage> deferred` drained
before each new `rx.recv()`, so deferred messages are processed
with no nav task in flight.
- `process_with_interception` `select!`: only handle `NewConnection`
and `Fetch.{continue,fulfill,fail}Request` inline (safe — no V8
enter on this side); push everything else into `deferred`.
- `process_with_interception` entry: explicitly `suspend_js()` every
other page in `ctx.pages` before spawning the nav task, mirroring
the invariant the regular dispatch path already enforces via
`get_session_page_mut`.
Verification:
- New test `crates/obscura-cdp/tests/concurrent_navigations_with_fetch.rs`
is the maintainer's exact repro (5 clients, `Fetch.enable patterns:["*"]`,
concurrent `Page.navigate` to https://example.com, auto-reply
`Fetch.continueRequest`). Aborts on PR #36 head, passes after fix
(5/5 stable runs, ~1.2s each).
- Existing data-URL smoke test (`concurrent_navigations`) still passes.
- Full workspace test suite (90 tests) still passes.
Run with:
cargo test -p obscura-cdp --test concurrent_navigations_with_fetch \
-- --nocapture --ignored
Honest caveat. This is still the lock/serialize approach — it converts
the abort into latency. The properly concurrent fix (Option 2 in
issue #19: pin each `JsRuntime` to its own OS thread and message-pass)
remains the right long-term direction.
* fix(cdp): bound deferred message queue to prevent OOM during stalled navigation
PR #36 comment 4341743194: process_with_interception deferred foreign
CDP messages into an unbounded VecDeque while a navigation task was in
flight. A stalled navigation (e.g. waiting for Fetch interception
resolution) could accumulate arbitrary queued messages and OOM the
process.
Add MAX_DEFERRED_MESSAGES = 256. When the cap is reached, return an
explicit CDP error (-32000) to the client instead of silently dropping
or growing without bound.
All 90 workspace tests pass.
- obscura-net/cookies.rs: #[allow(dead_code)] on CookieEntry.same_site
(parsed today, kept for future SameSite enforcement).
- obscura-js/runtime.rs: drop vestigial BOOTSTRAP_JS const; bootstrap.js
is already embedded into the V8 snapshot via build.rs.
- obscura-js/ops.rs: prefix the four unused fields in
InterceptResolution::Continue with _.
- obscura-cdp/dispatch.rs: drop unused InterceptResolution import.
- obscura-cdp/domains/fetch.rs: drop unused Arc / Mutex / CdpEvent imports.
- obscura-cdp/domains/input.rs: prefix _button / _click_count; CDP
parses them but the JS click stub is hardcoded for now.
- obscura-cdp/server.rs: prefix _processor_handle and _ctx; drop dead
initial Ok(())/None for navigate_result/page_back - they are
unconditionally overwritten in the only loop break arm.
cargo build --workspace: 0 warnings.
cargo test --workspace: 113 passed.
obscura serve --user-agent <ua> only logged the value via tracing::info!
and never passed it to the CDP server, so every navigation kept using
the default Chrome UA string and httpbin.org/user-agent reported the
default rather than the override.
Plumb the option end-to-end:
- BrowserContext::with_full_options(id, proxy, stealth, user_agent)
initializes both BrowserContext::user_agent and the underlying
ObscuraHttpClient::user_agent RwLock at construction time so the
first navigation already sees the override.
- CdpContext::new_with_full_options forwards the UA to the default
BrowserContext.
- obscura_cdp::start_with_full_options accepts the UA and passes it
through cdp_processor.
- main.rs Serve arm calls start_with_full_options with the parsed UA.
- run_multi_worker_serve forwards --user-agent <ua> when spawning
worker subprocesses so multi-worker mode honors the flag too.
Existing two-arg start_with_options and three-arg with_options stay
as thin shims (call the full variant with None) so older callers and
the public crate API remain source-compatible.
Three new unit tests in obscura-browser cover the propagation, the
Chrome default fallback, and the legacy with_options path.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
Puppeteer's FrameManager.initialize calls Audits.enable on connect.
Without it puppeteer.connect() fails with "Unknown domain: Audits"
before any user code runs. Playwright doesn't enable Audits, which is
why the workaround in #57 sidesteps the issue.
Add Audits to the existing accepted-but-no-op domain list, alongside
Log/Performance/Security/etc. that already swallow Puppeteer's connect
chatter. No new module needed.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
Fixes#1: replace unwrap() panics in run_multi_worker_serve with match on the Result.
Fixes#2: route /json round-robin via worker_port instead of hardcoded port+1.
Fixes#3: return 502 Bad Gateway when a worker is unreachable instead of silently dropping the connection.
Fixes#5: thread --stealth through start_with_options -> CdpContext -> BrowserContext::with_options so the flag actually has an effect in serve mode (was let _ before). Log message now reflects the actual capability based on the stealth cargo feature being compiled in.
Fixes#15: try/catch the parentNode assignment in the attachShadow polyfill so it doesn't throw a TypeError against the read-only Node.parentNode getter (Cloudflare Turnstile triggers this path).
Brings Obscura's CDP server in line with what Playwright's
`chromium.connectOverCDP()` expects. Each fix is small and surgical;
together they make `connect → newContext → newPage → goto → page.title()`
work end-to-end against `https://example.com`.
1. **server.rs — emit pending events before command response.**
Chromium CDP semantics require side-effect events (e.g.
`Target.targetCreated` + `Target.attachedToTarget` from
`Target.createTarget`) to arrive *before* the originating command's
response. Playwright's wire reader assumes this ordering — it
resolves the response promise on the response and then synchronously
looks up the freshly-created session. Reversing the order makes that
lookup miss and the page hang at `newPage()`.
2. **domains/browser.rs — accept Browser.setWindowBounds as a no-op.**
Playwright calls `Browser.setWindowBounds` during teardown. Returning
"Unknown Browser method" causes Playwright to treat the target as
broken and tear it down mid-test, surfacing as "Frame has been
detached" on the next `page.title()`.
3. **obscura-browser/page.rs — frame_id == target_id for main frame.**
Was `format!("{id}.1", id)`. Playwright's frame manager indexes
frames by frame id and looks up the main frame by target id, so the
`.1` suffix made every main-frame lookup miss.
4. **domains/page.rs + dispatch.rs — track isolated worlds and re-emit
them post-navigation.** Playwright registers a utility world via
`Page.createIsolatedWorld` and expects `Runtime.executionContextCreated`
for each registered world after every navigation. Without re-emission,
the utility-world handle goes stale post-`goto` and `page.title()`
hangs waiting for the world's execution context. Falls back to the
legacy `__puppeteer_utility_world__24.40.0` name when no worlds have
been registered, preserving prior Puppeteer support.
5. **obscura-js/runtime.rs — strip trailing semicolons before wrapping.**
Playwright's utility-script expression is an IIFE ending in `})();`.
Both `wrap_expression` (CDP `Runtime.evaluate`) and `evaluate_for_cdp`
(the `meta_code` IIFE) embed the user expression inside `({EXPR})` /
`return ({EXPR});`. With the trailing `;` left in, this produces
`({...;})` — a SyntaxError. Parse errors are not catchable, so the
surrounding try/catch never fires and the call silently returns
`undefined`, surfacing as `page.title() === ""`.
Verified end-to-end against `https://example.com`:
[probe] connect OK
[probe] newContext OK
[probe] newPage OK
[probe] goto OK
[probe] title = Example Domain
