57949a0efc63530ff0de4bc566cc369127d48f6c
23
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 SGavrl
|
d0b6fc4ffd |
fix(cdp): emit Network events for script-initiated requests (#415)
Requests made by page JS (fetch/XHR/dynamic script and stylesheet loads)
run in the V8 op layer and never produced a NetworkEvent, so CDP emitted
Network.requestWillBeSent / responseReceived only for the static
navigation subresources. Puppeteer/Playwright page.on('request') and
on('response') therefore missed every XHR/JSON response, which breaks
response capture (and is the root cause of the Aviasales half of #394).
op_fetch_url now records a JsNetworkEvent keyed by the same fetch-{N} id
under which it already stores the response body. The Page drains these
into its network_events before the CDP layer emits, so they flow through
emit_navigation_events as requestWillBeSent / responseReceived /
loadingFinished alongside the document subresources, and
Network.getResponseBody resolves for them via the shared id. Applied to
both navigation paths (server process_navigation and in-process
do_navigate). The queue is capped at 4096 events per page.
Closes #406.
|
||
|
Andrey Mnatsakanov
|
c915a11f59 |
fix(cdp): truncate client-controlled log previews on char boundaries (#397)
server.rs logged previews of attacker-controlled strings with raw byte slicing: `&text[..text.len().min(200)]` for a malformed CDP frame (inside a `warn!`, which is active at the worker's default log level) and the same for an intercepted event JSON. `&s[..n]` panics when byte `n` falls inside a multi-byte UTF-8 character, so a single WebSocket frame that is invalid JSON, >=201 bytes, with a multi-byte char straddling byte 200 panics the CDP processor task — an unauthenticated one-frame crash. page.rs had the same pattern (behind --verbose debug logging) for evaluated JS expressions. Add truncate_on_char_boundary (str::floor_char_boundary is still unstable) and route all four sites through it. Regression test constructs the exact 'a'*199 + '€' input, asserts the old raw slice panics (via catch_unwind) and the helper truncates on the boundary instead. |
||
|
Flo
|
82d20bcb7c | fix(cdp): allow large WebSocket response frames (<64Mb) (#405) | ||
|
Andrey Mnatsakanov
|
68234970b1 |
fix(cdp): apply Fetch.continueRequest overrides (#373)
handle_fetch_resolution built InterceptResolution::Continue with url, method,
headers, and body hardcoded to None, so Fetch.continueRequest silently sent the
request unmodified. Playwright route.continue({ headers, url, postData }) and
Puppeteer request.continue reported success while their overrides were dropped.
op_fetch_url already applies all four (and re-validates a rewritten URL through
the SSRF gate); only the CDP layer was discarding them. Read url, method,
headers, and postData from the continueRequest params and pass them through.
Fixes #365.
|
||
|
Andrey Mnatsakanov
|
daa0d62fbd |
fix(cdp): route navigation on exact method, not substring (#372)
The cdp_processor decided whether to take the spawn-and-defer navigation path
with `text.contains("Page.navigate")`, which also matches
`Page.navigateToHistoryEntry`. goBack / goForward (both Puppeteer and
Playwright drive Page.navigateToHistoryEntry) were misrouted into the raw
navigate path, which reads an absent `url` param and fails with
"Invalid URL: relative URL without a base", so history navigation never
reached its real handler. Any frame merely embedding the literal
"Page.navigate" (e.g. a Runtime.evaluate expression) was misrouted the same
way.
Match on the parsed CDP method instead, so only an exact Page.navigate takes
the navigation path.
Fixes #363.
|
||
|
SGavrl
|
b63125faed |
cdp: flush cookies on SIGTERM in serve (#333)
serve --storage-dir saved cookies only on Ctrl-C (SIGINT). docker stop and kill send SIGTERM, which was not caught, so the save_cookies() call at shutdown never ran and cookies were lost on restart. Watch SIGTERM as well as Ctrl-C on Unix. |
||
|
Sander Spilleman
|
9ae36764da | feat(stealth): many stealth improvements | ||
|
SGavrl
|
1b9be08767 |
fix(cdp): exit on Ctrl-C while serving (#225) (#226)
The LocalSet accept loop blocked on ws_rx.recv() forever because the detached accept thread kept a ws_tx clone alive, so the channel never closed and start_with_full_serve_options never returned. cdp_processor caught Ctrl-C and saved cookies, but that only ended its own task. Add a shared shutdown flag and Notify. On Ctrl-C cdp_processor sets the flag and notifies; the accept loop selects on the notify and breaks, so run_until returns and the process exits. The accept thread checks the flag between connections so it stops and drops ws_tx on the slow path. Closes #225 |
||
|
SGavrl
|
89692f3cb7 |
Perf/cdp network and lockfree (#214)
* perf(cdp): TCP_NODELAY + zero-buffer WS writes shave 130ms off newPage CDP traffic during browser.newPage() and Page.navigate is many small (~100-byte) request/response frames. Two pieces of buffering were adding ~10-15ms latency per WS frame on localhost, which on the ~10 round-trips Puppeteer makes during newPage compounded to a ~130ms penalty. 1. TCP_NODELAY on every accepted WS stream. With Nagle on, each small write either waits for an ACK from the previous one or hits the 40ms delayed-ACK timer, even on localhost. 2. WebSocketConfig.write_buffer_size = 0 on accept_async. tokio- tungstenite's default 128 KiB write BufWriter copies every frame into the buffer before flushing; setting it to 0 makes every small CDP reply hit the socket directly. Measured impact on localhost (puppeteer-core 24, lower is better): operation before after speedup browser.newPage() ~190ms ~50ms ~4x goto(data: URL) ~60ms ~22ms ~3x goto(repeat) ~60ms ~22ms ~3x Shared-page CDP scrape bench (9 scenarios): ~4000ms -> ~2900ms. 28/28 integration tests pass. * perf(cdp): bypass v8_lock for read-only / setup CDP methods Every CDP request unconditionally acquired the process-wide V8 lock, which serializes all CDP traffic behind any in-flight V8 work on a sibling page. During Puppeteer's browser.newPage() the ~8 setup calls (Page.enable, Runtime.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, etc.) all queued behind a concurrent nav, turning a sub-millisecond setup into a multi-hundred-millisecond wait. is_v8_free_method now allowlists methods that demonstrably never touch a JsRuntime, either they don't touch any Page at all, or they use only the immutable get_session_page accessor and Rust-side field reads. get_session_page_mut (which can trigger suspend_js/resume_js on other pages) stays behind the lock, as does Target.createTarget (which calls page.navigate). Allowlisted: most of Target/Browser/Storage, plus Page.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, Runtime.enable, Network.* cookie ops, Fetch.continue/fulfill/fail. Measured on the puppeteer-core 24 CDP scrape bench: browser.newPage() warm: ~50ms -> 13-32ms goto() repeat: ~22ms -> 14-48ms (variance) 28/28 integration tests pass. 18/18 realistic scrape scenarios pass. |
||
|
SGavrl
|
b98e457428 |
engine: large compat + reliability pass (#206)
CDP / navigation: - Route all Page.navigate through process_with_interception so the parent dispatcher stays responsive during long navs (fixes Target.createTarget timeouts after a handful of newPage calls). - Default waitUntil to domcontentloaded. Puppeteer and Playwright don't pass it server-side and wait client-side via lifecycleEvent, so the old Load default blocked nearly every nav on slow sites. - 30s navigation hard ceiling, OBSCURA_NAV_TIMEOUT_MS to override. - execute_scripts loop now checks wall-clock deadline per iteration. - Emit Network.requestWillBeSent before Page.frameNavigated so Playwright's FrameManager attaches the navigation response (#189). - Use loaderId as the Document request id so page.goto() resolves to a Response instead of null (#189). - Fresh executionContextId per navigation, no more 'Cannot find context with specified id' after the first nav (#192). - POST navigation params plumbed through the intercept path. DOM / Web API surface: - MutationObserver: subtree filter walks parent chain, fires on textContent / CharacterData.data / innerHTML, type filter respected, oldValue passed. - IntersectionObserver: multi-fire (initial + 120/500/1500ms), disconnect / unobserve / takeRecords, root / rootMargin / thresholds. - ResizeObserver: per-target firing from getBoundingClientRect, drop the no-op override that was clobbering the real class. - crypto.subtle.digest: real SHA-1/256/384/512 via op_subtle_digest. - DOMParser / document.implementation.createHTMLDocument / createDocument: detached doc with body, head, documentElement, querySelector*, getElementById, getElementsByTagName. - WebSocket / EventSource / BroadcastChannel: real EventTarget, open fires on microtask. - indexedDB: request, object store, transaction, database with async onsuccess and in-memory Map storage. - getComputedStyle: width/height/top/left from bbox so React virtualization libs render items. Site compatibility went from 3/40 to 175/175 reachable sites in the sweep. Perf is equal or faster than pristine main on every site tested (DCL default removes the block on long-tail subresources). 28/28 integration tests pass. |
||
|
|
ea89f050cd |
Fix puppeteer exposeFunction end-to-end: real bindings, preload timing, URL coercion (#166)
* fix(js): coerce URL constructor input/base to string
The polyfill at bootstrap.js called `base.match(...)` directly, which
threw `TypeError: base.match is not a function` whenever callers passed
a Location object (e.g. `new URL(href, window.location)`), a URL
instance, or any other non-string with a sensible toString. Per the
WHATWG URL Standard both `url` and `base` are stringified before
parsing.
Coerce explicitly so the existing regex paths keep working and the
constructor matches real-browser behaviour. AngularJS, jQuery, and
anything that resolves a relative URL against `window.location` now
loads under Obscura.
* fix(browser,cdp): run preload scripts before page scripts
`Page.addScriptToEvaluateOnNewDocument` was executed AFTER the page's
own scripts (and even after the load event), which silently broke
every CDP contract that depends on early injection — most visibly
puppeteer's `page.exposeFunction`, whose user-facing wrapper has to
override `globalThis[name]` before any `<script>` tag in the document
runs.
Move the registered preload sources onto `Page` and execute them
inside `navigate_single`, right after `__documentReadyState__ =
'loading'` and before the parser-discovered scripts. The CDP layer
hands the current preload set to the page via `set_preload_scripts`
before kicking off navigation, replacing the two post-navigate
`execute_preload_script` loops in `domains/page.rs` and `server.rs`.
* feat(cdp,js): emit Runtime.bindingCalled for addBinding shims
`Runtime.addBinding` was a no-op stub that installed an empty
`globalThis[name] = function () { return null; }` placeholder. Page JS
calls were silently swallowed and the Node-side callback registered
via puppeteer's `page.exposeFunction` never fired.
Wire the binding all the way through:
1. New `op_binding_called(name, payload)` op forwards every page-side
call into a `pending_binding_calls` queue on `ObscuraState`.
2. `Page::take_pending_binding_calls` drains the queue.
3. `dispatch()` empties every page's queue after each CDP request and
pushes one `Runtime.bindingCalled` event per entry into
`ctx.pending_events`, so the writer task forwards them to the
client like any other CDP event.
4. The `addBinding` handler installs a real shim that calls the op,
registers it as a preload script (keyed `__obscura_binding__<name>`)
so it survives navigations, and evaluates it now so the binding is
usable on the current document.
5. Implement `removeBinding` to mirror `addBinding`.
The shim coerces non-string arguments via `JSON.stringify`, matching
the wire contract that puppeteer's `exposeFunction` wrapper depends
on (`Runtime.bindingCalled.params.payload` is always a string).
* fix(cdp): emit canAccessOpener in all TargetInfo payloads, add aarch64 cross-compile CI
Merges PR #165 and PR #172.
PR #165: The CDP spec marks canAccessOpener as a required boolean on
TargetInfo but Obscura was omitting it. Strict Rust clients like
chromiumoxide reject the event entirely and panic with "Created target
not present" when the field is absent. Added canAccessOpener: false
to every TargetInfo emitted by setDiscoverTargets, getTargets,
createTarget, attachToBrowserTarget, attachToTarget, and getTargetInfo.
PR #172: Added a cross-compile matrix entry for aarch64-unknown-linux-gnu
running on ubuntu-22.04 x86_64, producing obscura-aarch64-linux-cross.
This is a fallback for when GitHub ARM runners are unavailable without
removing the native ARM build.
* fix(cdp): keep HTTP control plane reachable while V8 JS evaluation blocks (#173)
fix(cdp): unblock HTTP control plane during synchronous V8 execution
Fixes #62.
Previously, the TCP accept loop and CDP processor shared a single-threaded
tokio LocalSet. Long synchronous JS evaluations starved the accept loop,
causing incoming connections to hang in the kernel backlog and effectively
killing the HTTP control plane.
This moves the TCP listener and `/json/*` endpoints to a dedicated OS
thread using `std::net::TcpListener`. HTTP requests are now handled
directly via blocking I/O on this thread, completely bypassing V8.
WebSocket connections are handed off to the LocalSet via a bounded mpsc
channel (capacity 128). If V8 stalls and the queue fills, new WS
connections are dropped (ECONNRESET) rather than buffering unboundedly
to prevent OOMs.
Includes a new integration test to verify the control plane stays
responsive during a 5s JS loop, plus inline documentation clarifying
thread lifetime semantics and a minor test TOCTOU race.
* fix(js,dom): add Storage global, CharacterData ChildNode methods, fix remove/isConnected (#176)
fix(dom): add missing DOM/JS shims for modern frameworks
Closes #163.
Adds missing DOM and JS shims required by modern frameworks (Svelte 5,
Vue, Lit) and consent-management SDKs to execute without errors:
- Define the `Storage` global constructor to resolve `ReferenceError`s
and fix `Storage.prototype` access.
- Apply standard ChildNode methods (`before`, `after`, `replaceWith`,
`remove`) to `CharacterData.prototype`. This prevents `TypeError`s when
frameworks call these on Text or Comment nodes during hydration.
- Update `Element.prototype.isConnected` to perform a genuine parent-walk
rather than returning a hardcoded `true`.
- Fix `Element.prototype.remove` to properly clean up the internal ID
index so removed elements are no longer queryable.
- Remove outdated no-op stubs on the `Element` class that were shadowing
the proper polyfill implementations.
* feat(cli): add --dump assets to inventory page sub-resources (124) (#174)
feat: add `--dump assets` to enumerate page sub-resources
Adds a `--dump assets` flag to output all page-referenced sub-resources
(scripts, stylesheets, images, iframes, media, embeds, objects) as NDJSON,
allowing users to easily replay the asset graph.
- Output format: `{"url":"https://example.test/app.js","type":"script"}`
- URLs are resolved against the page origin (matching `--dump links`).
- The `type` tag is derived from the source element. `<link>` elements map
their `rel` attribute to specific types (stylesheet, icon, manifest, preload,
prefetch, modulepreload, dns-prefetch, preconnect, alternate), falling back
to "link" if missing or unknown.
- Adds 7 tests covering URL resolution, rel-mapping, supported selectors,
and empty-attribute skipping.
- Updates README dump-format table and examples.
* fix(cdp,js): playwright actionability stubs and objectId resolve (#178)
Playwright fails locator actions on Obscura for two avoidable reasons
that are separate from the missing layout engine. Both have trivial
fixes that unblock the actionability polling loop.
- Element.checkVisibility(opts) was undefined. Playwright >= 1.25 calls
it before every input event and retries until its action timeout if
the method isn't a function. Add a stub that returns true. Without a
layout pass we can't compute it for real, but a permissive stub still
beats the undefined-function path.
- ARIA reflection properties were missing on Element (role, ariaLabel,
ariaSelected, ariaExpanded, ariaHidden, ariaChecked, ariaDisabled,
ariaRoleDescription). Playwright's getByRole and getByLabel locators
read them. Reflect through to the underlying aria-* attributes for
both read and write.
- DOM.getBoxModel and DOM.getContentQuads ignored the objectId param.
Playwright passes only objectId (from a prior DOM.resolveNode), and
the handlers silently defaulted to node 0, so dispatchMouseEvent
always clicked the wrong element. Extract a resolve_node_id helper
that honors nodeId, backendNodeId, and objectId in that order.
- Page.captureScreenshot and Page.captureSnapshot now return a clear
"not supported, use a real Chromium for the screenshot leg" error
instead of the generic "Unknown Page method" fallback. Matches the
pattern already in place for Page.printToPDF.
Partial fix for #45. The architectural blocker (no layout engine, so
multi-element click coordinates are wrong) is unchanged.
* fix(mcp): allow browser MCP clients through CORS preflight (#179)
The OPTIONS preflight hardcoded Access-Control-Allow-Headers to just
Content-Type. Any browser MCP client sending mcp-protocol-version (per
the MCP spec) or Authorization / X-API-Key (typical for hosted
deployments) failed the preflight and the browser blocked the actual
request.
Add those three to the allow-list and a 24h Max-Age so browsers don't
re-preflight every call. Adds an integration test that drives the
server with a real preflight request.
Closes #175.
* fix(js,dom): scope element queries, real timers, form/prepend/equal (#180)
- Element.querySelector / querySelectorAll scope to the receiver subtree
via a new query_selector_scoped Rust op, not always document root.
Same for DocumentFragment.
- setTimeout respects its delay argument via a new async op_sleep.
setInterval is a real repeating timer instead of an alias to setTimeout.
- document.forms / images / links return live querySelectorAll results
instead of hardcoded [].
- HTMLFormElement subclasses Element with .elements / .length, and form
nodes are wrapped as HTMLFormElement so instanceof works. submit()
stays on Element.prototype to preserve the existing submit-event
dispatch and form-data navigation path.
- Element.prepend has a real implementation.
- Node.isEqualNode does a structural compare (nodeType, nodeName,
nodeValue, attributes, recursive children) instead of nid identity.
- Fix long-standing insert_before arg-order bug: bootstrap.js was
passing (parent, new, ref) but _dom only forwards two args, so ref
was silently dropped. Pass (new, ref) directly. Same fix in
replaceChild.
Adds 3 selector unit tests and 6 runtime regression tests covering
the JS shim fixes.
Closes #105.
* fix(cdp): unblock puppeteer connect, bound Runtime.evaluate, full TargetInfo (#181)
Three independent gaps that hurt browser-based MCP clients connecting
to obscura's CDP endpoint:
- Runtime.enable returned "No page" when called before a session was
attached. puppeteer-extra's FrameManager.initialize fires it on the
browser-level WebSocket BEFORE any page target exists, so the
standard puppeteer.connect / browser.newPage flow died on connect.
Now reply with {} when no session is attached and let the next
Target.attachToTarget emit executionContextCreated.
- Runtime.evaluate had no per-call timeout. A long synchronous JS
expression held the global V8 lock indefinitely and starved every
other CDP command on the session. Honor the CDP `timeout` parameter
and default to 30s (Chrome's protocolTimeout). Returns a clear error
on expiry so client-side deadlines fire cleanly instead of hanging.
- Target.getTargetInfo for the browser target omitted canAccessOpener.
Strict CDP clients (chromiumoxide) panic when the field is missing.
Fill it in (false for the browser target itself).
Partial fix for #122. The remaining items (networkIdle hard-deadline
on SPAs, /json/list hardcoded data, docs on cheap-vs-queued methods)
need bigger or non-code changes and are deferred.
* fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes #113.
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity (#170)
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity
Closes #168.
Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.
Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(cdp): fall back to default cookie jar, add Network.getAllCookies
Addresses @SGavrl review on PR #170:
- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
cookie jar via a shared helper that prefers the session page's jar and
falls back to ctx.default_context.cookie_jar when no target has been
attached. Mirrors how Storage.setCookies already routes when there is
no session, so Puppeteer/Playwright flows that issue cookie methods
before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
existing getCookies handler so both arms return the full jar.
Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).
* fix(net): use bundled webpki roots in stealth client (#184) (#185)
set_default_paths only resolves OpenSSL's compile-time CA dirs, which
exist on Linux but are empty on Windows. The stealth client's cert store
ended up empty there and every TLS handshake failed with
CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc.
CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's
trust list), which behaves the same on every platform and removes the
dependency on a system CA store.
* fix(js): Element.attributes, on* event defaults, per-nid getBoundingClientRect (#45) (#186)
Three DOM-completeness gaps that broke jQuery 1.11.3 init and Playwright
actionability:
1. Element.attributes was undefined, so legacy IE feature-detect code
like div.attributes[eventName].expando threw TypeError. Added a
NamedNodeMap-like backed by a new attribute_names DOM op (length,
numeric index, string name lookup, getNamedItem/setNamedItem,
iterable).
2. Standard on* event handler properties were missing from window, so
('on' + ev) in window returned false and forced jQuery into the
legacy IE branch. Initialised the 79 HTML-spec on* handlers to null.
3. getBoundingClientRect returned the same {x:8,y:8,width:100,height:20}
literal for every element, so Playwright's hit-testing couldn't tell
elements apart. Replaced with a deterministic synthesis from _nid
mapped into a 12-column grid sized to a 1280x720 viewport. Still not
pixel-accurate (no layout engine), but stable across reads and
distinct per node.
* feat: cookie jar persistence + --storage-dir flag + fix HTTP cookie header filtering (#114)
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
* fix(cdp): page.goto returns Response over Puppeteer and Playwright (#189) (#190)
Two correlation invariants real Chrome holds were missing, so the
nav Response was never attached to goto().
requestId === loaderId for the main Document event. Both clients
gate isNavigationRequest() on this; we were emitting a synthetic
per-page id.
Network.requestWillBeSent for the Document fires before
Page.frameNavigated. Playwright's FrameManager needs the request
present as pendingDocument before the frame commits, otherwise
currentDocument lands with request=undefined.
Subresource events and the Document's responseReceived /
loadingFinished stay where they were.
Verified end-to-end on both clients against example.com (200),
google.com (200), httpbin/404 (404).
* fix(browser,cdp): run preloads on about:blank, tighten binding shim arity
Two follow-ups from PR #166 review:
- navigate_single's about: branch returned before any preload sources
ran, so addScriptToEvaluateOnNewDocument scripts and the
Runtime.addBinding shim never installed on about:blank. Puppeteer
hits this every session: browser.newPage() lands on about:blank and
exposeFunction is dead until the first real navigation. Now the
about: branch runs the preload set after init_js(), guarded by the
same 5s timeout as regular preloads.
- The Runtime.addBinding shim used JSON.stringify on non-string args
and silently accepted any arity. Chromium's V8InspectorImpl rejects
calls without exactly one argument and ToString-coerces that arg
before emitting Runtime.bindingCalled. Match that: guard on
arguments.length and use String(arg) for the non-string branch.
Single-string calls (the path puppeteer's exposeFunction wrapper
uses) round-trip unchanged.
* fix(cdp): allocate fresh execution context ids per isolated world (#192) (#200)
Page.createIsolatedWorld and do_navigate both hardcoded context id 100
for every isolated world emission. After the second navigation Playwright
expected a fresh incrementing id (101, 102) while obscura kept re-using
100. Runtime.evaluate then failed with 'Cannot find context with
specified id: 101' because the id was never registered in
valid_context_ids and the client's bookkeeping had diverged from the
server's.
Added next_isolated_context_id counter on CdpContext (starts at 100) and
a next_isolated_context() helper that claims an id, registers it in
valid_context_ids, and increments. Both call sites now use it.
* feat(cli): default V8 heap to 4 GB on 64-bit so heavy SPAs don't OOM (#199) (#202)
V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with
heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/
web-scraping) blow past it and the worker dies with
'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session
with it.
Always set --max-old-space-size=4096 (1024 on 32-bit targets) before
V8 startup, matching headless Chrome's default. User-supplied
--v8-flags are appended after the default so they win (V8 parses
left-to-right and the last value wins), which keeps the existing
escape hatch intact for memory-constrained environments.
---------
Co-authored-by: SGavrl <[email protected]>
Co-authored-by: Ben Younes <[email protected]>
Co-authored-by: zhangyang-crazy-one <[email protected]>
|
||
|
SGavrl
|
e7bb0c676d |
feat(cli): --allow-private-network for localhost development (#33) (#203)
Picks up @Timtech4u's PR #33, which has been waiting since April. The design (CLI flag + threading through BrowserContext + http client) is theirs and is correct; this is a clean rewrite on top of current main because repeated rebase rounds had been reverting unrelated work that landed in the meantime. Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is process-wide and clunky in pipelines. Multiple users (@Timtech4u in the original PR, then @duartemvix and @necrogami in the comment thread) reported they cannot use obscura for development against http://localhost:N without it. Adds --allow-private-network as a global CLI flag, works on fetch, serve, and scrape. Threads through BrowserContext::with_storage_and_network and the http client's validate_url. CdpContext gets a new_full builder and obscura_cdp::start_with_full_serve_options entry point so 'obscura serve --allow-private-network' lets Puppeteer/Playwright navigate to localhost too. Default remains blocked. Backward compat: the env var still works (OR'd with the flag inside validate_url). All existing constructors and entry points unchanged. Verified end-to-end against a local http.server on 127.0.0.1: fetch and CDP Page.navigate both blocked by default, both succeed with the flag. Public URLs unaffected. Integration suite 28/28, full workspace cargo test all green. |
||
|
|
906dcfcde1 |
feat: cookie jar persistence + --storage-dir flag + fix HTTP cookie header filtering (#114)
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
|
||
|
Ben Younes
|
10810e250c |
fix(cdp): keep HTTP control plane reachable while V8 JS evaluation blocks (#173)
fix(cdp): unblock HTTP control plane during synchronous V8 execution Fixes #62. Previously, the TCP accept loop and CDP processor shared a single-threaded tokio LocalSet. Long synchronous JS evaluations starved the accept loop, causing incoming connections to hang in the kernel backlog and effectively killing the HTTP control plane. This moves the TCP listener and `/json/*` endpoints to a dedicated OS thread using `std::net::TcpListener`. HTTP requests are now handled directly via blocking I/O on this thread, completely bypassing V8. WebSocket connections are handed off to the LocalSet via a bounded mpsc channel (capacity 128). If V8 stalls and the queue fills, new WS connections are dropped (ECONNRESET) rather than buffering unboundedly to prevent OOMs. Includes a new integration test to verify the control plane stays responsive during a 5s JS loop, plus inline documentation clarifying thread lifetime semantics and a minor test TOCTOU race. |
||
|
SGavrl
|
18dc796df0 |
fix(cdp): gate file:// navigation behind --allow-file-access
GHSA-q55h-vfv9-qcr5. Any process able to reach the CDP port could
send Page.navigate {"url":"file:///etc/shadow"} and read arbitrary
local files (the process's UID's worth of read access). When obscura
runs as root inside Docker or as a systemd service that translates
straight into /etc/shadow, /root/.ssh/id_rsa, /proc/1/environ, etc.
Validate at the CDP boundary, not in obscura-net, so direct CLI
callers ("obscura fetch file://...") keep working. A new
BrowserContext.allow_file_access flag defaults to false; Page.navigate
returns an error when the requested URL is file:// and the flag is
off. CdpContext::new_with_security threads the flag from the server
entry point, and "obscura serve --allow-file-access" opts in for
local-HTML testing workflows.
Tested:
- Default obscura serve: CDP Page.navigate {file:///etc/passwd} ->
refused with descriptive error.
- obscura serve --allow-file-access: same call succeeds.
- obscura fetch file://... CLI path: unchanged, still reads the file.
- All earlier PoCs (GHSA-6pj9 JS->file SOP, GHSA-4789 sub-resource
SSRF, GHSA-4789 CSS template injection) still pass.
- cargo test --workspace clean.
|
||
|
SGavrl
|
a23cdc0ae7 |
feat: add Dockerfile + --host flag for containerized deployment
Adds a multi-stage Dockerfile (rust:1-slim-bookworm builder → distroless/cc runtime) plus a --host flag for obscura serve that defaults to 127.0.0.1 for safety but can be set to 0.0.0.0 so the container's port is reachable via docker run -p mapping. Dockerfile cherry-picked from #84. The SPA chunk loading commit from that PR was left out — verified it didn't actually fix the Next.js webpack errors on kiwi.com (#86). |
||
|
Timothy Olaleke
|
85739d3f6a |
fix(#19): serialize V8 work across pages with process-wide async lock (#36)
* fix(#19): serialize V8 work across pages with process-wide async lock Concurrent CDP clients (≥5) trip V8's `heap->isolate() == Isolate::TryGetCurrent()` invariant and abort the process via V8_Fatal — no Rust panic, just abort(3). Each Page owns its own JsRuntime/Isolate and they all run on a single OS thread via LocalSet+spawn_local; when two pages' V8-touching futures interleave across an .await, V8 trips the check. This adds a process-wide tokio::sync::Mutex (obscura_js::v8_lock) that is acquired: - At the top of obscura_cdp::dispatch::dispatch (covers every CDP handler that may call into a JsRuntime). - In server.rs's process_with_interception spawn_local that runs page.navigate_with_wait + execute_preload_script (this path bypasses dispatch). The guard keeps each V8-touching block contiguous on the thread: V8 fully exits one Isolate before another page is allowed in. Trade-off: non-V8 handlers (Browser.*, Storage.*, Emulation.*) also serialize, but those are cheap and the safety win dominates. The properly concurrent fix is to pin each JsRuntime to its own OS thread and message-pass; tracked as the larger #19 follow-up. Adds an opt-in smoke test (`--ignored`) that boots the CDP server and runs 5 parallel clients through createTarget+navigate. Note: the deterministic abort needs subresource-fetch yields (page.rs:285 join_all) which data: URLs skip — full repro requires JS-heavy real traffic (issue author used scrapegraphai.com). * fix(#19): suspend other Isolates and defer foreign CDP during interception nav Address maintainer feedback on PR #36 (comment 4328238087): the V8 fatal still reproduces with `Fetch.enable patterns:["*"]` + 5 concurrent `Page.navigate`s. The `obscura_js::v8_lock` mutex was insufficient because V8's actual invariant is "only one Isolate may be entered per OS thread", and the mutex can't enforce that across `.await` points inside V8 ops. Two distinct paths violated the invariant: 1. Foreign CDP messages dispatched while a nav task was running. `process_with_interception`'s `select!` routed every non-Fetch message through `process_cdp_message → dispatch → page::handle::*`. `get_session_page_mut` then `suspend_js`'es other pages (drops their `JsRuntime` → `Isolate::Exit`) while the spawned nav task is parked at `op_fetch_url`'s `resolve_rx.await` with its Isolate still entered. Drop trips `heap->isolate() == Isolate::TryGetCurrent()`. 2. Subsequent navs through `process_with_interception` did not suspend prior pages. After nav-1 completes (page-1's JsRuntime stays alive in `ctx.pages`), nav-2's `init_js` constructs Isolate-2 while Isolate-1 is still current. The next V8 scope unwind aborts via `Context::Exit`. Fix (3 changes in `crates/obscura-cdp/src/server.rs`): - `cdp_processor`: add a `VecDeque<ServerMessage> deferred` drained before each new `rx.recv()`, so deferred messages are processed with no nav task in flight. - `process_with_interception` `select!`: only handle `NewConnection` and `Fetch.{continue,fulfill,fail}Request` inline (safe — no V8 enter on this side); push everything else into `deferred`. - `process_with_interception` entry: explicitly `suspend_js()` every other page in `ctx.pages` before spawning the nav task, mirroring the invariant the regular dispatch path already enforces via `get_session_page_mut`. Verification: - New test `crates/obscura-cdp/tests/concurrent_navigations_with_fetch.rs` is the maintainer's exact repro (5 clients, `Fetch.enable patterns:["*"]`, concurrent `Page.navigate` to https://example.com, auto-reply `Fetch.continueRequest`). Aborts on PR #36 head, passes after fix (5/5 stable runs, ~1.2s each). - Existing data-URL smoke test (`concurrent_navigations`) still passes. - Full workspace test suite (90 tests) still passes. Run with: cargo test -p obscura-cdp --test concurrent_navigations_with_fetch \ -- --nocapture --ignored Honest caveat. This is still the lock/serialize approach — it converts the abort into latency. The properly concurrent fix (Option 2 in issue #19: pin each `JsRuntime` to its own OS thread and message-pass) remains the right long-term direction. * fix(cdp): bound deferred message queue to prevent OOM during stalled navigation PR #36 comment 4341743194: process_with_interception deferred foreign CDP messages into an unbounded VecDeque while a navigation task was in flight. A stalled navigation (e.g. waiting for Fetch interception resolution) could accumulate arbitrary queued messages and OOM the process. Add MAX_DEFERRED_MESSAGES = 256. When the cap is reached, return an explicit CDP error (-32000) to the client instead of silently dropping or growing without bound. All 90 workspace tests pass. |
||
|
404HumanandObscura Contributor
|
0de133eb2c |
fix(cdp): fast-path Target.getBrowserContexts for Puppeteer connect (#120)
Puppeteer's first CDP call after connect is Target.getBrowserContexts. When the single cdp_processor is busy (e.g. Page.navigate with fetch interception), queued Target commands stall and Puppeteer hits protocolTimeout on getBrowserContexts before any user code runs. Answer getBrowserContexts synchronously in fast_path_response with the same payload as domains::target::handle (default context id "default"), matching Browser context initialization in CdpContext. Co-authored-by: Obscura Contributor <[email protected]> |
||
|
Adam Ma
|
a8b1ec0968 |
chore: silence unused import/variable/dead-code warnings (#101) (#126)
- obscura-net/cookies.rs: #[allow(dead_code)] on CookieEntry.same_site (parsed today, kept for future SameSite enforcement). - obscura-js/runtime.rs: drop vestigial BOOTSTRAP_JS const; bootstrap.js is already embedded into the V8 snapshot via build.rs. - obscura-js/ops.rs: prefix the four unused fields in InterceptResolution::Continue with _. - obscura-cdp/dispatch.rs: drop unused InterceptResolution import. - obscura-cdp/domains/fetch.rs: drop unused Arc / Mutex / CdpEvent imports. - obscura-cdp/domains/input.rs: prefix _button / _click_count; CDP parses them but the JS click stub is hardcoded for now. - obscura-cdp/server.rs: prefix _processor_handle and _ctx; drop dead initial Ok(())/None for navigate_result/page_back - they are unconditionally overwritten in the only loop break arm. cargo build --workspace: 0 warnings. cargo test --workspace: 113 passed. |
||
|
Ben YounesandOra Studio
|
536072b323 |
fix(serve): plumb --user-agent through CDP server (#71) (#76)
obscura serve --user-agent <ua> only logged the value via tracing::info! and never passed it to the CDP server, so every navigation kept using the default Chrome UA string and httpbin.org/user-agent reported the default rather than the override. Plumb the option end-to-end: - BrowserContext::with_full_options(id, proxy, stealth, user_agent) initializes both BrowserContext::user_agent and the underlying ObscuraHttpClient::user_agent RwLock at construction time so the first navigation already sees the override. - CdpContext::new_with_full_options forwards the UA to the default BrowserContext. - obscura_cdp::start_with_full_options accepts the UA and passes it through cdp_processor. - main.rs Serve arm calls start_with_full_options with the parsed UA. - run_multi_worker_serve forwards --user-agent <ua> when spawning worker subprocesses so multi-worker mode honors the flag too. Existing two-arg start_with_options and three-arg with_options stay as thin shims (call the full variant with None) so older callers and the public crate API remain source-compatible. Three new unit tests in obscura-browser cover the propagation, the Chrome default fallback, and the legacy with_options path. Generated by Ora Studio Vibe coded by ousamabenyounes Co-authored-by: Ora Studio <[email protected]> |
||
|
h4ckf0r0day
|
99e75f1e62 |
fix: load balancer panics, stealth flag, shadow DOM polyfill
Fixes #1: replace unwrap() panics in run_multi_worker_serve with match on the Result. Fixes #2: route /json round-robin via worker_port instead of hardcoded port+1. Fixes #3: return 502 Bad Gateway when a worker is unreachable instead of silently dropping the connection. Fixes #5: thread --stealth through start_with_options -> CdpContext -> BrowserContext::with_options so the flag actually has an effect in serve mode (was let _ before). Log message now reflects the actual capability based on the stealth cargo feature being compiled in. Fixes #15: try/catch the parentNode assignment in the attachShadow polyfill so it doesn't throw a TypeError against the read-only Node.parentNode getter (Cloudflare Turnstile triggers this path). |
||
|
Mike Mooring
|
9e3e1e4ef7 |
fix(cdp): five Playwright drop-in compatibility patches
Brings Obscura's CDP server in line with what Playwright's `chromium.connectOverCDP()` expects. Each fix is small and surgical; together they make `connect → newContext → newPage → goto → page.title()` work end-to-end against `https://example.com`. 1. **server.rs — emit pending events before command response.** Chromium CDP semantics require side-effect events (e.g. `Target.targetCreated` + `Target.attachedToTarget` from `Target.createTarget`) to arrive *before* the originating command's response. Playwright's wire reader assumes this ordering — it resolves the response promise on the response and then synchronously looks up the freshly-created session. Reversing the order makes that lookup miss and the page hang at `newPage()`. 2. **domains/browser.rs — accept Browser.setWindowBounds as a no-op.** Playwright calls `Browser.setWindowBounds` during teardown. Returning "Unknown Browser method" causes Playwright to treat the target as broken and tear it down mid-test, surfacing as "Frame has been detached" on the next `page.title()`. 3. **obscura-browser/page.rs — frame_id == target_id for main frame.** Was `format!("{id}.1", id)`. Playwright's frame manager indexes frames by frame id and looks up the main frame by target id, so the `.1` suffix made every main-frame lookup miss. 4. **domains/page.rs + dispatch.rs — track isolated worlds and re-emit them post-navigation.** Playwright registers a utility world via `Page.createIsolatedWorld` and expects `Runtime.executionContextCreated` for each registered world after every navigation. Without re-emission, the utility-world handle goes stale post-`goto` and `page.title()` hangs waiting for the world's execution context. Falls back to the legacy `__puppeteer_utility_world__24.40.0` name when no worlds have been registered, preserving prior Puppeteer support. 5. **obscura-js/runtime.rs — strip trailing semicolons before wrapping.** Playwright's utility-script expression is an IIFE ending in `})();`. Both `wrap_expression` (CDP `Runtime.evaluate`) and `evaluate_for_cdp` (the `meta_code` IIFE) embed the user expression inside `({EXPR})` / `return ({EXPR});`. With the trailing `;` left in, this produces `({...;})` — a SyntaxError. Parse errors are not catchable, so the surrounding try/catch never fires and the call silently returns `undefined`, surfacing as `page.title() === ""`. Verified end-to-end against `https://example.com`: [probe] connect OK [probe] newContext OK [probe] newPage OK [probe] goto OK [probe] title = Example Domain |
||
|
h4ckf0r0day
|
8749725b45 | Initial Release |