6c7b2b0cecc075561d805acfa018c6b1fb5ed943
79
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 sycatle
|
6c7b2b0cec |
test(cdp): add heavy-page concurrent-connection regression for #430
concurrent_navigations uses data: URLs, which settle instantly and so never drove the abort. This test serves a local page with slow subresources plus a setInterval so the navigation settle loop keeps pumping, then drives it from four independent CDP connections at once. It aborts the process on the old single-LocalSet server (SIGTRAP, the V8 heap->isolate() == TryGetCurrent() fatal) and passes on the thread-per-connection server. Refs #430. |
||
|
sycatle
|
76fc3b9cae |
fix(cdp): run each connection on its own thread to confine V8 isolates
The CDP server ran every connection's pages on one LocalSet thread, so two pages' V8 isolates shared an OS thread. Under concurrent navigation a cancelled event-loop future could leave one isolate entered while another page's execute_script ran, tripping V8's heap->isolate() == Isolate::TryGetCurrent() check and aborting the process (#430). No single-thread serialization fixes this reliably: the cancellation-leak pattern is pervasive and partly inside deno_core. Give each WebSocket connection its own OS thread with its own current_thread runtime, LocalSet, and CdpContext, so every page's isolate is confined to one thread. V8's invariant is per-thread, so two connections' isolates can never collide. This is the "Option 2" fix named in obscura-js/src/v8_lock.rs, and it also lets connections run in parallel instead of serializing on one thread. - CdpContext::new_with_shared_context reuses one Arc<BrowserContext> across all connections, so they share one cookie jar and one HTTP client. - A single signal watcher flips the accept flag and wakes every connection's processor via notify_waiters(); the accept side persists cookies once on shutdown. - Serialize isolate construction with a process-wide lock: the leaptiering JSDispatchTable setup is not safe to run from several threads at once. - Warm up V8 once on the main thread before any connection thread creates an isolate; creating the first isolate off the main thread segfaults in InitializeBuiltinJSDispatchTable. obstacle course 33/33, obscura-cdp 75/75 including the concurrent-navigation and control-plane tests. Concurrency 2 and 4 against heavy pages no longer abort. Fixes #430. |
||
|
SGavrl
|
c12915a1a6 | fix(js): keep TreeWalker traversal in document order (#432) | ||
|
Andrey Mnatsakanov
|
c3ae054dde |
fix(js): give NodeFilter the full standard constant set (#439) (#440)
NodeFilter was defined twice: a partial live definition (SHOW_ELEMENT,
SHOW_TEXT, SHOW_ALL) and a complete one behind an `if (typeof NodeFilter
=== 'undefined')` guard that never fired because the partial one already
defined it. So NodeFilter.FILTER_ACCEPT/REJECT/SKIP and most SHOW_* values
were undefined at runtime, and the canonical filter idiom
`acceptNode() { return NodeFilter.FILTER_ACCEPT; }` returned undefined —
making TreeWalker/NodeIterator reject every node.
Make the live definition authoritative with the full standard constant set
and drop the dead duplicate. Adds tests asserting the constant values and
that a filter returning NodeFilter.FILTER_ACCEPT accepts the node.
|
||
|
Andrey Mnatsakanov
|
57949a0efc |
fix(cdp): backslash-escape key/code in Input.dispatchKeyEvent (#433) (#437)
The keyDown/keyUp paths interpolated `key`/`code` into the generated KeyboardEvent snippet with single-quote escaping only. Chrome sends `key: "\"` (U+005C) for the backslash key, which became `key:'\'` — the backslash escaped the closing quote, ran the string literal on, and made the whole page.evaluate a syntax error, so the keydown was silently dropped. Any key/code carrying a backslash broke dispatch. Escape backslash before single-quote (backslash first, so the quote-escape backslashes are not themselves doubled), matching the sibling text path in the same function. Adds a regression test asserting the backslash key arrives at a keydown listener alongside an ordinary key. |
||
|
Andrey MnatsakanovandSGavrl
|
b2e4bb49a7 |
fix(js): structuredClone cycles through Error.cause and clones own __proto__ (#419, #420) (#421)
* fix(js): structuredClone cycles through Error.cause and clones own __proto__ (#419, #420) The structuredClone polyfill had two correctness/parity gaps in _structuredClone: - The Error branch recursed into `cause` before recording the clone in `seen`, so a cycle through the error (e.cause === e) recursed until the stack overflowed. Chrome clones such graphs and preserves identity. Fix: seen.set the clone before recursing into cause. (#419) - The generic-object branch built the clone on the source's prototype and copied properties by assignment, so an own enumerable `__proto__` data property (exactly what JSON.parse('{"__proto__":…}') produces) went through the inherited __proto__ setter and reparented the clone instead of copying the property. Fix: clone plain objects onto Object.prototype (like Chrome) and define each property instead of assigning it. (#420) Adds two parity tests to structured_clone_crypto_parity.rs; full obscura-cdp suite (73) green. * perf(js): keep structuredClone fast path for all keys but __proto__ The own-__proto__ fix copied every own property with Object.defineProperty to avoid triggering the inherited prototype setter. That is ~3x slower than plain assignment per key, so cloning a large object regressed (~75ms -> ~228ms for a 1000-key object x200). Only the __proto__ key needs defineProperty; every other key is safe to assign. Special-case it so the fast path applies to all other keys. Measured back to parity with main (85ms vs 85ms); the own-__proto__ correctness test still passes. --------- Co-authored-by: SGavrl <[email protected]> |
||
|
Andrey Mnatsakanov
|
ccfa5fb87b |
fix(js): validate the submitter argument to requestSubmit() (#424) (#426)
HTMLFormElement.requestSubmit(submitter) accepted any element and submitted silently. Per the HTML spec (and Chrome) it must throw a TypeError when the submitter is not a submit button, and a NotFoundError DOMException when the submitter is not owned by the form; both checks run before the submit event. Adds an _isSubmitButton predicate (a <button> whose type is submit — the default, including a missing/invalid type — or an <input> of type submit/image) and validates the argument at the top of requestSubmit. submit() is untouched: it correctly bypasses the event (#395). Element.click() now gates on the same predicate rather than its own looser check, so an internal click can never hand requestSubmit a submitter it would reject (e.g. <div type="submit">, which is not a submit button and no longer submits — matching Chrome). That also aligns click() with the CDP click path in input.rs, which already treats <input type=image> as a submit button. The input.rs call sites are already guarded by equivalent conditions and pass no submitter on the Enter-key path, so they are unaffected. Constraint validation is deliberately out of scope: obscura has no validity engine (checkValidity is a stub returning true), so wiring it here would be a no-op. Noted in #424 as separate work. Full obscura-cdp suite (72) green. |
||
|
Andrey Mnatsakanov
|
8698afc065 |
fix(js): CryptoKey clone hook honours the structuredClone seen map (#423) (#425)
_structuredClone dispatches platform-object hooks as hook(value, seen), but the CryptoKey hook took only `src` and never touched the memo, and hook branches return before the generic seen.set. So one CryptoKey referenced twice in a graph cloned into two distinct objects (and re-registered its key material each time), where structuredClone preserves reference identity within a graph. Consult and populate `seen` in the hook. Adds a parity test asserting a key reached twice clones to one shared object that crypto.subtle still accepts. |
||
|
SGavrl
|
d0b6fc4ffd |
fix(cdp): emit Network events for script-initiated requests (#415)
Requests made by page JS (fetch/XHR/dynamic script and stylesheet loads)
run in the V8 op layer and never produced a NetworkEvent, so CDP emitted
Network.requestWillBeSent / responseReceived only for the static
navigation subresources. Puppeteer/Playwright page.on('request') and
on('response') therefore missed every XHR/JSON response, which breaks
response capture (and is the root cause of the Aviasales half of #394).
op_fetch_url now records a JsNetworkEvent keyed by the same fetch-{N} id
under which it already stores the response body. The Page drains these
into its network_events before the CDP layer emits, so they flow through
emit_navigation_events as requestWillBeSent / responseReceived /
loadingFinished alongside the document subresources, and
Network.getResponseBody resolves for them via the shared id. Applied to
both navigation paths (server process_navigation and in-process
do_navigate). The queue is capped at 4096 events per page.
Closes #406.
|
||
|
Jim Zhang @ wonderhoi
|
b460b37e93 |
fix(js): expose Document.adoptNode and Element.toggleAttribute (#392)
adoptNode was only stubbed on the createHTMLDocument shim object, never on Document.prototype, so document.adoptNode(node) threw. importNode already had a Document.prototype polyfill returning the node, so adoptNode mirrors it: with no second document to transfer ownership from the node is already ours, return it. toggleAttribute was missing entirely. Implement spec semantics: no force arg toggles, force true adds, force false removes, returns the new presence. Lit, Stencil, and several ad SDKs call it and die on the missing method today. Fixes #393. |
||
|
SGavrl
|
358100aa63 |
fix(cdp): prune valid_context_ids on executionContextsCleared (#413)
Runtime.executionContextsCleared on navigation told clients every prior context was gone, but valid_context_ids was insert-only, so a Runtime.callFunctionOn targeting a pre-navigation context id still ran instead of being rejected with Chrome's 'Cannot find context with specified id'. On a long-lived CDP connection the set also grew by one id per navigation and was never pruned. Clear valid_context_ids when executionContextsCleared is emitted, then re-register the default world (id 2) and the isolated worlds as their executionContextCreated events are emitted. Closes #407. |
||
|
Andrey MnatsakanovandSGavrl
|
5ada3b404b |
fix(net): default cookie Path to the request directory (RFC 6265 5.1.4) (#403)
* fix(net): default cookie Path to the request directory (RFC 6265 5.1.4) When a Set-Cookie carried no Path attribute, the jar stored the full request path as the cookie's path. RFC 6265 5.1.4 default-path is the request URI's directory — the path up to but not including the right-most '/'. Storing the full path scoped a session cookie to the exact URL that set it: a cookie set on GET /app/login was stored with path=/app/login and then, because path-match (5.1.4, fixed earlier) requires a '/' boundary, was not sent to /app/dashboard. The user appeared logged out on the very next navigation. Add default_cookie_path() implementing the RFC algorithm and use it as the initial path in both set_cookie (HTTP) and set_cookie_from_js (document.cookie). An explicit Path attribute still overrides it. Tests cover the algorithm and the end-to-end 'cookie set on /app/login is sent to /app/dashboard' path for both the HTTP and JS entry points; both failed before this change. * fix(cdp): apply RFC 6265 default cookie Path to CDP-injected cookies Network.setCookie/setCookies (the Puppeteer page.setCookie and Playwright addCookies surface) derived the cookie Path from the full request URL when the caller omitted path, the same bug the parent commit fixed for Set-Cookie responses. Reuse default_cookie_path (now pub) so a cookie set under /v1/things scopes to /v1 and is resent on sibling paths. Fixes the CDP half of #401. Correct the test that locked in the old full-path behavior and add a case confirming an explicit Path still wins. --------- Co-authored-by: SGavrl <[email protected]> |
||
|
SGavrl
|
c8c78ba229 |
fix(js): fire load/error for dynamically inserted stylesheet links (#412)
A <link rel=stylesheet href> appended via JS (createElement + appendChild) was never fetched and fired neither load nor error, so any Promise.all awaiting the link's onload (lazy CSS loaders, antd/bootstrap fetchers) hung forever and the page stalled at its pre-load stage. appendChild already drives this for <script src>; mirror it for links: resolve the href against <base>/document URL, fetch it, and dispatch load on success or error on failure. CSS is not applied (no layout engine yet) but the load-event contract matches Chrome. Read the stylesheet intent from both the rel attribute and the rel property, since obscura does not yet reflect the rel IDL attribute back to the content attribute. Closes #409. |
||
|
Jim Zhang @ wonderhoiandSGavrl
|
a92166897f |
fix(js): real structuredClone so CryptoKey and buffers survive cloning (#390)
* fix(js): real structuredClone so CryptoKey and buffers survive cloning
structuredClone was JSON.parse(JSON.stringify), which silently drops
ArrayBuffer and TypedArray to {}. A cloned CryptoKey also lost its WeakMap
entry, so crypto.subtle.sign/encrypt/deriveBits threw "Argument is not a
valid CryptoKey" on the clone.
Replace the JSON fallback with a recursive cloner covering ArrayBuffer,
TypedArray, DataView, Map, Set, Date, RegExp, Error, and plain objects,
with cycle and own symbol-keyed property handling. CryptoKey opts in via a
clone hook (keyed by Symbol.toStringTag) that rebuilds the key through
makeKey, so the clone re-enters the WeakMap and stays usable.
Fixes #389.
* fix(js): structuredClone handles DataView and rejects functions and symbols
The TypedArray branch called value.slice(), which DataView lacks, so
cloning any DataView threw TypeError, a new crash in the buffers
category this feature targets. Copy the underlying buffer slice over
the view range and wrap a fresh DataView.
The function/symbol rejection lived after the primitive early-return
and was dead code, so functions and symbols were passed through by
reference instead of throwing DataCloneError (HTML structured clone).
Check first and throw.
Register ArrayBuffer, TypedArray, and DataView in the seen map so
repeated references share one clone, and move the cycle check up.
---------
Co-authored-by: SGavrl <[email protected]>
|
||
|
sickoandSGavrl
|
7e2cabf411 |
fix(js): HTMLFormElement.submit() must not fire a cancelable submit event; add requestSubmit() (#395)
* fix(js): HTMLFormElement.submit() must not fire a cancelable submit event; add requestSubmit() Per the HTML standard, the submit() method submits without firing a submit event and cannot be canceled by a page submit listener; only requestSubmit() and user-initiated submits fire the cancelable event. bootstrap.js dispatched a cancelable submit event from submit() and aborted navigation when canceled, which breaks the common pattern where a submit listener preventDefault()s the native submit and a callback later calls form.submit() to actually send it (e.g. an invisible-reCAPTCHA data-callback): the listener also canceled the programmatic submit, so no navigation was ever issued. - submit(): navigate unconditionally, no submit event - requestSubmit(): fire the cancelable submit event, navigate only if not canceled - submit-button clicks route through requestSubmit() (a real click fires the event) Adds a regression test; full obscura-cdp suite (56 tests) passes. * fix(cdp): route synthetic submit-button clicks and Enter through requestSubmit A CDP Input.dispatchMouseEvent on a submit button and an Enter key in a form field are user-initiated submits, so per the HTML spec they must fire the cancelable submit event. input.rs called form.submit() (the method) directly, bypassing any preventDefault() submit listener, which mirrors the bootstrap.js bug fixed in the parent commit. Use requestSubmit() with a submit() fallback so the listener can veto. --------- Co-authored-by: SGavrl <[email protected]> |
||
|
Andrey Mnatsakanov
|
c915a11f59 |
fix(cdp): truncate client-controlled log previews on char boundaries (#397)
server.rs logged previews of attacker-controlled strings with raw byte slicing: `&text[..text.len().min(200)]` for a malformed CDP frame (inside a `warn!`, which is active at the worker's default log level) and the same for an intercepted event JSON. `&s[..n]` panics when byte `n` falls inside a multi-byte UTF-8 character, so a single WebSocket frame that is invalid JSON, >=201 bytes, with a multi-byte char straddling byte 200 panics the CDP processor task — an unauthenticated one-frame crash. page.rs had the same pattern (behind --verbose debug logging) for evaluated JS expressions. Add truncate_on_char_boundary (str::floor_char_boundary is still unstable) and route all four sites through it. Regression test constructs the exact 'a'*199 + '€' input, asserts the old raw slice panics (via catch_unwind) and the helper truncates on the boundary instead. |
||
|
Andrey Mnatsakanov
|
d8e9fc2196 |
fix(cdp): bound DOM.getDocument depth so deep DOMs cannot crash the worker (#396)
DOM.getDocument{depth:-1} (the standard full-tree call from Puppeteer/
Playwright/browser-use) arrived as u32::MAX and drove serialize_node to
recurse once per DOM nesting level. A page can nest nodes tens of thousands
deep (trivially scriptable; html5ever caps no generic nesting), so the
recursion overflowed the stack — a SIGABRT that catch_unwind cannot recover,
crashing the whole worker (AGENTS.md: "one page must never crash a worker").
Deriving the builder alone is not enough: the response is inherently nested
JSON, and serde_json's own serialization and the Value's Drop recurse over
that nesting too, overflowing the ~2 MiB tokio worker stack. So this both
rewrites serialize_node as an explicit heap worklist and clamps the effective
depth to MAX_SERIALIZE_DEPTH (256), mirroring DOMSnapshot's MAX_NODES guard.
Real DOMs are far shallower; clients wanting a deeper subtree re-request it
with DOM.requestChildNodes / describeNode.
Adds a regression test that builds a 50k-deep tree and serializes with
depth:-1; it SIGABRTs before this change and passes after (run on nextest's
2 MiB test thread, mirroring production).
|
||
|
Flo
|
82d20bcb7c | fix(cdp): allow large WebSocket response frames (<64Mb) (#405) | ||
|
Andrey Mnatsakanov
|
50e66320b0 |
fix(cdp): bound IO stream buffers so downloads cannot leak (#387)
* fix(js): fire trusted input/change from setFileInputFiles DOM.setFileInputFiles dispatched plain input/change events, so event.isTrusted was false. Upload flows and anti-bot code that gate their change handler on isTrusted ignored them, which silently broke the exact case the feature targets. The Input domain already marks its synthesized clicks/keys trusted via __obscura_markTrusted; do the same here. Fixes #384. * fix(cdp): bound IO stream buffers so downloads cannot leak Fetch.takeResponseBodyAsStream moved a response body out of the page's LRU-bounded cache into CdpContext.io_streams, a server-lifetime map freed only by an explicit IO.close. A client that opens a stream and disconnects, errors mid-download, or never closes it leaked the full buffer forever; no teardown path (WS close, Browser.close, Target.closeTarget) touched io_streams. That reintroduced the unbounded memory growth streaming was meant to avoid (#360). Replace the raw map + counter with an IoStreamStore that caps the number of open streams and their total bytes (OBSCURA_IO_STREAM_MAX_ENTRIES / _BYTES), evicting the oldest first and always keeping the just-opened stream. Reading an evicted handle fails cleanly, which is the right trade against an OOM. Read chunking is unchanged. Fixes #386. |
||
|
SGavrl
|
1ca6047351 |
feat(cdp): stream large response bodies via takeResponseBodyAsStream + IO (#381)
Network.getResponseBody returns the whole body in one base64 blob and the cache drops anything over 2 MiB, so large downloads (images, PDFs, archives) either failed or forced the long-running server to hold the full body in memory (issue #360). Add Fetch.takeResponseBodyAsStream + IO.read + IO.close. The body is moved out of the page cache into a stream handle and delivered in bounded chunks via IO.read, then freed on IO.close, so it is held once and released after the download. Binary bodies are byte-exact (stored base64). To stream a large body, raise OBSCURA_NETWORK_BODY_BUFFER_BYTES so it is cached. This is entirely opt-in: nothing runs unless a client calls the new methods, so the default fetch/render/scrape path is unchanged (verified: obstacle course 33/33, obscura-vs-Chrome head-to-head within noise of the pre-change baseline). Fixes #360. |
||
|
Andrey Mnatsakanov
|
68234970b1 |
fix(cdp): apply Fetch.continueRequest overrides (#373)
handle_fetch_resolution built InterceptResolution::Continue with url, method,
headers, and body hardcoded to None, so Fetch.continueRequest silently sent the
request unmodified. Playwright route.continue({ headers, url, postData }) and
Puppeteer request.continue reported success while their overrides were dropped.
op_fetch_url already applies all four (and re-validates a rewritten URL through
the SSRF gate); only the CDP layer was discarding them. Read url, method,
headers, and postData from the continueRequest params and pass them through.
Fixes #365.
|
||
|
Andrey Mnatsakanov
|
daa0d62fbd |
fix(cdp): route navigation on exact method, not substring (#372)
The cdp_processor decided whether to take the spawn-and-defer navigation path
with `text.contains("Page.navigate")`, which also matches
`Page.navigateToHistoryEntry`. goBack / goForward (both Puppeteer and
Playwright drive Page.navigateToHistoryEntry) were misrouted into the raw
navigate path, which reads an absent `url` param and fails with
"Invalid URL: relative URL without a base", so history navigation never
reached its real handler. Any frame merely embedding the literal
"Page.navigate" (e.g. a Runtime.evaluate expression) was misrouted the same
way.
Match on the parsed CDP method instead, so only an exact Page.navigate takes
the navigation path.
Fixes #363.
|
||
|
SGavrl
|
6788996b0b |
feat(cdp): implement DOM.setFileInputFiles for file uploads (#369)
Puppeteer's ElementHandle.uploadFile and Playwright's setInputFiles drive an <input type=file> through CDP DOM.setFileInputFiles, which obscura rejected as an unknown method, so any upload flow failed (issue #359). Implement it: resolve the node, read each local file, and hand the bytes (base64) to the JS layer, which builds real File objects (backed by the bytes, so page code can read them via FileReader or upload them via fetch/FormData) and fires input+change like a genuine selection. The file input exposes a FileList via .files, and .value reads C:\fakepath\<name> to match Chrome. Fixes #359. |
||
|
SGavrl
|
08c1f0d721 |
fix(cdp): fire onChange for React/Vue controlled inputs (#353)
Typing over CDP into a React controlled input never fired onChange, so the value never reached component state (issue #324). Two root causes, both needed: 1. Value tracker. React/Preact/Vue install a value tracker by redefining `value` on the element instance to record the last value they wrote. obscura's CDP typing assigned `el.value = ...` directly, which runs that wrapper and keeps the tracker in sync, so React saw no change on the input event. Synthetic value/backspace writes now go through a new __obscura_setFieldValue helper that writes via the element's prototype setter, bypassing the tracker. The tracker stays stale, so the input event reads as a real edit. 2. Event handler feature detection. React decides whether to use the modern `input` event via `("oninput" in document)`. obscura exposed the GlobalEventHandlers on* attributes only on window, so that check failed and React fell back to a legacy change-detection path that ignores input events. The on* attributes are now also present on Document and Element, matching the HTML spec (GlobalEventHandlers is mixed into Window, Document, and HTMLElement). Verified end to end over CDP against the real React 18 UMD build: typing into controlled inputs updates state, the component re-renders, and a submit handler reads the captured values. Closes #324. |
||
|
SGavrl
|
f72234d99e |
fix: harden comment serialization and DOMSnapshot recursion depth
fix: harden comment serialization and DOMSnapshot recursion depth |
||
|
SGavrl
|
e7300c5df7 |
fix(cdp): make DOMSnapshot walk iterative to bound recursion depth
The DOMSnapshot walk recursed one frame per nesting level. The MAX_NODES cap bounds the node count but not the depth, so a deeply nested linear chain (script can build thousands of nested elements) could overflow the stack before the count guard triggered. Rewrite it with an explicit stack so depth lives on the heap. Pre-order traversal and the MAX_NODES cap are preserved. Fixes #341. |
||
|
SGavrl
|
7440b534f3 |
fix(cdp): stub setDownloadBehavior, detachFromTarget, activateTarget
These Chrome CDP methods logged "Unknown method" warnings. obscura runs one page per session and does not download to disk, so there is nothing to do, but clients that call them should get a success ack instead of a warning. Added as no-op stubs. Refs #340. |
||
|
SGavrl
|
630f9894f6 |
fix(cdp): expose main-resource response body and real mime on navigation
Navigating straight to an image, PDF, or other resource could not read the main response body over CDP, and Page.frameNavigated always reported text/html. Two problems: the main resource's body was stored under its internal request id, but the client sees it as the navigation loaderId (Chrome's requestId === loaderId convention), so Network.getResponseBody(loaderId) missed. And binary bodies were stored as UTF-8-lossy text, corrupting the bytes. Alias the stored body to the loaderId, store non-text-like content types (images, PDF, octet-stream) base64, and report the response's real Content-Type in frameNavigated. HTML pages are unchanged. Fixes #340. |
||
|
SGavrl
|
b63125faed |
cdp: flush cookies on SIGTERM in serve (#333)
serve --storage-dir saved cookies only on Ctrl-C (SIGINT). docker stop and kill send SIGTERM, which was not caught, so the save_cookies() call at shutdown never ran and cookies were lost on restart. Watch SIGTERM as well as Ctrl-C on Unix. |
||
|
Gabriel
|
0913be86dc |
cdp: make Backspace and typing honor text selection (#316) (#321)
Backspace always trimmed the last character and typing always appended, ignoring any selection. So "select-all, then Backspace to clear" left all but one character, and typing over a selection accumulated text instead of replacing it. Backspace now deletes the selected range when one is set, and typed text replaces it, matching the browser. selectionStart is null during ordinary typing, so the previous append/trim behavior is preserved when no selection is tracked. Pairs with the triple-click select-all in #320 to make the common "triple-click, Backspace, retype" clear pattern work end to end. Also fixes the same pattern when a page sets the selection via select() or setSelectionRange(). |
||
|
zhangyang-crazy-one
|
5d01b5eb6a |
fix(cdp): implement triple-click select-all for text inputs (#316 follow-up) (#320)
Passing detail:clickCount to MouseEvent is necessary but not sufficient for triple-click to select all text. The select-all-on-triple-click is a browser native behavior that synthetic MouseEvent dispatch doesn't trigger. Add manual selection after click dispatch: when clickCount >= 3 and the target is an INPUT or TEXTAREA, set selectionStart=0 and selectionEnd=value.length. |
||
|
RohitMadhu
|
3bf0da2fae |
Add Network.setBlockedURLs support (#315)
Added Network.setBlockedURLs dispatch handling. Stores blocked URL patterns on each page. Applies blocking to page subresources and JS fetch / XHR requests. Supports wildcard CDP-style patterns like *://*.gstatic.com/*.woff2. |
||
|
zhangyang-crazy-one
|
cd0fbc02c2 |
fix(cdp): pass clickCount to MouseEvent.detail + fix char handler event loop (#316) (#317)
Bug 1: clickCount parameter was read with underscore prefix and silently discarded. MouseEvent constructor never received the detail field, so triple-click (clickCount:3) was always treated as a single click by the DOM — inputs never selected their full text. Fix: rename _click_count to click_count, pass it as detail in both mousedown and click MouseEvent constructors. Bug 2: page.type() intermittently dropped the last character because the char handler's quote-escaping order was wrong (single-quote before backslash), and the V8 event loop wasnt pumped after dispatching the input event. Fix: fix escaping order (backslash first, then single-quote), and add page.settle(50) after the char handler's evaluate to pump the event loop. |
||
|
Gabriel
|
d526a4da7d |
js: make event.isTrusted false for page-constructed events
isTrusted was hardcoded to true for every Event, so new Event('x').isTrusted returned true. Real browsers return false for script-constructed events and true only for user-agent dispatched ones, so this was a trivial bot-detection tell (Closes #303).
Track trusted events in a closure-private WeakSet that page JS cannot read or forge; default isTrusted to false. The CDP input pipeline marks its synthesized mouse/key events trusted via a non-enumerable __obscura_markTrusted helper, so automation clicks stay trusted like real Chrome over CDP while page-built events report false.
|
||
|
SGavrl
|
6264e6bfad |
js: make event.isTrusted false for page-constructed events
isTrusted was hardcoded to true for every Event, so new Event('x').isTrusted
returned true. Real browsers return false for script-constructed events and
true only for user-agent dispatched ones, so this was a trivial bot-detection
tell (issue #303).
Track trusted events in a closure-private WeakSet that page JS cannot read or
forge; default isTrusted to false. The CDP input pipeline marks its synthesized
mouse/key events trusted via a non-enumerable __obscura_markTrusted helper, so
automation clicks stay trusted like real Chrome over CDP while page-built
events report false.
|
||
|
Rohit Madhu
|
570ca1544c | add Network.getResponseBody support | ||
|
SGavrl
|
4a00a6ac2c |
Implement DOM.focus for keyboard input (#283)
CDP clients (browser-use and similar) focus an input via DOM.focus before typing; Input.dispatchKeyEvent then targets document.activeElement. Without a DOM.focus handler the focus call errored, keystrokes landed on nothing, and the field stayed empty. Resolve the node and call its JS focus(), which sets document.activeElement. Completes the type path for DOM-agent frameworks. |
||
|
SGavrl
|
f4cbb78f0b |
Add Target.targetInfoChanged and DOMSnapshot.captureSnapshot for DOM agents (#282)
CDP DOM-agent frameworks (browser-use and similar) could connect but saw a blank page: navigation never refreshed their cached target url/title, and their DOM build aborted with "Unknown domain: DOMSnapshot". - Emit Target.targetInfoChanged after navigation (both CDP Page.navigate and JS-initiated) carrying the post-nav url/title. Strict clients refresh a target only on this event; it also fixes Puppeteer/Playwright page.url() tracking. - Implement DOMSnapshot.captureSnapshot, built from the DOM with synthetic geometry (no paint engine) and plausible computed styles, keyed by the same backendNodeId as DOM.getDocument. Enough for tag/ARIA/accessibility element detection; it runs only when a client requests it, so the fetch path is unaffected. Coordinate clicking still requires a real layout engine. |
||
|
SGavrl
|
1a0644f7bc |
feat(cdp): bound every CDP command with a shared V8 watchdog (#250)
A hung page (a runaway Runtime.evaluate, a synchronous DOM op) holds the process-wide V8 lock and wedges every other session forever. The one-shot CLI has a process-level hard deadline for this; the long-running server cannot force-exit, so terminate just the offending isolate instead. dispatch() arms a watchdog on the session page's isolate after taking the V8 lock and disarms it after the handler; on overrun it terminates the isolate and clears the termination flag so the session keeps working. OBSCURA_CDP_COMMAND_TIMEOUT_MS tunes the bound (default 60s, 0 disables); navigation already self-bounds well under it. The watchdog is a single long-lived shared thread (cdp_watchdog), not a thread per command: the V8 lock serializes commands so at most one is ever armed, and arm/disarm are a mutex plus notify with no measurable per-command cost. Also fixes a lost-wakeup in the per-call watchdog (spawn_watchdog) where disarming a fast call before the watchdog thread started blocked for the full budget. Verified: a hung Runtime.evaluate is terminated at the budget and the session recovers; CDP integration 37/37, obstacle course 33/33; per-command latency unchanged (watchdog on equals off within noise). |
||
|
|
2e731a2ab9 |
feat(api): add obscura-api crate with Browser + Page + Element + CookieStore (#191)
* feat(api): add obscura-api crate with Browser + Page API Adds a Rust library crate exposing Obscura as a programmable browser: - Browser: build with config (proxy, stealth, UA, storage_dir) - Page: goto, url, evaluate, content - BrowserBuilder with fluent API - examples/basic.rs Part of #169 * feat(api): add Element API + CookieStore + wait_for_selector - Element: text(), attribute(), click() via JS evaluation - CookieStore: set/get_all/get_for_url/save_to_file/load_from_file - Page::wait_for_selector with 100ms polling - Page::query_selector for single element lookup - Updated examples/basic.rs with full interaction demo Part of #169 * feat(api): add Browser::cookies() method * refactor(api): move to obscura crate with feature flag - Renamed obscura-api → obscura (per maintainer feedback #169) - API enabled by default via 'api' feature flag - Usage: cargo add obscura (default=api) or cargo add obscura --features api - Removed separate obscura-api crate * chore: exclude docker workflow from fork * fix(api): make the obscura crate build, run, and document install correctly - examples/basic.rs and the lib.rs doctest imported `obscura_api`, but the crate is named `obscura`, so both failed to compile. Use the real crate name. - wait_for_selector / query_selector read the node id with `Value::as_u64`, but `evaluate` returns JS numbers as f64, so it was always None and every wait_for_selector timed out. Accept an integer or a non-negative finite float. - restore .github/workflows/docker.yml, which the PR removed (a fork-specific change that should not land upstream). - add crates/obscura/README.md with honest install instructions: the crate is not on crates.io, so depend on it via git (and it builds the embedded V8 from source). The `cargo add obscura` form does not work yet. Verified: workspace + example build, the lib doctest passes, the example runs end to end (goto, wait_for_selector, text, attribute), and the integration suite is 28/28 with no regressions. --------- Co-authored-by: zhangyang-crazy-one <[email protected]> Co-authored-by: SGavrl <[email protected]> |
||
|
Sander Spilleman
|
9ae36764da | feat(stealth): many stealth improvements | ||
|
SGavrl
|
1b9be08767 |
fix(cdp): exit on Ctrl-C while serving (#225) (#226)
The LocalSet accept loop blocked on ws_rx.recv() forever because the detached accept thread kept a ws_tx clone alive, so the channel never closed and start_with_full_serve_options never returned. cdp_processor caught Ctrl-C and saved cookies, but that only ended its own task. Add a shared shutdown flag and Notify. On Ctrl-C cdp_processor sets the flag and notifies; the accept loop selects on the notify and breaks, so run_until returns and the process exits. The accept thread checks the flag between connections so it stops and drops ws_tx on the slow path. Closes #225 |
||
|
SGavrl
|
89692f3cb7 |
Perf/cdp network and lockfree (#214)
* perf(cdp): TCP_NODELAY + zero-buffer WS writes shave 130ms off newPage CDP traffic during browser.newPage() and Page.navigate is many small (~100-byte) request/response frames. Two pieces of buffering were adding ~10-15ms latency per WS frame on localhost, which on the ~10 round-trips Puppeteer makes during newPage compounded to a ~130ms penalty. 1. TCP_NODELAY on every accepted WS stream. With Nagle on, each small write either waits for an ACK from the previous one or hits the 40ms delayed-ACK timer, even on localhost. 2. WebSocketConfig.write_buffer_size = 0 on accept_async. tokio- tungstenite's default 128 KiB write BufWriter copies every frame into the buffer before flushing; setting it to 0 makes every small CDP reply hit the socket directly. Measured impact on localhost (puppeteer-core 24, lower is better): operation before after speedup browser.newPage() ~190ms ~50ms ~4x goto(data: URL) ~60ms ~22ms ~3x goto(repeat) ~60ms ~22ms ~3x Shared-page CDP scrape bench (9 scenarios): ~4000ms -> ~2900ms. 28/28 integration tests pass. * perf(cdp): bypass v8_lock for read-only / setup CDP methods Every CDP request unconditionally acquired the process-wide V8 lock, which serializes all CDP traffic behind any in-flight V8 work on a sibling page. During Puppeteer's browser.newPage() the ~8 setup calls (Page.enable, Runtime.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, etc.) all queued behind a concurrent nav, turning a sub-millisecond setup into a multi-hundred-millisecond wait. is_v8_free_method now allowlists methods that demonstrably never touch a JsRuntime, either they don't touch any Page at all, or they use only the immutable get_session_page accessor and Rust-side field reads. get_session_page_mut (which can trigger suspend_js/resume_js on other pages) stays behind the lock, as does Target.createTarget (which calls page.navigate). Allowlisted: most of Target/Browser/Storage, plus Page.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, Runtime.enable, Network.* cookie ops, Fetch.continue/fulfill/fail. Measured on the puppeteer-core 24 CDP scrape bench: browser.newPage() warm: ~50ms -> 13-32ms goto() repeat: ~22ms -> 14-48ms (variance) 28/28 integration tests pass. 18/18 realistic scrape scenarios pass. |
||
|
SGavrl
|
326096e20f |
Engine tier1 compat fixes (#208)
* engine: document.currentScript, data: subresources, dynamic module scripts
document.currentScript:
- ScriptInfo now carries the script element's nid.
- Before each script body runs, set globalThis.__currentScriptNid; clear after.
- Document.currentScript getter wraps the current nid. Next.js / Turbopack
reads document.currentScript.src to derive the chunk base path. Without
this the loader saw null, threw 'chunk path empty', and hydration stalled.
- Also threaded through the dynamic-script appendChild path so synchronously
injected <script> elements see their own element as currentScript.
data: URIs as subresources:
- subresource_allowed permits 'data:' alongside http/https. The bytes are
inline in the URI, no network fetch, no SSRF.
- Script-fetch loop short-circuits data: URLs and decodes them locally
instead of routing through the HTTP client.
- Real Chrome allows this by default; Instagram and most Meta properties
ship their bootstrap as <script src=data:application/x-javascript;base64,...>.
Dynamic <script type=module>:
- appendChild SCRIPT handler now accepts type='module' instead of dropping it.
- For external module src, uses dynamic import() instead of fetch + eval.
- For inline modules, wraps the source in a data: URL and import()s that
so module syntax (import/export) parses correctly.
Before / after on previously broken sites (HTML bytes rendered):
vercel.com 0 -> 960 KB
instagram.com 0 -> 669 KB
nextjs.org 0 -> 290 KB
music.apple.com 0 -> 1.88 MB
tailwindcss.com 0 -> 900 KB
linear.app 0 -> 2.4 MB
github.com PR 0 -> 468 KB
28/28 integration tests pass.
* engine: customElements.define upgrades existing matches + connectedCallback
The old customElements was a registry stub: define() recorded the class,
but never instantiated it on matching elements, never called the lifecycle
callbacks, and whenDefined() resolved synchronously with whatever was
already registered (or undefined forever for late-defined components).
New behavior matches the Web Components spec well enough for lit /
MusicKit / Polymer to wire up:
- define(): on register, querySelectorAll the document for matching tags,
copy the prototype methods onto each element instance, call the class
constructor with the element as 'this' for init logic, then fire
connectedCallback if the element is currently in the document.
- whenDefined(): returns a real Promise that resolves when the class is
later defined, not before.
- upgrade(root): walks the subtree, upgrades any matching unupgraded
element. Used by frameworks that mount components asynchronously.
We don't truly subclass HTMLElement per spec (Element is shared across
the engine), so the upgrade copies prototype descriptors onto the
element instance instead of swapping its [[Prototype]]. Behavior is
identical for the common patterns (custom methods, shadow DOM attach
in constructor, connectedCallback render).
28/28 integration tests pass.
* scraping: fix puppeteer page.$, click, page.$$ + form / radio / checkbox actions
These bugs broke every realistic Puppeteer scraping flow. Repro: type into
a search box, click a button, scrape rows. Before this commit the very
first step (page.$('selector')) returned null on every site.
Runtime.evaluate (and the two related wrappers) put the user expression
on a single line as '({expr})'. Puppeteer's bundled injected.js ends with
a '//# sourceURL=pptr:internal' line comment; the comment swallowed our
closing paren and the wrapped function failed to parse with a SyntaxError
that never propagated back. Result: every page.$, page.$$, type(),
hover(), click(), waitForSelector(), getByRole(), etc. silently returned
null. Fix: newline before the trailing ')' so the comment can't eat the
wrapper. Applied to all three wrap paths (eval, store_object_with_meta,
wrap_expression).
documentElement and body now return viewport dimensions for clientWidth /
clientHeight / scrollWidth / scrollHeight / offsetWidth / offsetHeight and
for getBoundingClientRect itself. Puppeteer's #clickableBox clips each
candidate quad to document.documentElement.clientWidth/Height; with the
old 100x20 stub every synthesized element bbox was clipped to empty and
clickablePoint threw 'Node is either not clickable or not an Element'.
Also aligned innerWidth / innerHeight to 1280x720 to match
Page.getLayoutMetrics.
document.elementFromPoint now does real hit testing against the synthetic
bboxes (depth-first, deepest match wins) instead of always returning
<body>. Input.dispatchMouseEvent uses it to pick the click target, which
removed the dependency on a __obscura_click_target side-channel that
the previous getBoundingClientRect call had to set right before the
dispatch.
Element.dispatchEvent now fires inline event handlers
(onclick='...' content attribute and the IDL onclick property) in
addition to addEventListener registrations. Without this, every site
that uses inline onclick / onsubmit / onchange was a no-op.
Input.dispatchMouseEvent's click path now triggers the right default
action by element kind: submit button -> form.submit(); checkbox /
radio -> toggle checked + dispatch 'change'; anchor with href -> nav.
Previously only anchor was handled, so checking a checkbox or submitting
a form by clicking its button did nothing through CDP.
Runtime.getProperties now allocates a stable child objectId for every
object-valued property and stashes the value in __obscura_objects.
Items whose value has a numeric nodeType also get subtype:'node' +
className so Puppeteer's JSHandle.asElement() wraps them as
ElementHandle. Without this, page.$$('selector') returned an array
of plain JSHandles and checkboxes[0].click() raised
'click is not a function'.
Real-world Puppeteer scrape coverage (test_all.py style scenarios):
- HN top stories scrape: fail -> pass
- GitHub repo file rows scrape: fail -> pass
- Wikipedia search input type: fail -> pass
- httpbin form fill + check: fail -> pass
- Cookies persistence across nav: pass -> pass
28/28 integration tests pass.
* scraping: inline scripts run on DCL, JS nav emits CDP events, hit testing
Three remaining gaps from realistic Puppeteer scraping flows:
1. DCL path skipped execute_scripts entirely. Returning early on
waitUntil=domcontentloaded meant every inline <script> was silently
dropped: addEventListener calls in onload bootstraps never ran,
form submit handlers never registered, page.click() hit no-ops.
Spec: DOMContentLoaded fires AFTER scripts. Now scripts run before
the lifecycle flips to DCL regardless of waitUntil.
To keep heavy SPAs from blowing the 25s Puppeteer goto timeout
now that scripts always run, added a soft deadline on the
script-execution phase (default 10s, OBSCURA_SCRIPT_DEADLINE_MS
to override). Both fetch and per-script execution check it.
2. JS-initiated navigation (form.submit, location.assign) did not
emit CDP frameNavigated / lifecycleEvent. Puppeteer's
waitForNavigation hung forever. Runtime.evaluate and
Runtime.callFunctionOn now drain pending_navigation through a
helper that fires the same post-nav event sequence Page.navigate
does.
3. Document.elementFromPoint walked the tree only descending into
ancestors containing (x,y). Our synthetic per-nid bboxes do not
form a containment hierarchy, so a deep <input> inside <label><p>
was never reached and clicks resolved to <body>. Switched to a
flat sweep, deepest match wins (highest nid as a proxy for tree
depth).
Realistic Puppeteer scrape scenarios: 18 of 18 assertions pass across
HN, Wikipedia, GitHub, httpbin forms, cookie persistence, and inline
JS submit handlers. 28/28 integration tests pass.
* scraping: fetch in page context, textarea, select, basic history
Four more gaps from the broad API sweep:
1. fetch() and XHR from page JS hung forever. set_intercept_tx() in
the runtime auto-set state.intercept_enabled = true as a side
effect, so every navigation silently enabled Fetch interception
even when the CDP client never called Fetch.enable. op_fetch_url
then routed every request through the intercept channel waiting
for a Fetch.continueRequest that the client wasn't going to send.
Split the flag from the channel: set_intercept_tx wires the
channel without enabling; new set_intercept_enabled toggles the
flag. Fetch.enable / Fetch.disable now call both, and the nav
path only wires the channel.
2. page.type into textarea lost everything past the first newline.
The dispatchKeyEvent handler skipped text=='\n' entirely, so
'multi\nline' inserted only 'multi'. Also Enter on a textarea
tried to submit the nearest form instead of inserting a newline.
Now Enter inserts \n inside textareas and only submits forms
from input fields, matching Chrome.
3. <select>.value did not reflect the selected option. Generic value
getter returned the select's value attribute (empty) instead of
the matching option. Added select-aware getter that walks options
for the selected one and falls back to the first; setter accepts
page.select() and dispatches a 'change' event.
4. Basic navigation history. Pages now track a visited-URL stack
and history_index. Page.getNavigationHistory returns the real
history; Page.navigateToHistoryEntry honors entryId and emits the
same CDP nav-event sequence as Page.navigate so the response and
lifecycleEvent reach waiting clients. resetNavigationHistory
clears the stack. Push happens after every successful nav (skips
consecutive duplicates so Page.reload doesn't grow the stack).
Broad API sweep across 13 feature groups: 43/44 PASS.
All categories solid (navigation, selectors, typing, waitForSelector,
evaluate, cookies, keyboard, attributes, events, fetch/XHR, storage,
timers). goBack reaches its target URL but Puppeteer client-side
waitForNavigation still hangs on it; the CDP frameNavigated +
lifecycleEvent fire correctly, so the remaining gap is in event
shape rather than logic. 28/28 integration tests pass.
|
||
|
SGavrl
|
b98e457428 |
engine: large compat + reliability pass (#206)
CDP / navigation: - Route all Page.navigate through process_with_interception so the parent dispatcher stays responsive during long navs (fixes Target.createTarget timeouts after a handful of newPage calls). - Default waitUntil to domcontentloaded. Puppeteer and Playwright don't pass it server-side and wait client-side via lifecycleEvent, so the old Load default blocked nearly every nav on slow sites. - 30s navigation hard ceiling, OBSCURA_NAV_TIMEOUT_MS to override. - execute_scripts loop now checks wall-clock deadline per iteration. - Emit Network.requestWillBeSent before Page.frameNavigated so Playwright's FrameManager attaches the navigation response (#189). - Use loaderId as the Document request id so page.goto() resolves to a Response instead of null (#189). - Fresh executionContextId per navigation, no more 'Cannot find context with specified id' after the first nav (#192). - POST navigation params plumbed through the intercept path. DOM / Web API surface: - MutationObserver: subtree filter walks parent chain, fires on textContent / CharacterData.data / innerHTML, type filter respected, oldValue passed. - IntersectionObserver: multi-fire (initial + 120/500/1500ms), disconnect / unobserve / takeRecords, root / rootMargin / thresholds. - ResizeObserver: per-target firing from getBoundingClientRect, drop the no-op override that was clobbering the real class. - crypto.subtle.digest: real SHA-1/256/384/512 via op_subtle_digest. - DOMParser / document.implementation.createHTMLDocument / createDocument: detached doc with body, head, documentElement, querySelector*, getElementById, getElementsByTagName. - WebSocket / EventSource / BroadcastChannel: real EventTarget, open fires on microtask. - indexedDB: request, object store, transaction, database with async onsuccess and in-memory Map storage. - getComputedStyle: width/height/top/left from bbox so React virtualization libs render items. Site compatibility went from 3/40 to 175/175 reachable sites in the sweep. Perf is equal or faster than pristine main on every site tested (DCL default removes the block on long-tail subresources). 28/28 integration tests pass. |
||
|
|
ea89f050cd |
Fix puppeteer exposeFunction end-to-end: real bindings, preload timing, URL coercion (#166)
* fix(js): coerce URL constructor input/base to string
The polyfill at bootstrap.js called `base.match(...)` directly, which
threw `TypeError: base.match is not a function` whenever callers passed
a Location object (e.g. `new URL(href, window.location)`), a URL
instance, or any other non-string with a sensible toString. Per the
WHATWG URL Standard both `url` and `base` are stringified before
parsing.
Coerce explicitly so the existing regex paths keep working and the
constructor matches real-browser behaviour. AngularJS, jQuery, and
anything that resolves a relative URL against `window.location` now
loads under Obscura.
* fix(browser,cdp): run preload scripts before page scripts
`Page.addScriptToEvaluateOnNewDocument` was executed AFTER the page's
own scripts (and even after the load event), which silently broke
every CDP contract that depends on early injection — most visibly
puppeteer's `page.exposeFunction`, whose user-facing wrapper has to
override `globalThis[name]` before any `<script>` tag in the document
runs.
Move the registered preload sources onto `Page` and execute them
inside `navigate_single`, right after `__documentReadyState__ =
'loading'` and before the parser-discovered scripts. The CDP layer
hands the current preload set to the page via `set_preload_scripts`
before kicking off navigation, replacing the two post-navigate
`execute_preload_script` loops in `domains/page.rs` and `server.rs`.
* feat(cdp,js): emit Runtime.bindingCalled for addBinding shims
`Runtime.addBinding` was a no-op stub that installed an empty
`globalThis[name] = function () { return null; }` placeholder. Page JS
calls were silently swallowed and the Node-side callback registered
via puppeteer's `page.exposeFunction` never fired.
Wire the binding all the way through:
1. New `op_binding_called(name, payload)` op forwards every page-side
call into a `pending_binding_calls` queue on `ObscuraState`.
2. `Page::take_pending_binding_calls` drains the queue.
3. `dispatch()` empties every page's queue after each CDP request and
pushes one `Runtime.bindingCalled` event per entry into
`ctx.pending_events`, so the writer task forwards them to the
client like any other CDP event.
4. The `addBinding` handler installs a real shim that calls the op,
registers it as a preload script (keyed `__obscura_binding__<name>`)
so it survives navigations, and evaluates it now so the binding is
usable on the current document.
5. Implement `removeBinding` to mirror `addBinding`.
The shim coerces non-string arguments via `JSON.stringify`, matching
the wire contract that puppeteer's `exposeFunction` wrapper depends
on (`Runtime.bindingCalled.params.payload` is always a string).
* fix(cdp): emit canAccessOpener in all TargetInfo payloads, add aarch64 cross-compile CI
Merges PR #165 and PR #172.
PR #165: The CDP spec marks canAccessOpener as a required boolean on
TargetInfo but Obscura was omitting it. Strict Rust clients like
chromiumoxide reject the event entirely and panic with "Created target
not present" when the field is absent. Added canAccessOpener: false
to every TargetInfo emitted by setDiscoverTargets, getTargets,
createTarget, attachToBrowserTarget, attachToTarget, and getTargetInfo.
PR #172: Added a cross-compile matrix entry for aarch64-unknown-linux-gnu
running on ubuntu-22.04 x86_64, producing obscura-aarch64-linux-cross.
This is a fallback for when GitHub ARM runners are unavailable without
removing the native ARM build.
* fix(cdp): keep HTTP control plane reachable while V8 JS evaluation blocks (#173)
fix(cdp): unblock HTTP control plane during synchronous V8 execution
Fixes #62.
Previously, the TCP accept loop and CDP processor shared a single-threaded
tokio LocalSet. Long synchronous JS evaluations starved the accept loop,
causing incoming connections to hang in the kernel backlog and effectively
killing the HTTP control plane.
This moves the TCP listener and `/json/*` endpoints to a dedicated OS
thread using `std::net::TcpListener`. HTTP requests are now handled
directly via blocking I/O on this thread, completely bypassing V8.
WebSocket connections are handed off to the LocalSet via a bounded mpsc
channel (capacity 128). If V8 stalls and the queue fills, new WS
connections are dropped (ECONNRESET) rather than buffering unboundedly
to prevent OOMs.
Includes a new integration test to verify the control plane stays
responsive during a 5s JS loop, plus inline documentation clarifying
thread lifetime semantics and a minor test TOCTOU race.
* fix(js,dom): add Storage global, CharacterData ChildNode methods, fix remove/isConnected (#176)
fix(dom): add missing DOM/JS shims for modern frameworks
Closes #163.
Adds missing DOM and JS shims required by modern frameworks (Svelte 5,
Vue, Lit) and consent-management SDKs to execute without errors:
- Define the `Storage` global constructor to resolve `ReferenceError`s
and fix `Storage.prototype` access.
- Apply standard ChildNode methods (`before`, `after`, `replaceWith`,
`remove`) to `CharacterData.prototype`. This prevents `TypeError`s when
frameworks call these on Text or Comment nodes during hydration.
- Update `Element.prototype.isConnected` to perform a genuine parent-walk
rather than returning a hardcoded `true`.
- Fix `Element.prototype.remove` to properly clean up the internal ID
index so removed elements are no longer queryable.
- Remove outdated no-op stubs on the `Element` class that were shadowing
the proper polyfill implementations.
* feat(cli): add --dump assets to inventory page sub-resources (124) (#174)
feat: add `--dump assets` to enumerate page sub-resources
Adds a `--dump assets` flag to output all page-referenced sub-resources
(scripts, stylesheets, images, iframes, media, embeds, objects) as NDJSON,
allowing users to easily replay the asset graph.
- Output format: `{"url":"https://example.test/app.js","type":"script"}`
- URLs are resolved against the page origin (matching `--dump links`).
- The `type` tag is derived from the source element. `<link>` elements map
their `rel` attribute to specific types (stylesheet, icon, manifest, preload,
prefetch, modulepreload, dns-prefetch, preconnect, alternate), falling back
to "link" if missing or unknown.
- Adds 7 tests covering URL resolution, rel-mapping, supported selectors,
and empty-attribute skipping.
- Updates README dump-format table and examples.
* fix(cdp,js): playwright actionability stubs and objectId resolve (#178)
Playwright fails locator actions on Obscura for two avoidable reasons
that are separate from the missing layout engine. Both have trivial
fixes that unblock the actionability polling loop.
- Element.checkVisibility(opts) was undefined. Playwright >= 1.25 calls
it before every input event and retries until its action timeout if
the method isn't a function. Add a stub that returns true. Without a
layout pass we can't compute it for real, but a permissive stub still
beats the undefined-function path.
- ARIA reflection properties were missing on Element (role, ariaLabel,
ariaSelected, ariaExpanded, ariaHidden, ariaChecked, ariaDisabled,
ariaRoleDescription). Playwright's getByRole and getByLabel locators
read them. Reflect through to the underlying aria-* attributes for
both read and write.
- DOM.getBoxModel and DOM.getContentQuads ignored the objectId param.
Playwright passes only objectId (from a prior DOM.resolveNode), and
the handlers silently defaulted to node 0, so dispatchMouseEvent
always clicked the wrong element. Extract a resolve_node_id helper
that honors nodeId, backendNodeId, and objectId in that order.
- Page.captureScreenshot and Page.captureSnapshot now return a clear
"not supported, use a real Chromium for the screenshot leg" error
instead of the generic "Unknown Page method" fallback. Matches the
pattern already in place for Page.printToPDF.
Partial fix for #45. The architectural blocker (no layout engine, so
multi-element click coordinates are wrong) is unchanged.
* fix(mcp): allow browser MCP clients through CORS preflight (#179)
The OPTIONS preflight hardcoded Access-Control-Allow-Headers to just
Content-Type. Any browser MCP client sending mcp-protocol-version (per
the MCP spec) or Authorization / X-API-Key (typical for hosted
deployments) failed the preflight and the browser blocked the actual
request.
Add those three to the allow-list and a 24h Max-Age so browsers don't
re-preflight every call. Adds an integration test that drives the
server with a real preflight request.
Closes #175.
* fix(js,dom): scope element queries, real timers, form/prepend/equal (#180)
- Element.querySelector / querySelectorAll scope to the receiver subtree
via a new query_selector_scoped Rust op, not always document root.
Same for DocumentFragment.
- setTimeout respects its delay argument via a new async op_sleep.
setInterval is a real repeating timer instead of an alias to setTimeout.
- document.forms / images / links return live querySelectorAll results
instead of hardcoded [].
- HTMLFormElement subclasses Element with .elements / .length, and form
nodes are wrapped as HTMLFormElement so instanceof works. submit()
stays on Element.prototype to preserve the existing submit-event
dispatch and form-data navigation path.
- Element.prepend has a real implementation.
- Node.isEqualNode does a structural compare (nodeType, nodeName,
nodeValue, attributes, recursive children) instead of nid identity.
- Fix long-standing insert_before arg-order bug: bootstrap.js was
passing (parent, new, ref) but _dom only forwards two args, so ref
was silently dropped. Pass (new, ref) directly. Same fix in
replaceChild.
Adds 3 selector unit tests and 6 runtime regression tests covering
the JS shim fixes.
Closes #105.
* fix(cdp): unblock puppeteer connect, bound Runtime.evaluate, full TargetInfo (#181)
Three independent gaps that hurt browser-based MCP clients connecting
to obscura's CDP endpoint:
- Runtime.enable returned "No page" when called before a session was
attached. puppeteer-extra's FrameManager.initialize fires it on the
browser-level WebSocket BEFORE any page target exists, so the
standard puppeteer.connect / browser.newPage flow died on connect.
Now reply with {} when no session is attached and let the next
Target.attachToTarget emit executionContextCreated.
- Runtime.evaluate had no per-call timeout. A long synchronous JS
expression held the global V8 lock indefinitely and starved every
other CDP command on the session. Honor the CDP `timeout` parameter
and default to 30s (Chrome's protocolTimeout). Returns a clear error
on expiry so client-side deadlines fire cleanly instead of hanging.
- Target.getTargetInfo for the browser target omitted canAccessOpener.
Strict CDP clients (chromiumoxide) panic when the field is missing.
Fill it in (false for the browser target itself).
Partial fix for #122. The remaining items (networkIdle hard-deadline
on SPAs, /json/list hardcoded data, docs on cheap-vs-queued methods)
need bigger or non-code changes and are deferred.
* fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes #113.
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity (#170)
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity
Closes #168.
Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.
Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(cdp): fall back to default cookie jar, add Network.getAllCookies
Addresses @SGavrl review on PR #170:
- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
cookie jar via a shared helper that prefers the session page's jar and
falls back to ctx.default_context.cookie_jar when no target has been
attached. Mirrors how Storage.setCookies already routes when there is
no session, so Puppeteer/Playwright flows that issue cookie methods
before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
existing getCookies handler so both arms return the full jar.
Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).
* fix(net): use bundled webpki roots in stealth client (#184) (#185)
set_default_paths only resolves OpenSSL's compile-time CA dirs, which
exist on Linux but are empty on Windows. The stealth client's cert store
ended up empty there and every TLS handshake failed with
CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc.
CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's
trust list), which behaves the same on every platform and removes the
dependency on a system CA store.
* fix(js): Element.attributes, on* event defaults, per-nid getBoundingClientRect (#45) (#186)
Three DOM-completeness gaps that broke jQuery 1.11.3 init and Playwright
actionability:
1. Element.attributes was undefined, so legacy IE feature-detect code
like div.attributes[eventName].expando threw TypeError. Added a
NamedNodeMap-like backed by a new attribute_names DOM op (length,
numeric index, string name lookup, getNamedItem/setNamedItem,
iterable).
2. Standard on* event handler properties were missing from window, so
('on' + ev) in window returned false and forced jQuery into the
legacy IE branch. Initialised the 79 HTML-spec on* handlers to null.
3. getBoundingClientRect returned the same {x:8,y:8,width:100,height:20}
literal for every element, so Playwright's hit-testing couldn't tell
elements apart. Replaced with a deterministic synthesis from _nid
mapped into a 12-column grid sized to a 1280x720 viewport. Still not
pixel-accurate (no layout engine), but stable across reads and
distinct per node.
* feat: cookie jar persistence + --storage-dir flag + fix HTTP cookie header filtering (#114)
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
* fix(cdp): page.goto returns Response over Puppeteer and Playwright (#189) (#190)
Two correlation invariants real Chrome holds were missing, so the
nav Response was never attached to goto().
requestId === loaderId for the main Document event. Both clients
gate isNavigationRequest() on this; we were emitting a synthetic
per-page id.
Network.requestWillBeSent for the Document fires before
Page.frameNavigated. Playwright's FrameManager needs the request
present as pendingDocument before the frame commits, otherwise
currentDocument lands with request=undefined.
Subresource events and the Document's responseReceived /
loadingFinished stay where they were.
Verified end-to-end on both clients against example.com (200),
google.com (200), httpbin/404 (404).
* fix(browser,cdp): run preloads on about:blank, tighten binding shim arity
Two follow-ups from PR #166 review:
- navigate_single's about: branch returned before any preload sources
ran, so addScriptToEvaluateOnNewDocument scripts and the
Runtime.addBinding shim never installed on about:blank. Puppeteer
hits this every session: browser.newPage() lands on about:blank and
exposeFunction is dead until the first real navigation. Now the
about: branch runs the preload set after init_js(), guarded by the
same 5s timeout as regular preloads.
- The Runtime.addBinding shim used JSON.stringify on non-string args
and silently accepted any arity. Chromium's V8InspectorImpl rejects
calls without exactly one argument and ToString-coerces that arg
before emitting Runtime.bindingCalled. Match that: guard on
arguments.length and use String(arg) for the non-string branch.
Single-string calls (the path puppeteer's exposeFunction wrapper
uses) round-trip unchanged.
* fix(cdp): allocate fresh execution context ids per isolated world (#192) (#200)
Page.createIsolatedWorld and do_navigate both hardcoded context id 100
for every isolated world emission. After the second navigation Playwright
expected a fresh incrementing id (101, 102) while obscura kept re-using
100. Runtime.evaluate then failed with 'Cannot find context with
specified id: 101' because the id was never registered in
valid_context_ids and the client's bookkeeping had diverged from the
server's.
Added next_isolated_context_id counter on CdpContext (starts at 100) and
a next_isolated_context() helper that claims an id, registers it in
valid_context_ids, and increments. Both call sites now use it.
* feat(cli): default V8 heap to 4 GB on 64-bit so heavy SPAs don't OOM (#199) (#202)
V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with
heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/
web-scraping) blow past it and the worker dies with
'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session
with it.
Always set --max-old-space-size=4096 (1024 on 32-bit targets) before
V8 startup, matching headless Chrome's default. User-supplied
--v8-flags are appended after the default so they win (V8 parses
left-to-right and the last value wins), which keeps the existing
escape hatch intact for memory-constrained environments.
---------
Co-authored-by: SGavrl <[email protected]>
Co-authored-by: Ben Younes <[email protected]>
Co-authored-by: zhangyang-crazy-one <[email protected]>
|
||
|
SGavrl
|
e7bb0c676d |
feat(cli): --allow-private-network for localhost development (#33) (#203)
Picks up @Timtech4u's PR #33, which has been waiting since April. The design (CLI flag + threading through BrowserContext + http client) is theirs and is correct; this is a clean rewrite on top of current main because repeated rebase rounds had been reverting unrelated work that landed in the meantime. Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is process-wide and clunky in pipelines. Multiple users (@Timtech4u in the original PR, then @duartemvix and @necrogami in the comment thread) reported they cannot use obscura for development against http://localhost:N without it. Adds --allow-private-network as a global CLI flag, works on fetch, serve, and scrape. Threads through BrowserContext::with_storage_and_network and the http client's validate_url. CdpContext gets a new_full builder and obscura_cdp::start_with_full_serve_options entry point so 'obscura serve --allow-private-network' lets Puppeteer/Playwright navigate to localhost too. Default remains blocked. Backward compat: the env var still works (OR'd with the flag inside validate_url). All existing constructors and entry points unchanged. Verified end-to-end against a local http.server on 127.0.0.1: fetch and CDP Page.navigate both blocked by default, both succeed with the flag. Public URLs unaffected. Integration suite 28/28, full workspace cargo test all green. |
||
|
SGavrl
|
682431f766 |
fix(cdp): allocate fresh execution context ids per isolated world (#192) (#200)
Page.createIsolatedWorld and do_navigate both hardcoded context id 100 for every isolated world emission. After the second navigation Playwright expected a fresh incrementing id (101, 102) while obscura kept re-using 100. Runtime.evaluate then failed with 'Cannot find context with specified id: 101' because the id was never registered in valid_context_ids and the client's bookkeeping had diverged from the server's. Added next_isolated_context_id counter on CdpContext (starts at 100) and a next_isolated_context() helper that claims an id, registers it in valid_context_ids, and increments. Both call sites now use it. |
||
|
SGavrl
|
191c5ac7a2 |
fix(cdp): page.goto returns Response over Puppeteer and Playwright (#189) (#190)
Two correlation invariants real Chrome holds were missing, so the nav Response was never attached to goto(). requestId === loaderId for the main Document event. Both clients gate isNavigationRequest() on this; we were emitting a synthetic per-page id. Network.requestWillBeSent for the Document fires before Page.frameNavigated. Playwright's FrameManager needs the request present as pendingDocument before the frame commits, otherwise currentDocument lands with request=undefined. Subresource events and the Document's responseReceived / loadingFinished stay where they were. Verified end-to-end on both clients against example.com (200), google.com (200), httpbin/404 (404). |
||
|
|
906dcfcde1 |
feat: cookie jar persistence + --storage-dir flag + fix HTTP cookie header filtering (#114)
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
|