c19a21c14289378c1bed042986bcf52692862ec7
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 SGavrl
|
c19a21c142 |
api: honor interception Continue overrides and fix pre-navigation enable
op_fetch_url ignored the url/method/headers/body fields on InterceptResolution::Continue, so the interception channel could pass, mock, or block a request but not rewrite it. Apply them: shadow url/method/body and derive custom_headers from the override after the channel resolves Continue. Also fix enable_interception() called before the first navigation: init_js propagated intercept_tx to the new runtime but not intercept_enabled, so the runtime started with interception disabled and op_fetch_url never intercepted. Propagate intercept_enabled in init_js as well. Add a test that rewrites a fetch's url via Continue and asserts the rewritten response is captured. |
||
|
SGavrl
|
1f5066fc30 |
api: add request/response interception API to Page (#306)
Expose the interception infrastructure that already existed internally but was
private, so embedders can intercept, observe, mock, or block JS fetch()/XHR
without forking or unsafe casts.
On obscura::Page:
- add_preload_script: inject JS before any page <script> (CDP
Page.addScriptToEvaluateOnNewDocument). The field already ran the scripts;
this only adds the method.
- enable_interception: returns a receiver for every JS fetch()/XHR request;
resolve each through its resolver with InterceptResolution::{Continue,
Fulfill, Fail} to pass, mock, or block (CDP Fetch.enable equivalent).
- on_request / on_response: passive callbacks. on_response is the main path for
capturing API response payloads from SPAs.
on_request/on_response previously fired only for navigation; op_fetch_url now
invokes them for JS fetch()/XHR too (both reqwest and stealth paths), gated on
a non-empty callback vector so the common no-callback case pays nothing.
Re-export InterceptedRequest, InterceptResolution, RequestInfo, ResourceType,
RequestCallback, ResponseCallback, Response from the obscura crate.
Integration test tests/interception.rs exercises all three against a local echo
server.
|
||
|
Gabriel
|
12b927f476 |
Merge pull request #325 from h4ckf0r0day/feat/global-stealth-flag
cli: make --stealth a global flag, available on every subcommand |
||
|
SGavrl
|
8c14c4393d |
cli: make --stealth a global flag, available on every subcommand
--stealth was a per-subcommand flag defined separately on fetch, serve, and mcp, and missing from scrape entirely. You could not enable stealth for parallel scraping, and `obscura --stealth fetch` (flag before the subcommand) errored. Promote --stealth to a single global flag like --proxy, so it is valid before or after any subcommand and applies uniformly to fetch, serve, scrape, and mcp. scrape now forwards it to its workers via OBSCURA_STEALTH (mirroring OBSCURA_PROXY), and the bare `obscura` default CDP server honors it too. Verified: navigating directly to a blocklisted tracker renders 55503 chars without stealth and 39 (blocked) with it, identically on fetch and scrape. Obstacle course 33/33. |
||
|
Gabriel
|
d2206fea84 |
stealth: fingerprint consistency, scripted-request TLS, FormData
**Fingerprint consistency (stealth)** The stealth TLS, navigator, and WebGL surfaces previously disagreed. The wreq client reported Windows Chrome but navigator reported Linux, and WebGL returned Direct3D11 renderers for a Linux UA — a combination no real browser ever produces. Everything now presents one consistent Windows Chrome 145 identity: the HTTP User-Agent header, sec-ch-ua-platform, navigator.userAgent, navigator.platform, userAgentData, and WebGL renderer all agree. userAgentData brands are derived from the Chrome major version via Chromium's GREASE algorithm instead of a single hardcoded token. **sec-ch-ua client hints on the default (non-stealth) path** The plain navigation client hardcoded `sec-ch-ua-platform: "Linux"` and Chrome 149 brands regardless of which profile was selected. Both headers are now derived from the actual User-Agent using the same GREASE algorithm, so non-stealth requests are also internally consistent. **Route scripted fetch()/XHR through the stealth TLS client** In stealth mode, JS `fetch()` and `XMLHttpRequest` were still going through the plain reqwest/rustls stack, so subresource requests (sensors, WAF challenge verify calls) arrived with a non-browser TLS fingerprint. They now share the navigation's Chrome TLS fingerprint and sec-ch-ua client hints. The non-stealth path is unchanged. **FormData serialization in fetch() — the booking.com fix** Passing a `FormData` as a `fetch()` or `XHR.send()` body sent the literal string `"[object Object]"` instead of a multipart payload. booking.com's AWS WAF challenge POSTs its proof-of-work solution to `/mp_verify` as FormData, so the verify always failed and no `aws-waf-token` was issued. `FormData` is now serialized to a proper `multipart/form-data` body with a generated boundary; `URLSearchParams` gets `application/x-www-form-urlencoded`. XHR routes through `fetch()` so it is covered too. **navigator.permissions.query defaults** `permissions.query` returned `"granted"` for every permission. Real Chrome defaults camera, microphone, geolocation, and midi to `"prompt"`. Fixed. **SHA-512/224 and SHA-512/256 in crypto.subtle.digest** Added FIPS 180-4 truncated SHA-512 variants to the Rust op and the JS allowlist. WAF challenge scripts that use these algorithms now get real output instead of a NotSupportedError. **location updates synchronously on navigation, Blob-URL Worker race fixed** |
||
|
SGavrl
|
8582fcceca |
Merge stealth-overhaul into main
Resolved conflicts: - page.rs: keep both set_blocked_urls (#315) and stealth client wiring - ops.rs: keep SHA-512/224 + SHA-512/256 from stealth branch; keep the unreachable catch-all (vec![]) from #314 instead of silent SHA-256 fallback; updated comment accordingly - bootstrap.js: extend the digest algorithm allowlist to include SHA-512/224 and SHA-512/256 to match the Rust additions |
||
|
SGavrl
|
768cbc056b |
net: send non-stealth navigation headers in Chrome's order
The default HTTP path emitted headers with user-agent first and sec-fetch-dest before sec-fetch-site, an order no real browser produces and a known fingerprint signal. Reorder the explicit headers to Chrome's top-level navigation order (sec-ch-ua family, upgrade-insecure-requests, user-agent, accept, sec-fetch-site/mode/user/dest, accept-language). reqwest still appends accept-encoding and host after these, so accept-encoding lands after accept-language instead of before it; the rest matches Chrome exactly. |
||
|
SGavrl
|
db24d7a18c |
js: prompt for sensitive permissions in navigator.permissions.query
permissions.query returned "granted" for every permission except notifications. Real Chrome defaults camera, microphone, geolocation, and midi to "prompt", so answering "granted" for them is a bot tell. Return "prompt" for those (keeping notifications in sync with Notification.permission) and "granted" for the rest. |
||
|
SGavrl
|
f7b3ff5d75 |
net: derive sec-ch-ua client hints from the UA on the default HTTP path
The non-stealth navigation path hardcoded sec-ch-ua-platform "Linux" and a fixed sec-ch-ua using Chrome 149's GREASE token, regardless of the selected profile. With the default Windows Chrome 143 profile that meant a Windows UA and navigator.userAgentData traveling with a Linux sec-ch-ua-platform header and mismatched brands, a cross-layer inconsistency a site can check at a glance. Derive both headers from the profile User-Agent using Chromium's per-major GREASE algorithm, so the HTTP client hints agree with the JS surfaces. |
||
|
Gabriel
|
0913be86dc |
cdp: make Backspace and typing honor text selection (#316) (#321)
Backspace always trimmed the last character and typing always appended, ignoring any selection. So "select-all, then Backspace to clear" left all but one character, and typing over a selection accumulated text instead of replacing it. Backspace now deletes the selected range when one is set, and typed text replaces it, matching the browser. selectionStart is null during ordinary typing, so the previous append/trim behavior is preserved when no selection is tracked. Pairs with the triple-click select-all in #320 to make the common "triple-click, Backspace, retype" clear pattern work end to end. Also fixes the same pattern when a page sets the selection via select() or setSelectionRange(). |
||
|
SGavrl
|
3eb28da5c6 |
js: serialize FormData bodies to multipart/form-data in fetch
fetch() and XHR did String(body), so passing a FormData as the request body sent the literal "[object Object]" and lost the multipart payload along with its boundary. Servers expecting multipart/form-data replied 400 "Invalid boundary for multipart/form-data", which is what blocked booking.com: the AWS WAF challenge POSTs its proof-of-work solution to /mp_verify as FormData, so the verify always failed and no aws-waf-token was issued. Serialize FormData to a multipart/form-data body with a generated boundary and set the matching Content-Type, the way a real browser does; URLSearchParams bodies get application/x-www-form-urlencoded. XHR.send routes through fetch() so it is covered too. With this, obscura --stealth renders the real booking.com landing page the same as headless Chrome. |
||
|
SGavrl
|
402de261aa |
js: fix Blob-URL Worker race and update location synchronously on navigation
URL.createObjectURL stored blob content asynchronously (blob.text().then) but the Worker constructor read __blobStore synchronously, so new Worker(blobURL) in the same tick raced the store, fell through to fetch(blobURL), and failed with net::ERR_FAILED. This broke Blob-URL workers including the AWS WAF challenge proof-of-work worker. Store synchronously from Blob._bytes, which the obscura Blob materializes in its constructor. location.reload was a no-op, so challenges that reload after setting a token cookie never re-fetched. It now navigates like assign/replace. location.href/assign/replace/reload cleared __virtualUrl immediately, so a script that set location.href and re-read it saw the old URL until the async navigation completed; real browsers update location.href synchronously. Preview the resolved target in __virtualUrl and clear it in __obscura_init once the navigation completes, so immediate reads match and redirects still win. |
||
|
SGavrl
|
2e0981ef3f |
stealth: route scripted fetch()/XHR through the wreq client
op_fetch_url backed JS fetch() and XHR with a plain reqwest/rustls client, so in stealth mode scripted requests carried a non-browser TLS fingerprint and none of the Chrome client hints the main navigation sends. Bot managers that score subresource requests (the AWS WAF challenge verify call, Akamai sensors) read that as a script and reject it. Add StealthHttpClient::send_single and a stealth_fetch_all path that op_fetch_url uses when a stealth client is installed, preserving the redirect, SSRF, and CORS handling so JS-level requests share the navigation's Chrome TLS fingerprint and sec-ch-ua client hints. The reqwest path is unchanged for non-stealth. Adds a stealth feature to obscura-js gated on obscura-net/stealth. |
||
|
SGavrl
|
4309935abd |
stealth: keep the Chrome 145 identity consistent across TLS, JS, and WebGL
Stealth mode forced a Linux navigator UA and platform but reported Windows in userAgentData.getHighEntropyValues and served Windows Direct3D11 WebGL renderers, a combination no real Linux Chrome ever produces and that cross-layer fingerprint checks flag. Present one consistent Windows Chrome 145 identity instead: the wreq emulation uses Platform::Windows, and the navigator surfaces read the UA and platform from shared constants so the HTTP User-Agent header, sec-ch-ua-platform, navigator.userAgent, and userAgentData all agree. userAgentData brands are now derived from the Chrome major version via Chromium's GREASE algorithm rather than a single hardcoded token, and a Linux WebGL renderer pool is added so a Linux UA can never report a Direct3D11 renderer. |
||
|
SGavrl
|
dc780d77e4 |
js: handle SHA-512/224 and SHA-512/256 in crypto.subtle.digest
The digest op fell through to SHA-256 for any unrecognized algorithm name, so SHA-512/224 and SHA-512/256 silently returned a SHA-256 hash. Add the two FIPS 180-4 truncated variants, which the AWS WAF challenge proof-of-work (used by booking.com and other CloudFront/WAF sites) requires. Fixes #314. |
||
|
Gabriel
|
edde67ddd4 |
crypto.subtle.digest: reject unsupported algorithms (#314) (#319)
Unknown algorithm names (SHA-512/224, MD5, etc.) were silently falling
back to SHA-256, producing wrong bytes with no error. Now throws
DOMException('NotSupportedError') to match Chrome behavior.
Supported: SHA-1, SHA-256, SHA-384, SHA-512.
|
||
|
Gabriel
|
a8358cc82d |
add Plugin, MimeType, MimeTypeArray globals (#305) (#318)
These three interfaces were missing from the global scope entirely. Site bundles that reference Plugin as a constructor threw "ReferenceError: Plugin is not defined". Added as plain function declarations (same pattern as PluginArray) so they survive the V8 snapshot. navigator.plugins now returns real Plugin instances and navigator.mimeTypes returns a real MimeTypeArray with MimeType instances, so instanceof checks work correctly. |
||
|
Gabriel
|
cb6ed24d98 |
docs: add AGENTS.md
Repo guidance for AI coding agents and contributors, covering the things you can't infer from the code: build (V8 from source, cmake for stealth, per-crate scoping), why tests run under cargo nextest and not cargo test (one V8 isolate per process), the obstacle course as the behavioral gate, the DOM/runtime gotchas, the hang/panic-safety invariants, and a pre-finish validation checklist. |
||
|
SGavrl
|
3f2f507a7f | docs: add AGENTS.md | ||
|
Gabriel
|
28f1a7fd48 | Update README.md | ||
|
Gabriel
|
a919eb2234 | Added NodeMaven sponsor logo | ||
|
Gabriel
|
d9835e029b |
js: add document.evaluate and XPathResult (XPath subset)
Pages and scrapers that call document.evaluate() hit "TypeError: document.evaluate is not a function", which can break script execution and hydration. Add document.evaluate() plus the XPathResult global with the standard result-type constants so those paths work (needed for Maps scraping). The evaluator handles the common relative forms: //tag and .//tag with attribute predicates (@attr, @attr='v', contains(@attr,'v'), starts-with(@attr,'v')), backed by querySelectorAll and attribute filtering. Absolute paths, positional predicates ([n]), text() and axes are not supported and return an empty result. |
||
|
Gabriel
|
bee4bc5b93 | Update README.md | ||
|
Gabriel
|
6173cfdd43 | Sponsor logo update | ||
|
Gabriel
|
b8fa722870 | Update README.md | ||
|
Gabriel
|
3f120006cb | Added 9proxy sponsor logo | ||
|
SGavrl
|
3f242aa285 | Merge branch 'main' of https://github.com/h4ckf0r0day/obscura | ||
|
Gabriel
|
0525c329f3 |
cli: list fetch()/XHR resources in --dump assets
--dump assets only walked static DOM attributes (script[src], img[src], ...), so resources a page pulled in with fetch() or XHR never showed up (Closes #301). Record each absolute URL requested through op_fetch_url in ObscuraState, expose it via the runtime and Page, and append those URLs to --dump assets as type "fetch", deduped against URLs already listed from the DOM. |
||
|
SGavrl
|
ab27e9153d |
cli: list fetch()/XHR resources in --dump assets
--dump assets only walked static DOM attributes (script[src], img[src], ...), so resources a page pulled in with fetch() or XHR never showed up (issue #301). Record each absolute URL requested through op_fetch_url in ObscuraState, expose it via the runtime and Page, and append those URLs to --dump assets as type "fetch", deduped against URLs already listed from the DOM. |
||
|
Gabriel
|
d526a4da7d |
js: make event.isTrusted false for page-constructed events
isTrusted was hardcoded to true for every Event, so new Event('x').isTrusted returned true. Real browsers return false for script-constructed events and true only for user-agent dispatched ones, so this was a trivial bot-detection tell (Closes #303).
Track trusted events in a closure-private WeakSet that page JS cannot read or forge; default isTrusted to false. The CDP input pipeline marks its synthesized mouse/key events trusted via a non-enumerable __obscura_markTrusted helper, so automation clicks stay trusted like real Chrome over CDP while page-built events report false.
|
||
|
SGavrl
|
6264e6bfad |
js: make event.isTrusted false for page-constructed events
isTrusted was hardcoded to true for every Event, so new Event('x').isTrusted
returned true. Real browsers return false for script-constructed events and
true only for user-agent dispatched ones, so this was a trivial bot-detection
tell (issue #303).
Track trusted events in a closure-private WeakSet that page JS cannot read or
forge; default isTrusted to false. The CDP input pipeline marks its synthesized
mouse/key events trusted via a non-enumerable __obscura_markTrusted helper, so
automation clicks stay trusted like real Chrome over CDP while page-built
events report false.
|
||
|
Gabriel
|
1a3888c056 |
js: mark Event constructor as native for stealth compatibility
Mark the Event constructor as native so that Event.toString() returns "[native code]" instead of leaking the JavaScript implementation. This closes a stealth fingerprinting vector. |
||
|
Gabriel
|
d08a1deb11 |
Add Network.getResponseBody support
Add Network.getResponseBody support |
||
|
SGavrl
|
b27318dc78 | test: merge pr-300 | ||
|
Gabriel
|
af2f89a03e |
js: implement Element.getAttributeNames()
The method was missing, so calling it threw TypeError: ... is not a function, which broke Vue 3/Nuxt client hydration (Closes #297). Back it with the existing attribute_names op, the same source as the attributes getter, so the order matches attributes and outerHTML. |
||
|
SGavrl
|
09fe30ec39 |
js: implement Element.getAttributeNames()
The method was missing, so calling it threw TypeError: ... is not a function, which broke Vue 3/Nuxt client hydration (issue #297). Back it with the existing attribute_names op, the same source as the attributes getter, so the order matches attributes and outerHTML. |
||
|
SGavrl
|
5a0cbd1260 |
deps: drop unused reqwest cookies feature
cargo install --features stealth failed on Windows with E0119 in cookie 0.18.1: its blanket From<T: Into<Option>> impl fails the coherence check on some toolchains (issue #295). obscura never uses reqwest's cookie store; it manages cookies through its own CookieJar and sets the Cookie/Set-Cookie headers directly. The cookies feature only pulled in the cookie/cookie_store crates (and the whole time subtree). Drop it. Removes 12 packages including cookie, cookie_store, and time, so the failing crate is no longer compiled in any build. No behavior change: the cookie jar unit tests and the obstacle-course cookies stage still pass. |
||
|
SGavrl
|
dd49cefa71 |
deps: drop unused reqwest cookies feature
cargo install --features stealth failed on Windows with E0119 in cookie 0.18.1: its blanket From<T: Into<Option<OffsetDateTime>>> impl fails the coherence check on some toolchains (issue #295). obscura never uses reqwest's cookie store; it manages cookies through its own CookieJar and sets the Cookie/Set-Cookie headers directly. The cookies feature only pulled in the cookie/cookie_store crates (and the whole time subtree). Drop it. Removes 12 packages including cookie, cookie_store, and time, so the failing crate is no longer compiled in any build. No behavior change: the cookie jar unit tests and the obstacle-course cookies stage still pass. |
||
|
SGavrl
|
b8696a47ec |
release: build x86_64-macos on a native Intel runner
The x86_64-apple-darwin release crashed at V8 isolate init with SIGSEGV in ReadOnly snapshot deserialization (issue #290). obscura-js bakes a V8 startup snapshot at build time via a build script, which runs on the host arch. The job built x86_64 on macos-latest (Apple Silicon), so an arm64 snapshot got embedded into the Intel binary and V8 crashed deserializing it. Build on macos-15-intel so the snapshot matches the target. Drop the aarch64-linux-cross artifact: it has the same defect (x86_64-host snapshot in an arm64 binary) and the native ARM runner already covers it. Add a post-build smoke test that runs the V8 path on a data: URL so a release can never ship a binary that crashes at isolate init again. Closes #290 |
||
|
SGavrl
|
c3c0a6a814 |
release: build x86_64-macos on a native Intel runner
The x86_64-apple-darwin release crashed at V8 isolate init with SIGSEGV in ReadOnly snapshot deserialization (issue #290). obscura-js bakes a V8 startup snapshot at build time via a build script, which runs on the host arch. The job built x86_64 on macos-latest (Apple Silicon), so an arm64 snapshot got embedded into the Intel binary and V8 crashed deserializing it. Build on macos-15-intel so the snapshot matches the target. Drop the aarch64-linux-cross artifact: it has the same defect (x86_64-host snapshot in an arm64 binary) and the native ARM runner already covers it. Add a post-build smoke test that runs the V8 path on a data: URL so a release can never ship a binary that crashes at isolate init again. |
||
|
SGavrl
|
cd889d5659 |
Fix/stealth consistency
hasIframeProxy fix contentWindow now returns null when parentNode === null (iframe never inserted into any tree). The previous attempt used isConnected, which broke getBehemothIframe by returning null for iframes inside the phantom iframe's detached document. That caused PHANTOM_DARKNESS to fall back to self (main window), making all DOM measurement ops run on the main document instead of the isolated iframe body, which corrupted the fingerprint element lookup. Worker auto-execution Worker constructor now resolves relative URLs against location.href, fetches the script, and auto-runs it. The worker scope exposes WorkerGlobalScope so IS_WORKER_SCOPE resolves correctly in creepjs. All three worker types now complete within the 3s timeout instead of silently timing out. AudioBuffer + AudioContext AudioBuffer class added to globalThis with getChannelData, copyFromChannel, copyToChannel. AudioContext.createBuffer returns a real instance. AudioContext.addEventListener was a no-op; startRendering now dispatches to stored listeners so the complete event fires. Crypto Crypto class added to globalThis with randomUUID and getRandomValues on its prototype. Fixes ReferenceError in getPlatformEstimate. NetworkInformation + ContentIndex NetworkInformation class with downlinkMax on the prototype clears noDownlinkMax. ContentIndex stub on globalThis clears noContentIndex. navigator.connection is now a real NetworkInformation instance. getElementById / querySelector stale index fix id_index only registers nodes at creation time. When replaceChild(fragment, el) swapped in new nodes their IDs hit .or_insert() and were skipped, leaving the index pointing at the detached original. getElementById now verifies the indexed node is in the live document and falls back to a full attribute-selector scan. query_selector_from falls through to the full scan on both index miss and stale entry instead of returning None early. matchMedia Handles common queries instead of always returning false. prefers-color-scheme: dark returns true (headless Chrome defaults to light mode, so returning true for light is a bot signal). any-pointer: fine, hover: hover, prefers-reduced-motion: no-preference, min/max-width, and (color) return sensible values. Fingerprint consistency jsHeapSizeLimit raised to 4294705152 (4 GB, matching 64-bit Chrome). getHighEntropyValues platform corrected from "Linux" to "Windows", platformVersion from "6.1.0" to "10.0.0" to match the Windows NT 10.0 UA string. Canvas: real PNG encoder, arc fill, multiply blend mode. Screen class, performance.now, chrome runtime errors fixed. Build note deviceMemory and hardwareConcurrency use the high-value pools ([4, 8] GB and [4, 6, 8, 12, 16] cores) only when built with --features stealth. |
||
|
SGavrl
|
4f722dcfd3 |
stealth: fix fingerprint consistency, matchMedia, jsHeapSizeLimit
jsHeapSizeLimit raised to 4294705152 (~4 GB) to match 64-bit Chrome. Was 2172649472 (~2 GB). getHighEntropyValues platform corrected to "Windows" and platformVersion to "10.0.0" to match the Windows NT 10.0 user agent string. Was "Linux" / "6.1.0", which mismatched the UA and could be detected. matchMedia now handles common queries rather than always returning false: prefers-color-scheme:dark returns true (avoiding prefersLightColor headless flag — headless Chrome defaults to light so true is a bot signal), any-pointer:fine, hover:hover, prefers-reduced-motion:no-preference, min/max-width, and (color) return sensible values. Result: 0% like-headless, 0% headless, 0% stealth, 33/33 obstacle course. Note: build must include --features stealth for deviceMemory and hardwareConcurrency to use the high-value pools ([4,8] and [4,6,8,12,16]). |
||
|
SGavrl
|
4e8960e323 |
stealth: add NetworkInformation and ContentIndex shims
NetworkInformation class with downlinkMax on its prototype clears
noDownlinkMax from the like-headless flag list. creepjs checks
'downlinkMax' in (window.NetworkInformation?.prototype || {}).
ContentIndex stub on globalThis clears noContentIndex. creepjs checks
'ContentIndex' in window.
navigator.connection is now a real NetworkInformation instance instead
of a plain object literal.
Result: 0% like-headless, 0% headless, 0% stealth on creepjs. 33/33
obstacle course unchanged.
|
||
|
SGavrl
|
93b2fba6a1 |
fix: creepjs stealth, AudioBuffer, Crypto, Worker, hasIframeProxy
Worker auto-execution: Worker constructor resolves relative URLs against
location.href, fetches the script, and auto-runs it. WorkerGlobalScope is
exposed so IS_WORKER_SCOPE resolves correctly. Worker data (navigator,
locale, timezone) now arrives via postMessage within the 3s timeout.
AudioBuffer class added to globalThis with getChannelData, copyFromChannel,
copyToChannel. AudioContext.createBuffer returns a real AudioBuffer. Fixes
ReferenceError that blocked the audio fingerprint section.
Crypto class added to globalThis with randomUUID and getRandomValues on its
prototype. Fixes ReferenceError in getPlatformEstimate.
AudioContext.addEventListener was a no-op; startRendering only fired
oncomplete. Add _listeners map and dispatch to stored listeners.
getElementById and querySelector('#id') returned null for elements inserted
via replaceChild(documentFragment, el). id_index only registers nodes at
creation time; .or_insert skipped IDs on cloned fragments; None caused
query_selector_from to short-circuit the full scan.
Fix querySelector: fall through to full scan on index miss or stale entry.
Fix getElementById: verify node is in live document; fall back to attribute
selector scan if not.
contentWindow getter: return null when parentNode is null (iframe was never
inserted into any tree). Previously returning null for all disconnected
iframes broke getBehemothIframe, which caused PHANTOM_DARKNESS to fall back
to self (main window) instead of the phantom iframe. That made DOC point to
the main document, so rect and font measurements wrote fragment nodes into
the main body, and the subsequent full-DOM scan in getElementById could not
find fingerprint-data through nested fragments in time.
Limiting the null return to parentNode===null (never inserted) means
getBehemothIframe gets a valid nested contentWindow as expected. creepjs
now reports hasIframeProxy: false and 0% stealth detection.
DocumentFragment constructor: pass nid through when provided.
Range.getBoundingClientRect / getClientRects: return commonAncestor rect.
|
||
|
SGavrl
|
f58bb45956 |
fix(stealth): synchronize TLS and JS fingerprints to Chrome 145
Upgraded wreq to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension order, ALPS, GREASE). Introduced __obscura_stealth global to isolate stealth-specific JS behaviors from the standard execution path. Overrode navigator.userAgentData and getHighEntropyValues() in stealth mode to perfectly match the sec-ch-ua network headers. Restricted hardwareConcurrency and deviceMemory pools in stealth mode to desktop-realistic values (e.g., minimum 4 cores, 4GB RAM) to eliminate mobile-tier leaks. Fixed usedJSHeapSize occasionally exceeding totalJSHeapSize. |
||
|
SGavrl
|
56ab54e615 |
stealth: gate JS behaviours, upgrade wreq, fix fp consistency
wreq upgraded to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension order, ALPS, GREASE). __obscura_stealth global variable introduced and passed from Rust to JS, allowing stealth-specific JS behaviors to run only when the stealth flag is enabled. Tightened hardwareConcurrency and deviceMemory pools in stealth mode to desktop-realistic values (e.g. min 4GB RAM, 4 cores), eliminating mobile-tier values that leak the spoof. navigator.userAgentData and getHighEntropyValues() are now overridden in stealth mode to perfectly match the sec-ch-ua header sent by wreq_util::Profile::Chrome145. Fixed usedJSHeapSize sometimes exceeding totalJSHeapSize. |
||
|
SGavrl
|
dc2c585511 |
stealth: fix audio fingerprint, Screen class, chrome errors, performance.now
OfflineAudioContext.startRendering() now fires oncomplete asynchronously
(microtask) so callers that set ctx.oncomplete after calling startRendering
(fpCollect pattern) get the callback. The rendered buffer contains a
triangle wave scaled to produce the Chrome Linux audio fingerprint sum
(~124.04). createBuffer() now uses persistent Float32Array channels so
written sample values are read back correctly.
screen is now an instance of a Screen class with native getters on the
prototype. Object.getOwnPropertyDescriptor(Object.getPrototypeOf(screen),
"width").get.toString() now returns the expected native-code format instead
of throwing.
chrome.runtime.connect() and sendMessage() now throw the Chrome error
("Could not establish connection. Receiving end does not exist.") instead
of returning silently, matching what fpCollect records for detailChrome.
performance.now() now advances by 0.1ms per call within the same clock
tick, simulating ~100us timer resolution. Previously it aliased Date.now()
which returns the same integer for all calls in one tick, making
redPill2 = Infinity (serialized as null).
WebSocket constructor now throws SyntaxError for non-ws/wss URLs, matching
Chrome's behavior and populating errorsGenerated correctly.
|
||
|
SGavrl
|
b29f0c0eb4 |
canvas: real PNG encoder, arc fill, multiply blend mode
toDataURL() now encodes the real pixel buffer as a valid PNG using
stored-block DEFLATE, CRC32, and Adler32. The 106-byte stub it replaced
was a static value that any canvas fingerprinter could trivially flag.
fill() rasterizes arc paths from this._path using a distance-check
circle fill, so arc-based canvas tests produce real pixel data.
_setPixel() handles globalCompositeOperation='multiply' so fpCollect's
colored-circle blend test composites correctly.
Canvas dimensions now read from getAttribute('width'/'height') so
explicitly sized canvases (e.g. 400x200) use their declared dimensions
instead of always defaulting to 300x150.
|
||
|
SGavrl
|
a7a7b0f4d8 |
stealth: add HTMLMediaElement with canPlayType, Navigator constructor
canPlayType was undefined on <audio> and <video> elements. Adds HTMLMediaElement extending Element with canPlayType returning values that match Chrome 145 on Linux (probably/maybe/empty per MIME type). HTMLAudioElement and HTMLVideoElement extend HTMLMediaElement. _elementClassFor routes AUDIO and VIDEO tags to the correct class. Navigator: adds a global Navigator constructor so typeof Navigator !== 'undefined'. Links it into the navigator prototype chain (_wdProto = Object.create(Navigator.prototype)) so navigator instanceof Navigator returns true, matching Chrome. These are two of the signals in server-side fingerprint classifiers (fpCollect et al). |
||
|
SGavrl
|
a59e756da4 |
stealth: add PluginArray class, move webdriver to navigator proto
PluginArray: real browsers expose a native PluginArray type. navigator.plugins was returning a plain Array, so `navigator.plugins instanceof PluginArray` and `navigator.plugins[0].toString()` (which should return [object Plugin]) were failing sannysoft-style checks. Adds a PluginArray constructor whose prototype inherits from Array.prototype, with Symbol.toStringTag='PluginArray'. Each plugin entry now carries Symbol.toStringTag='Plugin' so Object.prototype.toString returns [object Plugin]. webdriver: moved the webdriver getter off the navigator instance and onto a thin prototype so Object.getOwnPropertyDescriptor(navigator, 'webdriver') returns undefined, matching Chrome where the property lives on Navigator.prototype. Getter is marked native. sannysoft "WebDriver (New)" now reports "missing (passed)". |
||
|
SGavrl
|
ac75cf52a6 |
stealth: fix platform/UA mismatch and hide internal globals from Object.keys
navigator.platform returned "Win32" while navigator.userAgent said Linux when
using the stealth client. page.rs now calls set_platform("Linux x86_64", "Linux", "")
alongside set_user_agent when the stealth client is active.
bootstrap.js: pre-declare 71 internal _* globals as enumerable:false before
var runs, so they don't show up in Object.keys(window) on real pages.
|
||
|
SGavrl
|
3eada2404c | docs: document new flags, stealth identity vars, MCP/SSRF hardening, and DOM-agent CDP methods (#288) | ||
|
SGavrl
|
3c92c36862 |
fix(js): console.log(Error) must not trigger Error.prepareStackTrace
deviceandbrowserinfo.com detects CDP by setting Error.prepareStackTrace as a spy, then calling console.log(new Error()). _consoleFn accessed a.stack, which fires prepareStackTrace if set. Fix: save/clear/restore around the .stack access. |
||
|
SGavrl
|
534de77882 |
Add EventTarget methods to navigator.connection and serviceWorker
NetworkInformation (navigator.connection) and ServiceWorkerContainer (navigator.serviceWorker) are EventTargets in real browsers. React 18 and mobile SPAs call addEventListener on them; the stubs lacked it, so the app bundle crashed with 'addEventListener is not a function' (e.g. goofish.com). |
||
|
SGavrl
|
ee86dbf996 | Scope MCP CORS to the allowlist and cache it at startup | ||
|
SGavrl
|
afef6ea5bd | Normalize SameSite on the CDP cookie path | ||
|
SGavrl
|
820643aff4 | Add missing #[test] to cookie save/load roundtrip | ||
|
SGavrl
|
fa4928faa6 | Trim redundant comments | ||
|
SGavrl
|
4a00a6ac2c |
Implement DOM.focus for keyboard input (#283)
CDP clients (browser-use and similar) focus an input via DOM.focus before typing; Input.dispatchKeyEvent then targets document.activeElement. Without a DOM.focus handler the focus call errored, keystrokes landed on nothing, and the field stayed empty. Resolve the node and call its JS focus(), which sets document.activeElement. Completes the type path for DOM-agent frameworks. |
||
|
SGavrl
|
f4cbb78f0b |
Add Target.targetInfoChanged and DOMSnapshot.captureSnapshot for DOM agents (#282)
CDP DOM-agent frameworks (browser-use and similar) could connect but saw a blank page: navigation never refreshed their cached target url/title, and their DOM build aborted with "Unknown domain: DOMSnapshot". - Emit Target.targetInfoChanged after navigation (both CDP Page.navigate and JS-initiated) carrying the post-nav url/title. Strict clients refresh a target only on this event; it also fixes Puppeteer/Playwright page.url() tracking. - Implement DOMSnapshot.captureSnapshot, built from the DOM with synthetic geometry (no paint engine) and plausible computed styles, keyed by the same backendNodeId as DOM.getDocument. Enough for tag/ARIA/accessibility element detection; it runs only when a client requests it, so the fetch path is unaffected. Coordinate clicking still requires a real layout engine. |
||
|
SGavrl
|
f64aca8064 |
Drop unused context field from obscura::Page (#274)
The wrapper Page cloned a second Arc<BrowserContext> the struct never read. The inner obscura_browser::Page already holds the context and keeps it alive, so the field was dead weight. Removes the dead_code warning reported on aarch64 Closes #273 |
||
|
SGavrl
|
82b7a0baa8 |
Make navigator.geolocation coordinates configurable (#272)
The geolocation shim returned a fixed Frankfurt position. With the timezone now configurable, a deployment in another region would report a location that does not match its exit IP or timezone, the mismatch issue #228 describes. Set OBSCURA_GEOLOCATION="lat,lon" to override the reported coordinates; the value is range-checked and falls back to the Frankfurt default when unset or malformed. The existing positional jitter is preserved. |
||
|
SGavrl
|
6754c1caac |
Make AbortSignal a real constructor (prototype, throwIfAborted, events) (#267)
The AbortController/AbortSignal globals were minimal stubs: AbortSignal had no prototype and no throwIfAborted, and AbortController.abort() only set a flag without firing listeners. Pages that read AbortSignal.prototype or call throwIfAborted threw non-fatal TypeErrors. AbortSignal is now a class with a prototype, aborted/reason, throwIfAborted(), working addEventListener/onabort firing, and static abort()/timeout()/any(). AbortController.abort(reason) fires the abort event. Bootstrap-only, no perf impact. |
||
|
SGavrl
|
2e82b6c85f |
Add --quiet to serve (#265)
serve rejected --quiet ("unexpected argument"), so there was no way to
silence the per-page WARN logs (obscura_browser::page script errors) that
flood the console when serving pages with broken scripts. Add the flag to
the Serve command and cover it in is_quiet_command, so it maps to the same
"off" log filter that fetch and scrape already use. Closes #264.
|
||
|
SGavrl
|
260e138318 | Update README.md | ||
|
SGavrl
|
3cce892795 | Sponsor assets | ||
|
SGavrl
|
b25cc8fc61 |
Cap V8 young generation and optimize for size to cut RSS (#263)
The default V8 flags only set --max-old-space-size (a ceiling, not actual use). The young generation (default 16MB per semi-space) plus V8's memory-heavy codegen inflate RSS during parse and JS allocation. Add --max-semi-space-size=4 and --optimize-for-size to the defaults. Cuts RSS ~10-17% on real pages (ycombinator.com 170MB -> 141MB, react.dev 94MB -> 83MB, wikipedia 58MB -> 53MB) with no measurable speed cost: a JIT-sensitive 8M-iteration loop is unchanged (V8 still optimizes hot paths), and the obstacle course stays 33/33 at the same latency. Users can still override via --v8-flags. |
||
|
SGavrl
|
3fdf0a0be4 |
perf: bound slow ES modules with a DOM-rendered-aware budget
A slow non-essential ES module could block navigation for ~10s. On real pages (www.ycombinator.com) obscura spent ~11.5s loading YC's bookface module, whose top-level eval idle-waits forever, until the phase watchdog killed it. The page renders fully without it (CPU was ~3s, the rest pure idle), but wall time was ~2.5x lightpanda. load_module / load_inline_module now take a per-module budget and bound both the import-graph fetch and the eval loop with it. page.rs sizes the budget by whether the body is already rendered: an enhancement module on a populated page (>50 body descendants) gets a short budget (3s, override OBSCURA_MODULE_BUDGET_MS); an unmounted SPA shell keeps the full script budget so the app module still mounts (#205). Fetch concurrency is unchanged; only the idle eval wait is bounded. www.ycombinator.com: wall 13.1s -> 6.6s, still 475 links, CPU unchanged. teragrowth.org (Vite SPA) still renders. Obstacle course 33/33 including the es-modules / ssr-hydrate / spa stages. Post-drop eval works. |
||
|
SGavrl
|
fb2841b7ab |
fix: CJK browser_search panic and multi-tab tab_close crash (#257, #258)
* Fix UTF-8 char boundary panic in browser_search The search snippet window sliced body[..start] / body[end..] at byte offsets derived from char counts and needle.len(), which can land inside a multi-byte (CJK) character and panic, killing the MCP stdio transport. Snap start/end to char boundaries before slicing and advance past whitespace by the actual char width. Closes #257. * Add setLineDash and friends to the canvas 2D mock Charting libraries (Highcharts, ECharts) call setLineDash/getLineDash/ ellipse every animation frame. The missing setLineDash threw 'is not a function' from a timer each tick, spamming the console. Adds the line-dash methods plus ellipse/roundRect/createConicGradient/ getContextAttributes stubs. Addresses the error spam in #258; the multi-tab isolate-teardown crash is a separate deeper issue. * Fix multi-tab tab_close V8 isolate abort rusty_v8 enters each isolate on creation and requires them dropped in reverse order of creation. The MCP gave each tab its own isolate and dropped them in arbitrary (user-chosen) order on tab_close, so closing a non-newest tab aborted the process (Check failed: heap->isolate() == Isolate::TryGetCurrent()). Pages with active timers (Highcharts/ECharts) made it easy to hit. Enforce the CDP server's single-live-isolate discipline in the MCP: at most one tab's isolate is ever live. page_mut calls a new activate() that suspends every other tab (drops its isolate, keeps its DOM) and resumes the active one. close_tab drops the closed tab's isolate via suspend_js while it is still the only entered isolate, and tool_close suspends all tabs before clearing. Confined to obscura-mcp; the engine, CDP server and obstacle course are untouched, and peak memory drops to one isolate instead of N. Closes #258. |
||
|
SGavrl
|
c1e0b0fb5a |
fix: resolve element .src to an absolute URL (#255) (#260)
HTMLScriptElement/HTMLImageElement .src reflected the literal attribute instead of the resolved absolute URL. Loaders that derive their base with new URL(document.currentScript.src).origin threw "Invalid URL" on a relative src and aborted, so the page bundle never loaded and the DOM stayed empty (issue #255). getAttribute("src") still returns the literal. |
||
|
SGavrl
|
5135e3cb4d |
perf: DOM read fast paths (constant nodeType, borrowed op reads, base64 table) (#259)
* DOM read fast paths: constant nodeType, fewer ops and allocations - Element.nodeType is now a constant getter instead of inheriting Node's dynamic op-backed getter. nodeType is invariant (1) for element wrappers (wrappers are only built for element nodes and node ids are never freed and reused), so this drops one JS/Rust op per read on a pervasive path. Micro-bench: ~600k nodeType reads go from ~380ms to ~5ms. - attributes getter reads each attribute once instead of twice. - dataset Proxy is cached per element instead of reallocated on every access, which also makes el.dataset === el.dataset (spec-correct). - stylesheet <link> scan borrows the node (with_node) instead of deep cloning it and compares rel via eq_ignore_ascii_case, dropping a clone and a lowercased String alloc per link per navigation. No regression: obstacle course 33/33, WPT dom/nodes Node- subset identical to baseline (3551/4163 subtests), targeted DOM probe identical. * DOM op reads: borrow the node instead of deep-cloning it The scalar read arms in op_dom (node_type, node_name, tag_name, get_attribute, attribute_names, parent/first/last/next/prev lookups, has_child_nodes, pi_target, doctype_name, doctype_public_id) and the node_root / node_ancestors / node_child_index walks called get_node, which clones the whole Node (for elements a deep clone of the attribute Vec and its Strings) just to read one field. Switch them to with_node, which borrows under the RefCell. element_children and the document_element / document_doctype child scans stay on get_node on purpose (they iterate, so the borrow story is left untouched). Output is identical. Traversal/attribute micro-bench ~25-30% faster with fewer allocations per op. No regression: obstacle course 33/33, WPT dom/nodes identical to baseline (7057/10620 across 314 files). * base64 decode: reverse-lookup table instead of indexOf scan _base64ToUint8Array decodes every fetch/XHR response body. It ran an O(64) alphabet.indexOf plus a one-char substring alloc per character. Build a charCode->6-bit reverse table once at module load and index it by charCodeAt, folding out-of-range (NaN or code >= 128) to -1 so the bitmath stays byte-identical to the old indexOf path on valid and malformed input alike. ~20-25% faster decoding a 2MB body; output byte-identical (differential test across valid, unpadded, '='-position, >127, non-alphabet and truncated inputs matches the old code exactly). No regression: obstacle course 33/33. |
||
|
SGavrl
|
24cf7cad56 |
Give relList and sandbox their supported tokens (#256)
DOMTokenList.supports() threw "DOMTokenList has no supported tokens"
for relList and sandbox because they were created without a supported
set. Vite's modulepreload polyfill runs link.relList.supports('modulepreload')
at the top of every bundle, so the throw aborted the whole ES module and
React/Vite SPAs rendered blank. relList and sandbox now carry their token
sets; classList still has none and correctly throws.
|
||
|
SGavrl
|
a727fe82ad |
Add GitBook-synced docs under docs/
Mirror the wiki content into a docs/ tree so GitBook Git Sync can connect to this repo. Adds SUMMARY.md for the table of contents, .gitbook.yaml pointing GitBook at docs/, and converts wiki page links to relative .md paths. |
||
|
SGavrl
|
ce67008072 |
fix(js): time out scripted fetch/XHR so a hung request fires error, not nothing (#251) (#252)
build_request_client (the client behind every JS fetch(), XHR, and ES-module load) set no request timeout. A cross-origin XHR with a non-simple header triggers a CORS preflight OPTIONS; if the server accepts the connection but never answers it, op_fetch_url's send().await hung forever, the fetch promise never settled, and the XHR was stuck at readyState 1 with no completion event ever firing. That stranded Angular HttpClient on live.deutsche-boerse.com. Give the client a 30s request timeout (matching the other clients in the workspace); OBSCURA_FETCH_TIMEOUT_MS overrides it. On timeout the request errors, op_fetch_url propagates it, and the fetch shim turns it into an XHR error/loadend, so readyState advances to 4 and Angular's error path runs. Verified: a server that hangs on OPTIONS now makes the XHR fire error at the timeout (was stuck forever); a CORS-allowed cross-origin XHR still succeeds with load/200; obstacle course 33/33. Closes #251 |
||
|
SGavrl
|
1a0644f7bc |
feat(cdp): bound every CDP command with a shared V8 watchdog (#250)
A hung page (a runaway Runtime.evaluate, a synchronous DOM op) holds the process-wide V8 lock and wedges every other session forever. The one-shot CLI has a process-level hard deadline for this; the long-running server cannot force-exit, so terminate just the offending isolate instead. dispatch() arms a watchdog on the session page's isolate after taking the V8 lock and disarms it after the handler; on overrun it terminates the isolate and clears the termination flag so the session keeps working. OBSCURA_CDP_COMMAND_TIMEOUT_MS tunes the bound (default 60s, 0 disables); navigation already self-bounds well under it. The watchdog is a single long-lived shared thread (cdp_watchdog), not a thread per command: the V8 lock serializes commands so at most one is ever armed, and arm/disarm are a mutex plus notify with no measurable per-command cost. Also fixes a lost-wakeup in the per-call watchdog (spawn_watchdog) where disarming a fast call before the watchdog thread started blocked for the full budget. Verified: a hung Runtime.evaluate is terminated at the budget and the session recovers; CDP integration 37/37, obstacle course 33/33; per-command latency unchanged (watchdog on equals off within noise). |
||
|
SGavrl
|
32ddf93885 |
Fix/engine hardening (#249)
* fix(dom): read insert_before prev_sibling after detach to avoid a sibling cycle insert_before captured the reference node's prev_sibling BEFORE detaching the node being inserted. When page JS inserts a node before its own immediate previous sibling (a no-op reorder frameworks do constantly), detach moves that pointer, so the stale prev_id spliced new.next_sibling = new. The resulting sibling self-cycle made descendants() inner next_sibling walk loop forever in native Rust, uninterruptible by tokio or the V8 watchdog. This hung ebay.com, found by a 1500-page crash/hang sweep. Read prev_sibling after detach so it reflects the post-detach tree. Cap the inner sibling-collection loops in descendants() and log when a descendants cap fires, as defense in depth so a sibling cycle from any path cannot hang the engine. Add test_insert_before_previous_sibling_no_cycle. ebay.com now renders (192K chars) instead of hanging. Verified: 33/33 obscura-dom tests, obstacle course 33/33. * fix(cli): settle async --eval work before reading the page (#248) --eval ran after the selector check and returned its value immediately, so an expression that only kicks off async work (e.g. fetch().then(t => DOM write)) returned the still-pending Promise as {} and any --selector/--dump were ignored. When --eval is combined with --selector and/or --dump, run the eval to kick off its async work, settle the event loop again so it completes, then read the page (selector wait + dump) instead of returning the Promise. A bare --eval still returns its value unchanged, so synchronous expressions are not affected, and the eval stays bounded by evaluate_with_timeout. dump becomes Option<DumpFormat> so an explicit --dump is distinguishable from the default. Verified: the repro now returns the fetched content, a plain JSON.stringify eval is unchanged, 37 + 16 cli tests pass. |
||
|
SGavrl
|
6dea35bd0c |
docs: link the benchmark repo (#240)
Add a pointer to the benchmark repo (h4ckf0r0day/obscura-benchmark) in the benchmark section. |
||
|
SGavrl
|
1e9e9478fb |
Fix/v8 watchdog panic safety (#239)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
* conformance: HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser, fetch UA
Next WPT Core batch, all verified live and benchmarked for no hot-path regression.
DOM:
- compareDocumentPosition rewritten to spec: real PRECEDING/FOLLOWING via the
compare_order op, CONTAINS/CONTAINED_BY, and DISCONNECTED|IMPLEMENTATION_SPECIFIC
for different roots. Adds the DOCUMENT_POSITION_* and missing nodeType constants
on Node and its prototype. WPT getPosition() depends on this, so it cascades into
the Range comparePoint/compareBoundaryPoints/isPointInRange expectations.
- HTML elements now ASCII-lowercase attribute names in getAttribute/setAttribute/
removeAttribute/hasAttribute, so el.setAttribute('accessKey') reflects via
getAttribute('accesskey'). The fold is gated behind a cheap uppercase scan and a
namespaceURI HTML guard (SVG/foreign attributes keep their case), and getAttribute
only folds on a miss, so the all-lowercase hot path is unchanged (benchmarked flat).
- Range comparePoint/isPointInRange ToUint32-coerce the offset (negative throws
IndexSizeError); createTreeWalker/createNodeIterator keep an explicit whatToShow of 0.
URL:
- URLSearchParams uses a proper form-urlencoded byte decoder (partial/invalid escapes
stay literal instead of decodeURIComponent throwing on the whole string), 2-arg
has(name,value)/delete(name,value), and a sequence-pair TypeError. Adds URL.parse
and URL.canParse statics.
Other:
- fetch()/XHR send a default Chrome User-Agent (the navigation path set one but
scripted requests went out with none, so UA-gated servers rejected them); honors an
explicit override.
- DOMParser documents expose URL/compatMode/characterSet/readyState/styleSheets/
defaultView and createTreeWalker/createNodeIterator; XMLDocument is defined globally.
* perf(dom): O(1) querySelector('#id') via the id index
querySelector('#id') went through the full selector engine, scanning descendants
in tree order until the first match (O(n), ~68us per call over a 2000-element
document in the micro-bench). A bare ASCII id selector now resolves through the
existing id index in O(1): ~68us -> ~1.25us per call, about 55x faster, with no
change to any other path (create/getAttribute/setAttribute/querySelectorAll/href
all flat). querySelector by id is one of the hottest scraping operations.
The fast path is conservative: only a bare `#name` selector with an ASCII
letter/underscore-led identifier takes it; escapes, combinators, commas,
whitespace, compounds, and non-ASCII fall through to the full engine unchanged.
For a scoped element.querySelector it returns the indexed element only when it is
a strict descendant of the scope, and falls back to the scan otherwise, so
duplicate-id scoped queries stay correct.
Also makes the id index keep the FIRST element in tree order per id (or_insert at
parse time) instead of the last, so getElementById and querySelector('#id') return
the first-in-tree-order element on duplicate ids, per spec (previously last).
* fix: bound synchronous V8 hangs with a termination watchdog and add op panic safety
Heavy pages could hang obscura fetch indefinitely: tokio::time::timeout only
cancels at await points, so synchronous V8 work (a page's script bootstrap, a
microtask storm) pinned the thread and ran unbounded.
Add a reusable V8 terminate_execution watchdog (arm_watchdog/disarm_watchdog) and
apply it to every place that runs synchronous V8 work: the post-load settle, both
navigation event-loop pumps, the whole script-execution phase, and --eval. A CLI
process-level hard deadline backstops the rest, since a hang inside a Rust op
cannot be cancelled by tokio or terminate_execution.
Also harden against panics: op_dom is wrapped in catch_unwind so a DOM-op panic
degrades to a null result instead of unwinding into V8's FFI frame and aborting
the process; the append_text unwrap is removed; panic = unwind is pinned in the
release profile.
Verified: obstacle course 33/33, no perf regression (per-op deltas within the
measurement noise floor), 6 of 8 previously-hanging real sites now render and the
other 2 exit deterministically at the hard deadline.
* fix(dom): reject cyclic reparenting so descendants() cannot hang the engine
appendChild/insertBefore only guarded the self-append case (parent == child),
never the case where the moved node is an ancestor of the new parent. When page
JS does node.appendChild(ancestorOfNode) the parent/child graph becomes cyclic;
the next querySelector runs descendants() (a DFS with no visited set) and loops
forever, growing a Vec unbounded and pinning the thread in native Rust where
neither tokio nor the V8 watchdog can interrupt it. This hung techcrunch and
kayak.
Reject the cycle at the mutation site per the DOM HierarchyRequestError rule: if
the moved node is an inclusive ancestor of its target parent, treat the move as a
no-op. The walk is gated on the moved node having children, so a fresh/leaf
append (the common case) stays O(1); only relocating a populated subtree pays
O(depth). descendants() also gets a result-length cap as defense in depth so a
cycle from any future path cannot wedge the engine.
techcrunch and kayak now render (37K and 1.26M chars). Verified: new
test_reparent_cycle_is_rejected, obstacle course 33/33, no perf regression.
|
||
|
SGavrl
|
a703b7ac30 |
WPT conformance + perf: O(1) querySelector('#id'), HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser (#238)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
* conformance: HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser, fetch UA
Next WPT Core batch, all verified live and benchmarked for no hot-path regression.
DOM:
- compareDocumentPosition rewritten to spec: real PRECEDING/FOLLOWING via the
compare_order op, CONTAINS/CONTAINED_BY, and DISCONNECTED|IMPLEMENTATION_SPECIFIC
for different roots. Adds the DOCUMENT_POSITION_* and missing nodeType constants
on Node and its prototype. WPT getPosition() depends on this, so it cascades into
the Range comparePoint/compareBoundaryPoints/isPointInRange expectations.
- HTML elements now ASCII-lowercase attribute names in getAttribute/setAttribute/
removeAttribute/hasAttribute, so el.setAttribute('accessKey') reflects via
getAttribute('accesskey'). The fold is gated behind a cheap uppercase scan and a
namespaceURI HTML guard (SVG/foreign attributes keep their case), and getAttribute
only folds on a miss, so the all-lowercase hot path is unchanged (benchmarked flat).
- Range comparePoint/isPointInRange ToUint32-coerce the offset (negative throws
IndexSizeError); createTreeWalker/createNodeIterator keep an explicit whatToShow of 0.
URL:
- URLSearchParams uses a proper form-urlencoded byte decoder (partial/invalid escapes
stay literal instead of decodeURIComponent throwing on the whole string), 2-arg
has(name,value)/delete(name,value), and a sequence-pair TypeError. Adds URL.parse
and URL.canParse statics.
Other:
- fetch()/XHR send a default Chrome User-Agent (the navigation path set one but
scripted requests went out with none, so UA-gated servers rejected them); honors an
explicit override.
- DOMParser documents expose URL/compatMode/characterSet/readyState/styleSheets/
defaultView and createTreeWalker/createNodeIterator; XMLDocument is defined globally.
* perf(dom): O(1) querySelector('#id') via the id index
querySelector('#id') went through the full selector engine, scanning descendants
in tree order until the first match (O(n), ~68us per call over a 2000-element
document in the micro-bench). A bare ASCII id selector now resolves through the
existing id index in O(1): ~68us -> ~1.25us per call, about 55x faster, with no
change to any other path (create/getAttribute/setAttribute/querySelectorAll/href
all flat). querySelector by id is one of the hottest scraping operations.
The fast path is conservative: only a bare `#name` selector with an ASCII
letter/underscore-led identifier takes it; escapes, combinators, commas,
whitespace, compounds, and non-ASCII fall through to the full engine unchanged.
For a scoped element.querySelector it returns the indexed element only when it is
a strict descendant of the scope, and falls back to the scan otherwise, so
duplicate-id scoped queries stay correct.
Also makes the id index keep the FIRST element in tree order per id (or_insert at
parse time) instead of the last, so getElementById and querySelector('#id') return
the first-in-tree-order element on duplicate ids, per spec (previously last).
|
||
|
SGavrl
|
3b2c6c0deb |
WPT conformance: legacy-charset URL encoding, IDL attribute reflection, FileAPI, Storage, Range (#237)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
|
||
|
SGavrl
|
9c3dd6b21a |
WPT conformance: dom/ranges, FileAPI, detached documents (#236)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe createHTMLDocument returned a document whose body was null: parsing the skeleton string as innerHTML of <html> collapsed through the fragment parser and kept only <title>. Build head>title and body explicitly instead. The DOMParser doc node also gains appendChild/removeChild/insertBefore and a doctype getter, and createDocument carries its doctype argument. This lets dom/common.js setupRangeTests() finish (it builds fixtures with new Document(), createHTMLDocument(), createDocument()) instead of aborting at harness setup, so the dom/ranges files run. Range-collapse 186/186, Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17. The large O(ranges x points) matrix files (Range-set, comparePoint, compareBoundaryPoints) still time out: the boundary-point comparison walks the tree through per-step DOM ops, which is too slow for those matrices. Speeding that up (a Rust node-position op) is a separate change. Also add globalThis.ProgressEvent (Event subclass with lengthComputable/ loaded/total) and the static Document.parseHTMLUnsafe(html). * perf(dom): compute Range tree-order in Rust ops The Range boundary-point algorithms compared nodes by walking ancestor chains and sibling lists from JS, one DOM op per step. The dom/ranges matrices (testRanges x testPoints, thousands of comparisons) made that time out. Add node_index, compare_order, and node_root ops that do the walk in Rust over the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex, _rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange 5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s where they previously did not finish. Not on any engine hot path (querySelector, mutation, navigation, fetch do not use these). * fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob Blob was a one-line stub that string-joined its parts (wrong size for binary or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size, slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared when non-ASCII), the endings option (native CRLF normalization), and Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and its own toStringTag. FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws InvalidStateError when a read starts while already LOADING, and fires loadstart/progress/load/loadend (and abort) to both on* handlers and addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct results. Request gains blob(). FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice 141/150. |
||
|
SGavrl
|
387c74f26a |
fix(net): pin exact wreq and wreq-util versions for stealth build (#235)
wreq and wreq-util are pre-release crates whose public API shifts between rc releases. The caret version requirements let a fresh resolve pull a newer rc where CertStore moved out of tls, cert_store was renamed to tls_cert_store, and the Emulation builder changed. That broke cargo build --features stealth (issue #234). Pin to =6.0.0-rc.28 and =3.0.0-rc.10, the versions wreq_client.rs is written against, so the build always resolves to a compatible API. |
||
|
SGavrl
|
07c6e735fa |
WPT conformance: WHATWG URL, DOM Range, and post-navigation event-loop settle (#233)
* fix: restore regressions introduced by #227 #227 deleted build.rs (reverting #224's version-from-tag feature), removed DOM cycle guards in append_child/insert_before that prevent infinite loops on self-append/self-insert, dropped the CharacterData textContent fast-path, removed PI/Doctype Rust ops, and cut the env_allows_private_network() check from validate_fetch_url. Restored: - crates/obscura-cli/build.rs (version from tag/env) - OBSCURA_BUILD_VERSION in version string and banner - allow_private_network env var propagation in main.rs - append_child/insert_before self-cycle guards in tree.rs - textContent CharacterData fast-path in tree.rs - create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops in ops.rs - ProcessingInstruction branch in set_text_content op - env_allows_private_network() check in validate_fetch_url * fix(js): Europe/Berlin timezone, plugins iterator recursion Two bugs in the stealth bootstrap code from #227: - 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat silently fell back to UTC. Changed to 'Europe/Berlin'. - plugins[Symbol.iterator] called this[Symbol.iterator]() recursively, blowing the stack on Array.from(navigator.plugins). Fixed to bind Array.prototype[Symbol.iterator] directly to the array. * url: back URL/URLSearchParams with the url crate, add <a>/<area> decomposition The URL global was a regex shim: it mis-parsed userinfo (hostname came back as the username, port as "pass@host"), had no username/password, and its setters were no-ops. Replace it with two ops (op_url_parse, op_url_set) backed by the WHATWG-compliant url crate, and add the HTMLHyperlinkElementUtils members (protocol/host/hostname/port/pathname/search/hash/origin/username/password and their setters) to <a> and <area>, which had none. This was the single largest WPT failure. a.search returned undefined, so the encoding encode-href tests threw "Cannot read properties of undefined (reading 'substr')" about 225k times. With the fix url-constructor passes 831/888, url-setters 192/279, urlsearchparams-sort 17/17, url/a-element 415/887, and the encode-href files run their assertions instead of crashing. op_url_set runs the url crate inside catch_unwind: a few setters panic on the pathological inputs the url-setters tests use, and a panicking setter must be a no-op (the WHATWG "do nothing on invalid value" rule) rather than abort. URLSearchParams also gained getAll/sort/size/keys/values/entries/iterator and a back-reference so mutations write through to the bound URL. * dom: real Range/StaticRange, createDocumentType, XML Document type Range was a no-op stub (setStart/setEnd discarded their arguments, so startContainer/endContainer read back undefined and every dom/ranges helper that walked range.endContainer crashed with "reading 'nodeType' of undefined"). Replace it with a property-storing Range over the live tree: setStart/End and the *Before/*After/selectNode/selectNodeContents/collapse positioning, plus the boundary-point algorithms comparePoint, compareBoundaryPoints, isPointInRange, intersectsNode, commonAncestorContainer, cloneRange and a real stringifier, all built on a correct preorder tree-order comparison. Add StaticRange. new Document() now reports an XML contentType (it has no backing node id), so createCDATASection no longer throws NotSupportedError on it. Add DOMImplementation.createDocumentType with the lenient name validation browsers use and an owner-document backref so DocumentType.ownerDocument is correct. StaticRange-constructor 14/17. This does not fully unblock dom/ranges yet: dom/common.js also needs createHTMLDocument()/createDocument() to return documents with a working body/createComment/etc., which is a separate fix. * fix(browser,cli): settle the event loop after navigation obscura fetch read the page at the load event and never pumped the event loop afterward on the eval/dump path, so --wait was effectively ignored there. Pages that schedule timers, do async work, or only report through a completion callback that runs after load (testharness add_completion_callback, SPAs) were observed frozen at the load event and never produced their result. Add Page::settle, which drives run_event_loop until the loop goes idle or --wait elapses (returns early when idle, so static pages stay fast), and run it after navigation in the fetch path. |
||
|
SGavrl
|
6b3cc1a686 |
Fix/rescue 227 regressions (#232)
* fix: restore regressions introduced by #227 #227 deleted build.rs (reverting #224's version-from-tag feature), removed DOM cycle guards in append_child/insert_before that prevent infinite loops on self-append/self-insert, dropped the CharacterData textContent fast-path, removed PI/Doctype Rust ops, and cut the env_allows_private_network() check from validate_fetch_url. Restored: - crates/obscura-cli/build.rs (version from tag/env) - OBSCURA_BUILD_VERSION in version string and banner - allow_private_network env var propagation in main.rs - append_child/insert_before self-cycle guards in tree.rs - textContent CharacterData fast-path in tree.rs - create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops in ops.rs - ProcessingInstruction branch in set_text_content op - env_allows_private_network() check in validate_fetch_url * fix(js): Europe/Berlin timezone, plugins iterator recursion Two bugs in the stealth bootstrap code from #227: - 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat silently fell back to UTC. Changed to 'Europe/Berlin'. - plugins[Symbol.iterator] called this[Symbol.iterator]() recursively, blowing the stack on Array.from(navigator.plugins). Fixed to bind Array.prototype[Symbol.iterator] directly to the array. |
||
|
SGavrl
|
990f2a63c8 |
feat(js): popover, legacy events, CDATA/PI, DOMTokenList, namedItem, attachShadow (#231)
Next batch of WPT dom/shadow-dom conformance fixes in bootstrap.js: - Popover API: popover reflection, showPopover/hidePopover/togglePopover, ToggleEvent, :popover-open - Legacy event init: initEvent/initUIEvent/initMouseEvent/initKeyboardEvent/initCompositionEvent, CompositionEvent - Document.createCDATASection / createProcessingInstruction (XML-gated) - classList as a real DOMTokenList (+ relList/sandbox/sizes/htmlFor), spec-correct namespaceURI - getElementsByClassName ASCII-whitespace tokenization - HTMLCollection.namedItem and named access; createRange on document/iframe/DOMParser - attachShadow mode validation and a real ShadowRoot (cloneNode throws, activeElement, styleSheets) querySelectorAll returns a NodeList; named access on live HTMLCollections is resolved lazily via a Proxy so query hot paths stay at native speed. |
||
|
SGavrl
|
dc2a2e3b52 |
feat(js): DOM, namespace, validity, and parsing APIs surfaced by WPT conformance work (#230)
* feat(js): DOMException, ParentNode methods, composedPath, HTMLCollection/NodeList Add the DOMException global (name, message, code, and the legacy code constants on both the constructor and instances). It was referenced by AbortSignal and others but never defined, so any test touching it failed with 'DOMException is not defined'. ParentNode append/prepend/replaceChildren now live on Element, Document, and DocumentFragment and use the spec node conversion: a non-Node argument (including null and undefined) becomes a Text node instead of being dropped or throwing. replaceChildren was missing entirely; append and prepend were Element-only and skipped null/undefined. Add Event.composedPath() and the HTMLCollection and NodeList globals. Verified with fetch --eval; integration suite still 28/28. * feat(js): setHTMLUnsafe/getHTML, namespace lookups, getElementsByTagNameNS, Attr, form validity, Response.blob Element and ShadowRoot setHTMLUnsafe / getHTML as shims over innerHTML. Node lookupNamespaceURI / lookupPrefix / isDefaultNamespace walking the element ancestry and xmlns declarations, with the built-in xml and xmlns namespaces. Element and Document getElementsByTagNameNS with namespace and localName wildcard support. Attr nodes plus Document createAttribute / createAttributeNS and Element getAttributeNode(NS) / setAttributeNode(NS) / removeAttributeNode, bridged to get and set attribute. Form control constraint validation (ValidityState, validity, willValidate, checkValidity, reportValidity, setCustomValidity, validationMessage) and text selection (selectionStart / End / Direction, setSelectionRange, setRangeText, select) as Element.prototype shims. Patch blob / text / json onto the engine Response prototype so fetch(...).blob() works; the real Response prototype lacked blob. Verified with fetch --eval; integration suite still 28/28. |
||
|
SGavrl
|
c8d98845ad |
fix(build): switch reqwest from native-tls-vendored to rustls-tls (#229)
Building with --features stealth linked both wreq's BoringSSL and reqwest's vendored OpenSSL into the same binary, causing undefined symbol errors at link time on Linux. Switching reqwest to rustls-tls removes the vendored OpenSSL dependency. rustls is pure Rust and coexists cleanly with wreq's BoringSSL. Verified: stealth and non-stealth release builds succeed, 28/28 integration tests pass. |
||
|
SGavrl
|
1b9be08767 |
fix(cdp): exit on Ctrl-C while serving (#225) (#226)
The LocalSet accept loop blocked on ws_rx.recv() forever because the detached accept thread kept a ws_tx clone alive, so the channel never closed and start_with_full_serve_options never returned. cdp_processor caught Ctrl-C and saved cookies, but that only ended its own task. Add a shared shutdown flag and Notify. On Ctrl-C cdp_processor sets the flag and notifies; the accept loop selects on the notify and breaks, so run_until returns and the process exits. The accept thread checks the flag between connections so it stops and drops ws_tx on the slow path. Closes #225 |
||
|
SGavrl
|
5e23903063 |
fix: DocumentFragment flattening in appendChild / insertBefore / replaceChild (closes #217) (#221)
Heavy-JS sites build subtrees off-DOM in a DocumentFragment then
appendChild the fragment to a real parent. The DOM spec says: move
the fragment's children up and leave the fragment empty. Our shim
was passing the fragment's _nid straight to the Rust op, so the
fragment node itself got grafted instead of its children, and
framework code that read the inserted nodes' properties hit
"Cannot read properties of undefined (reading 'length')" the moment
it expected a real Element but found our fragment wrapper.
That is what was killing Google Shopping and Bing Shopping for
@Tarun-004 — Bing's anti-malware shim, Google's result-row renderer,
and a sticky-container helper (n.Setup / n.InitializeStickyContainer
in the stack) all use the off-DOM-fragment pattern.
Same defect existed in insertBefore and replaceChild — all three
now flatten DocumentFragment per spec.
While there, three smaller defenses that the same kind of page
exposes:
- _nid validation. Passing a non-Node truthy value (e.g. undefined,
a plain object) used to silently call op_dom("append_child",
parent, undefined), which the Rust side parses with unwrap_or(0)
and grafts as a child of the document root.
- Self-append guard mirrors the insert_before self-cycle fix:
appendChild(self) is a no-op rather than corrupting the
sibling-pointer linked list.
- Tree::append_child gains a parent_id == child_id early return as
defense in depth, parallel to the insert_before guard.
Sanity test in eval mode: DocumentFragment with two children appended
to a host now produces "<span>A</span><span>B</span>" and the
fragment is empty; insertBefore with a fragment containing C produces
"<span>C</span><span>A</span><span>B</span>"; self-append and
appendChild(null) no longer throw.
Verified against the actual reproducer: `obscura fetch --dump text
https://www.bing.com/shop?q=test` no longer produces the
TypeError / "Inline script error" / "Timer error" pattern from
the issue.
|
||
|
SGavrl
|
ee10e8ddb5 |
feat(engine): DOM + AbortSignal + storage fixes surfaced by WPT conformance work (#219)
Engine-only changes pulled out of the WPT conformance branch. The tests, runner, and harness vendoring that drove these have moved to a separate benchmark repo (obscura-bench). Real bug fixes: - Tree::insert_before created a self-cycle when the inserted node was its own reference (the spec-conformant `child.after(x)` path that the testharness's ChildNode helpers exercise). The fixup wrote node.prev_sibling = node.next_sibling = own id; every subsequent childNodes / querySelectorAll walk then looped forever and obscura burned RAM until OOM. Per spec the op is a no-op; we now early-return. - Tree::text_content extended to return own data for direct CharacterData access (Text / Comment / ProcessingInstruction). - Element.namespaceURI now reflects the namespace passed to createElementNS instead of collapsing every non-svg namespace to xhtml. Without this, code that creates elements in MathML, custom XML namespaces, or the empty namespace gets misclassified as HTML. - AbortController / AbortSignal: the previous shim was a plain object whose addEventListener was a no-op, so listeners for 'abort' silently never fired and any AbortSignal.timeout / AbortSignal.any chains broke. New impl tracks aborted/reason, has a real addEventListener / removeEventListener / dispatchEvent surface, honors `once`, and implements the static abort / timeout / any factories crawlers use for budgeted fetches. - StorageEvent class added. localStorage / sessionStorage mutations now dispatch 'storage' events on the parent window and every iframe window so cross-frame onstorage listeners fire. - ProcessingInstruction (nodeType 7) and DocumentType (nodeType 10) constructors plus document.createProcessingInstruction and document.implementation.createDocumentType. textContent returns null for Document and DocumentType per spec. - insertAdjacentText / insertAdjacentElement on Element. Without these the testharness output renderer threw in a forEach without try/catch, breaking the completion callback chain for every test with subtests. - Window + Node addEventListener filter non-callable listeners, accept the EventListenerObject shape (object with handleEvent), dedupe by (type, callback, capture), honor the `once` option, and snapshot the listener list before dispatch so a `once: true` removal mid-iteration doesn't corrupt the loop. - Iframe HTML loader now extracts head <script> blocks and body on-handler attributes (`<body onstorage=...>`), evals scripts in the parent realm after the iframe window is built, re-attaches handlers as window listeners. Same iframe element reusing its src no longer double-registers in the iframe registry. - ops.rs validate_fetch_url honors the OBSCURA_ALLOW_PRIVATE_NETWORK env var, and the CLI --allow-private-network flag mirrors itself into that env so the js-side fetch gate matches the http_client gate. Without this, iframe loads of 127.0.0.1 resources were blocked even with the flag set. - ops.rs: create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops added; set_text_content extended to handle PI nodes. |
||
|
SGavrl
|
fd2421940b |
Perf/cdp fetch cache and cookie alloc (#216)
* perf: cache HTTP clients + selector parsing + exponential wait backoff Three low-risk wins discovered while reading the hot paths. 1. op_fetch_url cached its reqwest::Client. Previously every JS-side fetch(), XHR, and dynamic script load built a fresh client, which re-loaded TLS roots and started with an empty connection pool. On an asset-heavy page with 30+ subresources this added ~150ms of wasted setup with zero connection reuse. The cache is a process- wide map keyed by proxy URL (empty string for no-proxy), so the first fetch on each proxy pays the build cost once and every subsequent fetch reuses the same pool. Also bumped pool_idle_timeout to 5 minutes and tcp_keepalive to 60s so connections stay warm for SPA-heavy workloads that hit the same origin dozens of times across a navigation. 2. DomTree::parse_selector now caches parsed selectors in a thread- local HashMap (cap 256). querySelector / querySelectorAll re- parsed the selector string every call before; batch DOM access patterns (an agent scraping a table row-by-row, a framework polling for elements) paid the cssparser cost N times. 3. browser_wait_for / browser_wait_for_text now exponentially back off (5 -> 10 -> 20 -> ... -> 200 ms). The old fixed 200ms tick meant a selector that appeared 30ms after the call still took the next full 200ms tick to be detected. Worst case is still 200ms per check, but the median wait drops sharply for fast appearances. 28/28 integration tests pass. Asset-heavy bench (rust-lang/rust, docs.stripe.com, etc) doesn't regress. * perf: module loader client cache, fetch concurrency cap, snapshot hide list, cookie alloc Round two of perf wins after a deep code audit. 1. ES module loader (module_loader.rs) was building a fresh reqwest::Client on every dynamic import. Modern SPAs (Next, Vite) load 20-50 chunks via dynamic import; each one used to spin up an empty connection pool with fresh TLS init. Now reuses the process-wide FETCH_CLIENT_CACHE that op_fetch_url already uses (the cached_request_client helper). Saves ~30-100ms on chunk- heavy first navigations. 2. Script and CSS fetches in execute_scripts used futures::join_all with unbounded concurrency. A 100-script page would open 100 TCP sockets at once, exhausting the connection pool and the OS ephemeral port range. Capped at 16 in-flight (well above the per- host pool ceiling real Chrome uses). 3. bootstrap.js __obscura_init() ran Object.keys(globalThis).filter() on every page setup to find internals to hide. The set of keys is static at snapshot-build time, so we now precompute it once (top-level statement just below the function) and __obscura_init just iterates the precomputed array. Was 5-40ms per page on heavy SPAs (1000+ globals); now ~0.2ms per page. 4. CookieJar::domain_matches allocated 2 lowercase Strings and a format! per call. The lookup runs per fetch per cookie domain; on a heavily-cookied site with 100 cookies across 30 domains that was thousands of allocations per page. Now zero-alloc with eq_ignore_ascii_case and char-boundary checks. |
||
|
SGavrl
|
171dba23e7 |
Feat/mcp agent skill tier1 (#215)
* feat(mcp): 10 new agent tools + element refs (12 -> 22 tools)
Brings obscura-mcp from a Playwright-style core (navigate/click/fill/
type/snapshot/wait_for/evaluate/select/press_key/network_requests/
console_messages/close) up to a full agent surface.
New tools:
- browser_markdown: render the page as Markdown (reuses the
HTML_TO_MARKDOWN_JS converter behind `fetch --dump markdown`).
Token-dense alternative to browser_snapshot.
- browser_links: every <a href> with anchor text, JSON-per-line.
Optional internal_only and limit args.
- browser_interactive_elements: list every clickable / typeable
element with a stable ref (e1, e2, ...) and a one-line label.
- browser_back / browser_forward / browser_reload: history nav,
uses the Page history stack from the engine-tier1 commits.
- browser_get_cookies / browser_set_cookie / browser_clear_cookies:
full cookie jar ops so an agent can skip a login flow when it
already has a session token.
- browser_wait_for_text: wait for a substring to appear anywhere
in the visible page text. Complements browser_wait_for which
only checks for selectors.
Element refs (the playwright-mcp pattern, agent UX win):
- BrowserState now keeps an interactive_refs HashMap built on each
browser_snapshot / browser_interactive_elements call.
- Each interactive element gets a data-obscura-ref="eN" attribute
written into the DOM so refs survive across calls.
- browser_click / browser_fill / browser_type accept `ref` (preferred)
or `selector` (fallback). Removes the agent's 'guess a CSS
selector then retry' loop.
- Refs are cleared on every navigation / reload / back / forward / close.
Output size limits:
- browser_snapshot, browser_markdown accept max_chars (default 4000).
- browser_links, browser_interactive_elements accept limit (default 100).
- Truncated responses say '...(truncated, N more chars)' so the agent
can ask for more if needed.
Verified end-to-end via /tmp/mcp-test.sh: tools/list returns 22 tools,
ref-based click on a real element works, markdown converts H1+P+UL
correctly, cookies round-trip. 28/28 integration tests pass.
* feat(mcp): 10 more agent tools, multi-tab, structured extract (22 -> 32 tools)
Tier 2: closes the gaps in agentic flows the Tier 1 surface still
needed multi-call workarounds for (login forms, paginated lists,
multi-step research across multiple destinations).
Multi-tab:
- BrowserState now holds a BTreeMap of tabs (Page per tab) with an
active_tab cursor. page_mut() auto-creates a default tab so every
legacy single-tab tool keeps working.
- browser_tab_new: open and optionally navigate.
- browser_tab_list: list all open tabs with active marker.
- browser_tab_switch: change which tab subsequent tools operate on.
- browser_tab_close: close (default: active tab), promotes another.
Form ergonomics:
- browser_detect_forms: returns each <form> with action, method, and
every field's tag, type, name, current value, label (from <label for>
or aria-label or placeholder), and option list for selects. Agents
call this once to see the whole form shape instead of guessing
selectors field by field.
- browser_fill_form: fill N inputs in one call. Each entry chooses
type='text' (default value-set), 'check'/'uncheck', or 'select'
(matches by value OR visible text). Optional submit_ref clicks
after filling.
Structured extraction:
- browser_extract: pass {field: 'selector'} and get a JSON object back.
Selector suffix '@attr' reads attribute (e.g. 'a@href'). Field
suffix '[]' returns an array (queries all matches). One call returns
the whole structured object instead of an evaluate per field.
- browser_count: existence / pagination probe.
- browser_get_attribute: read href / src / value / data-* etc.
Page control:
- browser_scroll: 'top'|'bottom'|'up'|'down'|'left'|'right' with
optional amount, or scroll an element into view. Dispatches a real
'scroll' event on window AND document so infinite-scroll handlers
fire even though our viewport isn't a real layout.
Other:
- Bumped #![recursion_limit = "512"] because the tools/list JSON
literal is now too big for the default macro recursion limit.
- browser_count: coerce f64 -> u64 since V8 numbers come back as floats.
Verified end-to-end: 32 tools registered, multi-tab open/list/switch/
close all work, detect_forms returns proper JSON, fill_form fills
text/check/select in one call, extract returns the structured object
with arrays and attribute reads. 28/28 integration tests pass.
* feat(mcp): search + storage state export/restore (32 -> 35 tools)
Three additions that close the long-tail agentic flows:
- browser_search: substring search in visible page text, returns each
match with N chars of surrounding context (default 80). Lets an
agent confirm content exists before scraping and locate a section
without pulling the full page into its window. case_sensitive +
limit + context_chars args.
- browser_storage_state: export the full auth surface (cookies +
localStorage + sessionStorage for the current origin) as one JSON
object. Agents stash this between runs to skip a login flow.
- browser_set_storage_state: apply a previously-exported state. Sets
cookies via the cookie jar; writes localStorage / sessionStorage
via evaluate on the current page (so a navigate to the right origin
must come first).
Tool count: 35. 28/28 integration tests pass.
|
||
|
SGavrl
|
89692f3cb7 |
Perf/cdp network and lockfree (#214)
* perf(cdp): TCP_NODELAY + zero-buffer WS writes shave 130ms off newPage CDP traffic during browser.newPage() and Page.navigate is many small (~100-byte) request/response frames. Two pieces of buffering were adding ~10-15ms latency per WS frame on localhost, which on the ~10 round-trips Puppeteer makes during newPage compounded to a ~130ms penalty. 1. TCP_NODELAY on every accepted WS stream. With Nagle on, each small write either waits for an ACK from the previous one or hits the 40ms delayed-ACK timer, even on localhost. 2. WebSocketConfig.write_buffer_size = 0 on accept_async. tokio- tungstenite's default 128 KiB write BufWriter copies every frame into the buffer before flushing; setting it to 0 makes every small CDP reply hit the socket directly. Measured impact on localhost (puppeteer-core 24, lower is better): operation before after speedup browser.newPage() ~190ms ~50ms ~4x goto(data: URL) ~60ms ~22ms ~3x goto(repeat) ~60ms ~22ms ~3x Shared-page CDP scrape bench (9 scenarios): ~4000ms -> ~2900ms. 28/28 integration tests pass. * perf(cdp): bypass v8_lock for read-only / setup CDP methods Every CDP request unconditionally acquired the process-wide V8 lock, which serializes all CDP traffic behind any in-flight V8 work on a sibling page. During Puppeteer's browser.newPage() the ~8 setup calls (Page.enable, Runtime.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, etc.) all queued behind a concurrent nav, turning a sub-millisecond setup into a multi-hundred-millisecond wait. is_v8_free_method now allowlists methods that demonstrably never touch a JsRuntime, either they don't touch any Page at all, or they use only the immutable get_session_page accessor and Rust-side field reads. get_session_page_mut (which can trigger suspend_js/resume_js on other pages) stays behind the lock, as does Target.createTarget (which calls page.navigate). Allowlisted: most of Target/Browser/Storage, plus Page.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, Runtime.enable, Network.* cookie ops, Fetch.continue/fulfill/fail. Measured on the puppeteer-core 24 CDP scrape bench: browser.newPage() warm: ~50ms -> 13-32ms goto() repeat: ~22ms -> 14-48ms (variance) 28/28 integration tests pass. 18/18 realistic scrape scenarios pass. |
||
|
SGavrl
|
282b4646d3 |
perf(js): resolve_promises_until exits as soon as the awaited IIFE finishes (#213)
Every Puppeteer click ran isIntersectingViewport (an actionability check that wraps IntersectionObserver in a promise). Our IO.observe schedules re-fires at 120/500/1500/3500/7000ms to cover infinite-scroll sentinels. Puppeteer's callback disconnects the observer on the first fire, so the re-fires are no-ops, but they keep the deno_core event loop 'not idle' until they all fire. resolve_promises was waiting for the loop to go idle (5s hard cap), so every awaitPromise call paid the full 5s even though the result was ready in <1ms. Effect: page.click() took ~7s instead of ~150ms because clickableBox + isIntersectingViewport each hit the cap. Fix: each evaluate / callFunctionOn now writes a unique sentinel (globalThis.__obscura_done_<counter> = true) in a finally block. The new resolve_promises_until polls that sentinel with a 1ms / 2ms / ... backoff up to 50ms and exits as soon as the IIFE is done, instead of waiting for unrelated timers to drain. Worst-case miss is one tick (50ms). Measured impact (Puppeteer CDP scenarios, lower is better): scenario before after data: click button 7420ms ~530ms (14x) data: addEvt click 14536ms ~680ms (21x) data: form submit 7676ms ~810ms (9x) CDP bench total 35190ms ~7400ms (~5x) 28/28 integration tests still pass. |
||
|
SGavrl
|
50462d41a9 |
fix(js): load_module actually fetches the module source (#205) (#212)
load_module registered an empty string as the module body and called it loaded. Every Vite / Next / SvelteKit external module bundle 'loaded' in 1ms with zero code, so the framework never executed and the SPA shell sat empty (#root.children.length == 0). Now fetches the URL through the page's http_client and passes the real source. getvix.dev/docs (Vite + React) goes from rootChildren=0 / bodyTextLength=10 (SSR shell only) to rootChildren=4 / bodyTextLength=3708 (React app mounted). Closes #205. |
||
|
SGavrl
|
3c3798b197 |
fix(js): HTMLMetaElement.content is read/write (#210) (#211)
`<meta>.content` was a getter-only stub. `next/head` and `next/document` iterate <meta> tags and assign `.content` during hydration, which threw `Cannot set property content`, React caught and retried in a tight loop, the page never mounted, and the error spammed the console thousands of times. Traveloka and other Next.js sites doing meta management were unusable. Now reflects the content attribute with both get and set, per spec. Template's `.content` (DocumentFragment) is unchanged. Closes #210. |
||
|
SGavrl
|
326096e20f |
Engine tier1 compat fixes (#208)
* engine: document.currentScript, data: subresources, dynamic module scripts
document.currentScript:
- ScriptInfo now carries the script element's nid.
- Before each script body runs, set globalThis.__currentScriptNid; clear after.
- Document.currentScript getter wraps the current nid. Next.js / Turbopack
reads document.currentScript.src to derive the chunk base path. Without
this the loader saw null, threw 'chunk path empty', and hydration stalled.
- Also threaded through the dynamic-script appendChild path so synchronously
injected <script> elements see their own element as currentScript.
data: URIs as subresources:
- subresource_allowed permits 'data:' alongside http/https. The bytes are
inline in the URI, no network fetch, no SSRF.
- Script-fetch loop short-circuits data: URLs and decodes them locally
instead of routing through the HTTP client.
- Real Chrome allows this by default; Instagram and most Meta properties
ship their bootstrap as <script src=data:application/x-javascript;base64,...>.
Dynamic <script type=module>:
- appendChild SCRIPT handler now accepts type='module' instead of dropping it.
- For external module src, uses dynamic import() instead of fetch + eval.
- For inline modules, wraps the source in a data: URL and import()s that
so module syntax (import/export) parses correctly.
Before / after on previously broken sites (HTML bytes rendered):
vercel.com 0 -> 960 KB
instagram.com 0 -> 669 KB
nextjs.org 0 -> 290 KB
music.apple.com 0 -> 1.88 MB
tailwindcss.com 0 -> 900 KB
linear.app 0 -> 2.4 MB
github.com PR 0 -> 468 KB
28/28 integration tests pass.
* engine: customElements.define upgrades existing matches + connectedCallback
The old customElements was a registry stub: define() recorded the class,
but never instantiated it on matching elements, never called the lifecycle
callbacks, and whenDefined() resolved synchronously with whatever was
already registered (or undefined forever for late-defined components).
New behavior matches the Web Components spec well enough for lit /
MusicKit / Polymer to wire up:
- define(): on register, querySelectorAll the document for matching tags,
copy the prototype methods onto each element instance, call the class
constructor with the element as 'this' for init logic, then fire
connectedCallback if the element is currently in the document.
- whenDefined(): returns a real Promise that resolves when the class is
later defined, not before.
- upgrade(root): walks the subtree, upgrades any matching unupgraded
element. Used by frameworks that mount components asynchronously.
We don't truly subclass HTMLElement per spec (Element is shared across
the engine), so the upgrade copies prototype descriptors onto the
element instance instead of swapping its [[Prototype]]. Behavior is
identical for the common patterns (custom methods, shadow DOM attach
in constructor, connectedCallback render).
28/28 integration tests pass.
* scraping: fix puppeteer page.$, click, page.$$ + form / radio / checkbox actions
These bugs broke every realistic Puppeteer scraping flow. Repro: type into
a search box, click a button, scrape rows. Before this commit the very
first step (page.$('selector')) returned null on every site.
Runtime.evaluate (and the two related wrappers) put the user expression
on a single line as '({expr})'. Puppeteer's bundled injected.js ends with
a '//# sourceURL=pptr:internal' line comment; the comment swallowed our
closing paren and the wrapped function failed to parse with a SyntaxError
that never propagated back. Result: every page.$, page.$$, type(),
hover(), click(), waitForSelector(), getByRole(), etc. silently returned
null. Fix: newline before the trailing ')' so the comment can't eat the
wrapper. Applied to all three wrap paths (eval, store_object_with_meta,
wrap_expression).
documentElement and body now return viewport dimensions for clientWidth /
clientHeight / scrollWidth / scrollHeight / offsetWidth / offsetHeight and
for getBoundingClientRect itself. Puppeteer's #clickableBox clips each
candidate quad to document.documentElement.clientWidth/Height; with the
old 100x20 stub every synthesized element bbox was clipped to empty and
clickablePoint threw 'Node is either not clickable or not an Element'.
Also aligned innerWidth / innerHeight to 1280x720 to match
Page.getLayoutMetrics.
document.elementFromPoint now does real hit testing against the synthetic
bboxes (depth-first, deepest match wins) instead of always returning
<body>. Input.dispatchMouseEvent uses it to pick the click target, which
removed the dependency on a __obscura_click_target side-channel that
the previous getBoundingClientRect call had to set right before the
dispatch.
Element.dispatchEvent now fires inline event handlers
(onclick='...' content attribute and the IDL onclick property) in
addition to addEventListener registrations. Without this, every site
that uses inline onclick / onsubmit / onchange was a no-op.
Input.dispatchMouseEvent's click path now triggers the right default
action by element kind: submit button -> form.submit(); checkbox /
radio -> toggle checked + dispatch 'change'; anchor with href -> nav.
Previously only anchor was handled, so checking a checkbox or submitting
a form by clicking its button did nothing through CDP.
Runtime.getProperties now allocates a stable child objectId for every
object-valued property and stashes the value in __obscura_objects.
Items whose value has a numeric nodeType also get subtype:'node' +
className so Puppeteer's JSHandle.asElement() wraps them as
ElementHandle. Without this, page.$$('selector') returned an array
of plain JSHandles and checkboxes[0].click() raised
'click is not a function'.
Real-world Puppeteer scrape coverage (test_all.py style scenarios):
- HN top stories scrape: fail -> pass
- GitHub repo file rows scrape: fail -> pass
- Wikipedia search input type: fail -> pass
- httpbin form fill + check: fail -> pass
- Cookies persistence across nav: pass -> pass
28/28 integration tests pass.
* scraping: inline scripts run on DCL, JS nav emits CDP events, hit testing
Three remaining gaps from realistic Puppeteer scraping flows:
1. DCL path skipped execute_scripts entirely. Returning early on
waitUntil=domcontentloaded meant every inline <script> was silently
dropped: addEventListener calls in onload bootstraps never ran,
form submit handlers never registered, page.click() hit no-ops.
Spec: DOMContentLoaded fires AFTER scripts. Now scripts run before
the lifecycle flips to DCL regardless of waitUntil.
To keep heavy SPAs from blowing the 25s Puppeteer goto timeout
now that scripts always run, added a soft deadline on the
script-execution phase (default 10s, OBSCURA_SCRIPT_DEADLINE_MS
to override). Both fetch and per-script execution check it.
2. JS-initiated navigation (form.submit, location.assign) did not
emit CDP frameNavigated / lifecycleEvent. Puppeteer's
waitForNavigation hung forever. Runtime.evaluate and
Runtime.callFunctionOn now drain pending_navigation through a
helper that fires the same post-nav event sequence Page.navigate
does.
3. Document.elementFromPoint walked the tree only descending into
ancestors containing (x,y). Our synthetic per-nid bboxes do not
form a containment hierarchy, so a deep <input> inside <label><p>
was never reached and clicks resolved to <body>. Switched to a
flat sweep, deepest match wins (highest nid as a proxy for tree
depth).
Realistic Puppeteer scrape scenarios: 18 of 18 assertions pass across
HN, Wikipedia, GitHub, httpbin forms, cookie persistence, and inline
JS submit handlers. 28/28 integration tests pass.
* scraping: fetch in page context, textarea, select, basic history
Four more gaps from the broad API sweep:
1. fetch() and XHR from page JS hung forever. set_intercept_tx() in
the runtime auto-set state.intercept_enabled = true as a side
effect, so every navigation silently enabled Fetch interception
even when the CDP client never called Fetch.enable. op_fetch_url
then routed every request through the intercept channel waiting
for a Fetch.continueRequest that the client wasn't going to send.
Split the flag from the channel: set_intercept_tx wires the
channel without enabling; new set_intercept_enabled toggles the
flag. Fetch.enable / Fetch.disable now call both, and the nav
path only wires the channel.
2. page.type into textarea lost everything past the first newline.
The dispatchKeyEvent handler skipped text=='\n' entirely, so
'multi\nline' inserted only 'multi'. Also Enter on a textarea
tried to submit the nearest form instead of inserting a newline.
Now Enter inserts \n inside textareas and only submits forms
from input fields, matching Chrome.
3. <select>.value did not reflect the selected option. Generic value
getter returned the select's value attribute (empty) instead of
the matching option. Added select-aware getter that walks options
for the selected one and falls back to the first; setter accepts
page.select() and dispatches a 'change' event.
4. Basic navigation history. Pages now track a visited-URL stack
and history_index. Page.getNavigationHistory returns the real
history; Page.navigateToHistoryEntry honors entryId and emits the
same CDP nav-event sequence as Page.navigate so the response and
lifecycleEvent reach waiting clients. resetNavigationHistory
clears the stack. Push happens after every successful nav (skips
consecutive duplicates so Page.reload doesn't grow the stack).
Broad API sweep across 13 feature groups: 43/44 PASS.
All categories solid (navigation, selectors, typing, waitForSelector,
evaluate, cookies, keyboard, attributes, events, fetch/XHR, storage,
timers). goBack reaches its target URL but Puppeteer client-side
waitForNavigation still hangs on it; the CDP frameNavigated +
lifecycleEvent fire correctly, so the remaining gap is in event
shape rather than logic. 28/28 integration tests pass.
|
||
|
SGavrl
|
b98e457428 |
engine: large compat + reliability pass (#206)
CDP / navigation: - Route all Page.navigate through process_with_interception so the parent dispatcher stays responsive during long navs (fixes Target.createTarget timeouts after a handful of newPage calls). - Default waitUntil to domcontentloaded. Puppeteer and Playwright don't pass it server-side and wait client-side via lifecycleEvent, so the old Load default blocked nearly every nav on slow sites. - 30s navigation hard ceiling, OBSCURA_NAV_TIMEOUT_MS to override. - execute_scripts loop now checks wall-clock deadline per iteration. - Emit Network.requestWillBeSent before Page.frameNavigated so Playwright's FrameManager attaches the navigation response (#189). - Use loaderId as the Document request id so page.goto() resolves to a Response instead of null (#189). - Fresh executionContextId per navigation, no more 'Cannot find context with specified id' after the first nav (#192). - POST navigation params plumbed through the intercept path. DOM / Web API surface: - MutationObserver: subtree filter walks parent chain, fires on textContent / CharacterData.data / innerHTML, type filter respected, oldValue passed. - IntersectionObserver: multi-fire (initial + 120/500/1500ms), disconnect / unobserve / takeRecords, root / rootMargin / thresholds. - ResizeObserver: per-target firing from getBoundingClientRect, drop the no-op override that was clobbering the real class. - crypto.subtle.digest: real SHA-1/256/384/512 via op_subtle_digest. - DOMParser / document.implementation.createHTMLDocument / createDocument: detached doc with body, head, documentElement, querySelector*, getElementById, getElementsByTagName. - WebSocket / EventSource / BroadcastChannel: real EventTarget, open fires on microtask. - indexedDB: request, object store, transaction, database with async onsuccess and in-memory Map storage. - getComputedStyle: width/height/top/left from bbox so React virtualization libs render items. Site compatibility went from 3/40 to 175/175 reachable sites in the sweep. Perf is equal or faster than pristine main on every site tested (DCL default removes the block on long-tail subresources). 28/28 integration tests pass. |
||
|
SGavrl
|
e7bb0c676d |
feat(cli): --allow-private-network for localhost development (#33) (#203)
Picks up @Timtech4u's PR #33, which has been waiting since April. The design (CLI flag + threading through BrowserContext + http client) is theirs and is correct; this is a clean rewrite on top of current main because repeated rebase rounds had been reverting unrelated work that landed in the meantime. Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is process-wide and clunky in pipelines. Multiple users (@Timtech4u in the original PR, then @duartemvix and @necrogami in the comment thread) reported they cannot use obscura for development against http://localhost:N without it. Adds --allow-private-network as a global CLI flag, works on fetch, serve, and scrape. Threads through BrowserContext::with_storage_and_network and the http client's validate_url. CdpContext gets a new_full builder and obscura_cdp::start_with_full_serve_options entry point so 'obscura serve --allow-private-network' lets Puppeteer/Playwright navigate to localhost too. Default remains blocked. Backward compat: the env var still works (OR'd with the flag inside validate_url). All existing constructors and entry points unchanged. Verified end-to-end against a local http.server on 127.0.0.1: fetch and CDP Page.navigate both blocked by default, both succeed with the flag. Public URLs unaffected. Integration suite 28/28, full workspace cargo test all green. |
||
|
SGavrl
|
1a1253118f |
feat(cli): default V8 heap to 4 GB on 64-bit so heavy SPAs don't OOM (#199) (#202)
V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/ web-scraping) blow past it and the worker dies with 'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session with it. Always set --max-old-space-size=4096 (1024 on 32-bit targets) before V8 startup, matching headless Chrome's default. User-supplied --v8-flags are appended after the default so they win (V8 parses left-to-right and the last value wins), which keeps the existing escape hatch intact for memory-constrained environments. |