previousNode() considered exactly one candidate — the previous sibling's
deepest last child — and returned null as soon as the filter rejected it,
so a backward walk died mid-tree. This is the backward mirror of #432,
which was fixed forward-only in #447.
<div id=root><a><b></b></a><c></c></div>, SHOW_ELEMENT, B -> FILTER_SKIP
forward: ["A","C"] backward: [] (Chrome: ["A"])
Replaced with the DOM 6.1 algorithm verbatim: scan previous siblings,
descend to the deepest last descendant of each, then step up to the
parent, continuing until a node is accepted or root is reached. The
descent stops on FILTER_REJECT, which is what makes rejection prune
subtrees in the backward direction too.
Unlike nextNode this stays in JS rather than adding a DOM traversal op:
the descent has to consult the filter at every step to honour REJECT, so
there is no run of JS/native crossings for a native helper to collapse.
Depends on the three-valued `_filter` from #461.
`_accept` collapsed every non-ACCEPT filter result into a single boolean,
so FILTER_REJECT behaved exactly like FILTER_SKIP: the node was not
returned, but the walk still descended into its subtree. Per DOM 6.1,
FILTER_REJECT rejects a node *and* its descendants.
This was unreachable until #440 exposed the real NodeFilter constants,
which made the canonical prune idiom
if (node.tagName === 'SCRIPT') return NodeFilter.FILTER_REJECT;
start yielding nodes from inside the pruned subtree instead of failing
loudly — sanitizers and text extractors built on TreeWalker silently got
extra content.
Split the filter into a three-valued `_filter` (keeping `_accept` as the
boolean wrapper the sibling/child movers use) and give nextNode a second
traversal step for the reject case. That step lives in the DOM layer as
`DomTree::next_after_subtree`, mirroring how #447 put `next_in_subtree`
there: both share the ancestor climb, and REJECT costs no more JS/native
crossings than SKIP.
NodeIterator shares this walker but has no subtree pruning (DOM 6.2:
FILTER_REJECT behaves as FILTER_SKIP), so `createNodeIterator` marks the
walker `_rejectIsSkip` and keeps the plain step. Covered by its own test
so the shared implementation cannot drift.
Move fragment children into the target for appendChild, insertBefore, and replaceChild while preserving their order and emptying the fragment.
Add runtime regression coverage for the DOM attachment portion of #452.
The premise was wrong. deno_core 0.350's run_event_loop is:
pub async fn run_event_loop(&mut self, opts) -> Result<(), CoreError> {
poll_fn(|cx| self.poll_event_loop(cx, opts)).await
}
so replacing `timeout(budget, run_event_loop())` with a hand-rolled
`poll_fn(poll_event_loop)` loop is the same code with the same cancellation
semantics, not safer ones. The mechanism I described cannot happen either:
`poll` is a synchronous `&mut self` call, a future can only be dropped between
polls, and the HandleScope/ContextScope live inside poll_event_loop as stack
locals that are destroyed before it returns. The isolate is exited between
polls both before and after that change.
What actually fixes#430 is confining each connection's isolates to their own
OS thread, which the next commits do. The regression test still reproduces the
abort on the pre-thread-per-connection tree and still passes here without this
commit, so nothing depended on it.
It was not merely a no-op. Two regressions it introduced:
- resolve_promises_until inverted the meaning of tick_ms. It was "pump the
event loop for up to tick_ms, returning early on idle"; it became one single
poll followed by an unconditional sleep(tick_ms). With the 1->50ms backoff
that settles at 20 event-loop turns per second, so an awaited promise chain
needing many turns crawls, and past the budget the caller silently reads a
stale __obscura_await_meta.
- drive_module_eval replaced `e?` with `let _ = poll_event_loop(...)`,
discarding event-loop errors. A module stalled on a top-level await used to
exit immediately with a logged error; it instead spun for the whole budget
logging nothing -- the exact #374 starvation the surrounding comment claims
to prevent.
Keep the drive_module_eval extraction (it genuinely dedupes load_module and
load_inline_module) but restore the original select!-based body.
Refs #430.
- Delete the now-unused CdpContext::new_full (its only caller switched to
new_with_shared_context); have _new_inner build the shared BrowserContext and
delegate, so the struct's field list lives in one place.
- Extract drive_module_eval, shared verbatim by load_module and
load_inline_module.
- Update comments that still described the deleted process-wide V8 lock and the
single-LocalSet model to the per-connection reality, drop the dangling
reference to the removed obscura-js/src/v8_lock.rs, and soften the
ISOLATE_CREATE_LOCK note now that the main-thread warmup is the real guard
against the InitializeBuiltinJSDispatchTable segfault.
Refs #430.
The per-command V8 watchdog kept a single global slot whose correctness relied
on the old process-wide lock (at most one command armed at a time). With
thread-per-connection several connections arm concurrently, so a second arm
overwrote the first's slot and left that command's isolate unbounded. Key the
armed slots by generation so each connection's command has its own; the worker
fires whichever have overrun and terminates each via its IsolateHandle.
Refs #430.
With every connection on its own thread (previous commit), the process-wide V8
lock served no safety purpose across connections yet forced all of them to
serialize on one mutex, so throughput did not scale with concurrency. Move the
lock into CdpContext so each connection has its own: a connection's nav task and
command dispatch still coordinate on one page's isolate at a time on its thread,
but different connections no longer block each other. Delete the now-unused
process-wide obscura_js::v8_lock.
Measured on a local CPU-bound fixture, page throughput stays flat (~75/s from
concurrency 1 to 8) with the global lock and scales to about 2.5x at concurrency
8 with the per-connection lock, with no crash at any concurrency. Obstacle
course 33/33, obscura-cdp 76/76.
Refs #430.
The CDP server ran every connection's pages on one LocalSet thread, so two
pages' V8 isolates shared an OS thread. Under concurrent navigation a cancelled
event-loop future could leave one isolate entered while another page's
execute_script ran, tripping V8's heap->isolate() == Isolate::TryGetCurrent()
check and aborting the process (#430). No single-thread serialization fixes
this reliably: the cancellation-leak pattern is pervasive and partly inside
deno_core.
Give each WebSocket connection its own OS thread with its own current_thread
runtime, LocalSet, and CdpContext, so every page's isolate is confined to one
thread. V8's invariant is per-thread, so two connections' isolates can never
collide. This is the "Option 2" fix named in obscura-js/src/v8_lock.rs, and it
also lets connections run in parallel instead of serializing on one thread.
- CdpContext::new_with_shared_context reuses one Arc<BrowserContext> across all
connections, so they share one cookie jar and one HTTP client.
- A single signal watcher flips the accept flag and wakes every connection's
processor via notify_waiters(); the accept side persists cookies once on
shutdown.
- Serialize isolate construction with a process-wide lock: the leaptiering
JSDispatchTable setup is not safe to run from several threads at once.
- Warm up V8 once on the main thread before any connection thread creates an
isolate; creating the first isolate off the main thread segfaults in
InitializeBuiltinJSDispatchTable.
obstacle course 33/33, obscura-cdp 75/75 including the concurrent-navigation and
control-plane tests. Concurrency 2 and 4 against heavy pages no longer abort.
Fixes#430.
Several event-loop operations bounded their runtime with
tokio::time::timeout(budget, run_event_loop()) or a select! against a
timer. When the timer wins, tokio drops the run_event_loop future
mid-poll; deno_core enters the isolate while polling, so the drop leaves
the isolate entered on the OS thread. Under concurrent CDP work a second
page's execute_script then trips heap->isolate() == TryGetCurrent() and
aborts the process (#430).
Drive the loop with poll_event_loop inside a poll_fn instead. It creates
and drops its own isolate scope per call, so stopping between ticks never
leaves the isolate entered. Convert resolve_promises_until,
run_event_loop_bounded, resolve_promises, the mod_evaluate settle in
load_module and load_inline_module, and the two navigation settle loops
in page.rs. Add poll_event_loop_once as the cancellation-safe single-tick
primitive the settle loops use.
Keeps the obstacle course at 33/33. This is hardening toward #430; the
residual concurrency abort still needs the thread-per-runtime refactor
that v8_lock.rs calls the "Option 2" fix.
Refs #430.
The on_request/on_response Vec lived on the ObscuraHttpClient shared by
every page in the context, so a callback registered on one page fired for
all pages (#408 cross-page bleed) and outlived its page.
Callbacks now live in a per-page CallbackRegistry (Arc shared with the
page's JS runtime state). Page-driven fetch paths (navigation, POST,
scripts, stylesheets, robots.txt, module loads, JS fetch/XHR in both
plain and stealth modes) pass the registry through; the HTTP client
itself is callback-free.
Adds callbacks_do_not_bleed_across_pages: page A's callback stays silent
for page B's requests and stops firing after page A is dropped.
on_request/on_response stored callbacks in append-only Vecs with no way
to remove one, so a crawler that installed a response-body capture hook
for one phase could never detach it: it fired for the shared client's
whole lifetime, re-registering across navigations piled up duplicates,
and (because the client is shared per context) it saw sibling pages'
requests too.
Tag each callback with a stable id. on_request/on_response now return
the id and off_request/off_response (remove_on_request/remove_on_response
on the client) detach a specific one. Storage is Vec<(u64, Callback)>;
all fire sites destructure the tuple. Closes#408.
Requests made by page JS (fetch/XHR/dynamic script and stylesheet loads)
run in the V8 op layer and never produced a NetworkEvent, so CDP emitted
Network.requestWillBeSent / responseReceived only for the static
navigation subresources. Puppeteer/Playwright page.on('request') and
on('response') therefore missed every XHR/JSON response, which breaks
response capture (and is the root cause of the Aviasales half of #394).
op_fetch_url now records a JsNetworkEvent keyed by the same fetch-{N} id
under which it already stores the response body. The Page drains these
into its network_events before the CDP layer emits, so they flow through
emit_navigation_events as requestWillBeSent / responseReceived /
loadingFinished alongside the document subresources, and
Network.getResponseBody resolves for them via the shared id. Applied to
both navigation paths (server process_navigation and in-process
do_navigate). The queue is capped at 4096 events per page.
Closes#406.
load_module and load_inline_module drove run_event_loop to full idle
before reading the module's mod_evaluate result. A page timer keeps the
loop busy forever (Vite's HMR / React-Refresh client installs a
setInterval), so the preamble inline module burned the entire budget
waiting for an idle that never came, and the later module that mounts the
app was skipped, leaving #root empty while the production preview rendered
fine (issue #374).
Drive the event loop and the mod_evaluate result together and return as
soon as the module finishes evaluating, so a busy background timer no
longer starves the remaining modules. Post-eval async work is still
drained by the normal post-load settle.
Fixes#374.
Regression test for #355: an uncaught `throw` in one inline <script> must
not prevent later independent <script>s from running. This was fixed by
the #147 poison-guard (present on main) but reported again against 0.1.9
with an explicit throw; this pins the exact scenario so it cannot regress.
Closes#355.
el.style (CSSStyleDeclaration) and el.dataset (DOMStringMap) were plain
Proxies with only get/set traps, so `'color' in el.style`, `Object.keys
(el.style)`, `'foo' in el.dataset`, and `Object.keys(el.dataset)` all
failed, and Object.keys(el.style) leaked the internal `_props` (issue
#356). cssText also serialized camelCase names without a trailing
semicolon.
Add has / ownKeys / getOwnPropertyDescriptor traps to both proxies so the
`in` operator and enumeration behave like real CSSOM. CSSStyleDeclaration
now stores properties by their dashed name (matching the spec), so
el.style.fontSize and getPropertyValue('font-size') stay in sync, and
cssText serializes dashed names with a trailing semicolon. A known CSS
property list backs `in`/enumeration so feature detection ('object-fit'
in el.style) works for unset properties.
Closes#356.
Typing over CDP into a React controlled input never fired onChange, so
the value never reached component state (issue #324). Two root causes,
both needed:
1. Value tracker. React/Preact/Vue install a value tracker by redefining
`value` on the element instance to record the last value they wrote.
obscura's CDP typing assigned `el.value = ...` directly, which runs
that wrapper and keeps the tracker in sync, so React saw no change on
the input event. Synthetic value/backspace writes now go through a new
__obscura_setFieldValue helper that writes via the element's prototype
setter, bypassing the tracker. The tracker stays stale, so the input
event reads as a real edit.
2. Event handler feature detection. React decides whether to use the
modern `input` event via `("oninput" in document)`. obscura exposed the
GlobalEventHandlers on* attributes only on window, so that check failed
and React fell back to a legacy change-detection path that ignores
input events. The on* attributes are now also present on Document and
Element, matching the HTML spec (GlobalEventHandlers is mixed into
Window, Document, and HTMLElement).
Verified end to end over CDP against the real React 18 UMD build: typing
into controlled inputs updates state, the component re-renders, and a
submit handler reads the captured values.
Closes#324.
crypto.subtle only implemented digest. sign returned a fixed 32-byte
buffer, verify always returned true, encrypt and decrypt threw, and
generateKey, importKey and deriveBits handed back placeholder data.
Pages that use WebCrypto (anti-bot probes, PKCE and other auth flows,
client-side crypto) silently got wrong results.
Implement the secret-key half of WebCrypto on RustCrypto, following the
existing op_subtle_digest pattern:
- HMAC sign and verify (SHA-1/256/384/512)
- AES-GCM, AES-CBC and AES-CTR encrypt and decrypt
- PBKDF2 and HKDF deriveBits and deriveKey
- raw and symmetric JWK (oct) importKey and exportKey, generateKey,
wrapKey and unwrapKey
- getRandomValues and randomUUID now draw from the OS CSPRNG instead of
Math.random, so they are uniform across typed-array widths and no
longer a fingerprinting tell
Unsupported algorithms (RSA, ECDSA, ECDH) and key formats (pkcs8, spki)
throw NotSupportedError instead of returning fake data. The crypto runs
in stateless Rust ops; the bootstrap.js shim only marshals bytes and
normalizes algorithm parameters, so the native fast paths are untouched
and there is no cost unless a page calls crypto.
Verified against RFC 4231 (HMAC), the NIST GCM vectors, NIST SP800-38A
(AES-CTR), RFC 6070 (PBKDF2) and RFC 5869 (HKDF), plus roundtrip and
tag/padding tamper-rejection checks.
A Continue resolution that rewrites the request URL applied the new URL
after validate_fetch_url had already run on the original, so a rewrite
could reach a private or internal address the original request was
blocked from. Run the overridden URL back through validate_fetch_url,
matching how redirects are re-validated, and return a blocked response
if it fails.
Also fix a doc comment that ended up attached to the new fetch_response
helper instead of stealth_fetch_all.
op_fetch_url ignored the url/method/headers/body fields on
InterceptResolution::Continue, so the interception channel could pass, mock, or
block a request but not rewrite it. Apply them: shadow url/method/body and
derive custom_headers from the override after the channel resolves Continue.
Also fix enable_interception() called before the first navigation: init_js
propagated intercept_tx to the new runtime but not intercept_enabled, so the
runtime started with interception disabled and op_fetch_url never intercepted.
Propagate intercept_enabled in init_js as well.
Add a test that rewrites a fetch's url via Continue and asserts the rewritten
response is captured.
Expose the interception infrastructure that already existed internally but was
private, so embedders can intercept, observe, mock, or block JS fetch()/XHR
without forking or unsafe casts.
On obscura::Page:
- add_preload_script: inject JS before any page <script> (CDP
Page.addScriptToEvaluateOnNewDocument). The field already ran the scripts;
this only adds the method.
- enable_interception: returns a receiver for every JS fetch()/XHR request;
resolve each through its resolver with InterceptResolution::{Continue,
Fulfill, Fail} to pass, mock, or block (CDP Fetch.enable equivalent).
- on_request / on_response: passive callbacks. on_response is the main path for
capturing API response payloads from SPAs.
on_request/on_response previously fired only for navigation; op_fetch_url now
invokes them for JS fetch()/XHR too (both reqwest and stealth paths), gated on
a non-empty callback vector so the common no-callback case pays nothing.
Re-export InterceptedRequest, InterceptResolution, RequestInfo, ResourceType,
RequestCallback, ResponseCallback, Response from the obscura crate.
Integration test tests/interception.rs exercises all three against a local echo
server.
Resolved conflicts:
- page.rs: keep both set_blocked_urls (#315) and stealth client wiring
- ops.rs: keep SHA-512/224 + SHA-512/256 from stealth branch; keep
the unreachable catch-all (vec![]) from #314 instead of silent
SHA-256 fallback; updated comment accordingly
- bootstrap.js: extend the digest algorithm allowlist to include
SHA-512/224 and SHA-512/256 to match the Rust additions
op_fetch_url backed JS fetch() and XHR with a plain reqwest/rustls client, so
in stealth mode scripted requests carried a non-browser TLS fingerprint and
none of the Chrome client hints the main navigation sends. Bot managers that
score subresource requests (the AWS WAF challenge verify call, Akamai sensors)
read that as a script and reject it. Add StealthHttpClient::send_single and a
stealth_fetch_all path that op_fetch_url uses when a stealth client is
installed, preserving the redirect, SSRF, and CORS handling so JS-level
requests share the navigation's Chrome TLS fingerprint and sec-ch-ua client
hints. The reqwest path is unchanged for non-stealth. Adds a stealth feature to
obscura-js gated on obscura-net/stealth.
The digest op fell through to SHA-256 for any unrecognized algorithm name,
so SHA-512/224 and SHA-512/256 silently returned a SHA-256 hash. Add the
two FIPS 180-4 truncated variants, which the AWS WAF challenge proof-of-work
(used by booking.com and other CloudFront/WAF sites) requires.
Fixes#314.
Unknown algorithm names (SHA-512/224, MD5, etc.) were silently falling
back to SHA-256, producing wrong bytes with no error. Now throws
DOMException('NotSupportedError') to match Chrome behavior.
Supported: SHA-1, SHA-256, SHA-384, SHA-512.
Added Network.setBlockedURLs dispatch handling.
Stores blocked URL patterns on each page.
Applies blocking to page subresources and JS fetch / XHR requests.
Supports wildcard CDP-style patterns like *://*.gstatic.com/*.woff2.
--dump assets only walked static DOM attributes (script[src], img[src], ...),
so resources a page pulled in with fetch() or XHR never showed up (issue #301).
Record each absolute URL requested through op_fetch_url in ObscuraState,
expose it via the runtime and Page, and append those URLs to --dump assets as
type "fetch", deduped against URLs already listed from the DOM.
hasIframeProxy fix contentWindow now returns null when parentNode === null (iframe never inserted into any tree). The previous attempt used isConnected, which broke getBehemothIframe by returning null for iframes inside the phantom iframe's detached document. That caused PHANTOM_DARKNESS to fall back to self (main window), making all DOM measurement ops run on the main document instead of the isolated iframe body, which corrupted the fingerprint element lookup.
Worker auto-execution Worker constructor now resolves relative URLs against location.href, fetches the script, and auto-runs it. The worker scope exposes WorkerGlobalScope so IS_WORKER_SCOPE resolves correctly in creepjs. All three worker types now complete within the 3s timeout instead of silently timing out.
AudioBuffer + AudioContext AudioBuffer class added to globalThis with getChannelData, copyFromChannel, copyToChannel. AudioContext.createBuffer returns a real instance. AudioContext.addEventListener was a no-op; startRendering now dispatches to stored listeners so the complete event fires.
Crypto Crypto class added to globalThis with randomUUID and getRandomValues on its prototype. Fixes ReferenceError in getPlatformEstimate.
NetworkInformation + ContentIndex NetworkInformation class with downlinkMax on the prototype clears noDownlinkMax. ContentIndex stub on globalThis clears noContentIndex. navigator.connection is now a real NetworkInformation instance.
getElementById / querySelector stale index fix id_index only registers nodes at creation time. When replaceChild(fragment, el) swapped in new nodes their IDs hit .or_insert() and were skipped, leaving the index pointing at the detached original. getElementById now verifies the indexed node is in the live document and falls back to a full attribute-selector scan. query_selector_from falls through to the full scan on both index miss and stale entry instead of returning None early.
matchMedia Handles common queries instead of always returning false. prefers-color-scheme: dark returns true (headless Chrome defaults to light mode, so returning true for light is a bot signal). any-pointer: fine, hover: hover, prefers-reduced-motion: no-preference, min/max-width, and (color) return sensible values.
Fingerprint consistency jsHeapSizeLimit raised to 4294705152 (4 GB, matching 64-bit Chrome). getHighEntropyValues platform corrected from "Linux" to "Windows", platformVersion from "6.1.0" to "10.0.0" to match the Windows NT 10.0 UA string. Canvas: real PNG encoder, arc fill, multiply blend mode. Screen class, performance.now, chrome runtime errors fixed.
Build note deviceMemory and hardwareConcurrency use the high-value pools ([4, 8] GB and [4, 6, 8, 12, 16] cores) only when built with --features stealth.
Worker auto-execution: Worker constructor resolves relative URLs against
location.href, fetches the script, and auto-runs it. WorkerGlobalScope is
exposed so IS_WORKER_SCOPE resolves correctly. Worker data (navigator,
locale, timezone) now arrives via postMessage within the 3s timeout.
AudioBuffer class added to globalThis with getChannelData, copyFromChannel,
copyToChannel. AudioContext.createBuffer returns a real AudioBuffer. Fixes
ReferenceError that blocked the audio fingerprint section.
Crypto class added to globalThis with randomUUID and getRandomValues on its
prototype. Fixes ReferenceError in getPlatformEstimate.
AudioContext.addEventListener was a no-op; startRendering only fired
oncomplete. Add _listeners map and dispatch to stored listeners.
getElementById and querySelector('#id') returned null for elements inserted
via replaceChild(documentFragment, el). id_index only registers nodes at
creation time; .or_insert skipped IDs on cloned fragments; None caused
query_selector_from to short-circuit the full scan.
Fix querySelector: fall through to full scan on index miss or stale entry.
Fix getElementById: verify node is in live document; fall back to attribute
selector scan if not.
contentWindow getter: return null when parentNode is null (iframe was never
inserted into any tree). Previously returning null for all disconnected
iframes broke getBehemothIframe, which caused PHANTOM_DARKNESS to fall back
to self (main window) instead of the phantom iframe. That made DOC point to
the main document, so rect and font measurements wrote fragment nodes into
the main body, and the subsequent full-DOM scan in getElementById could not
find fingerprint-data through nested fragments in time.
Limiting the null return to parentNode===null (never inserted) means
getBehemothIframe gets a valid nested contentWindow as expected. creepjs
now reports hasIframeProxy: false and 0% stealth detection.
DocumentFragment constructor: pass nid through when provided.
Range.getBoundingClientRect / getClientRects: return commonAncestor rect.
Upgraded wreq to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension order, ALPS, GREASE).
Introduced __obscura_stealth global to isolate stealth-specific JS behaviors from the standard execution path.
Overrode navigator.userAgentData and getHighEntropyValues() in stealth mode to perfectly match the sec-ch-ua network headers.
Restricted hardwareConcurrency and deviceMemory pools in stealth mode to desktop-realistic values (e.g., minimum 4 cores, 4GB RAM) to eliminate mobile-tier leaks.
Fixed usedJSHeapSize occasionally exceeding totalJSHeapSize.
wreq upgraded to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the
latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension
order, ALPS, GREASE).
__obscura_stealth global variable introduced and passed from Rust to JS,
allowing stealth-specific JS behaviors to run only when the stealth flag
is enabled.
Tightened hardwareConcurrency and deviceMemory pools in stealth mode
to desktop-realistic values (e.g. min 4GB RAM, 4 cores), eliminating
mobile-tier values that leak the spoof.
navigator.userAgentData and getHighEntropyValues() are now overridden
in stealth mode to perfectly match the sec-ch-ua header sent by
wreq_util::Profile::Chrome145.
Fixed usedJSHeapSize sometimes exceeding totalJSHeapSize.
deviceandbrowserinfo.com detects CDP by setting Error.prepareStackTrace as a spy, then calling console.log(new Error()). _consoleFn accessed a.stack, which fires prepareStackTrace if set. Fix: save/clear/restore around the .stack access.
NetworkInformation (navigator.connection) and ServiceWorkerContainer (navigator.serviceWorker) are EventTargets in real browsers. React 18 and mobile SPAs call addEventListener on them; the stubs lacked it, so the app bundle crashed with 'addEventListener is not a function' (e.g. goofish.com).
Add is_forbidden_ip() (loopback, RFC1918, link-local incl. 169.254.169.254,
broadcast, documentation, 0.0.0.0/::, IPv6 unique-local fc00::/7, IPv4-mapped
IPv6) and SsrfGuardResolver (impl reqwest::dns::Resolve) that resolves and
rejects forbidden addresses at connect time, closing the DNS-rebinding bypass.
Wired into the navigation client and the JS fetch()/XHR path. The
--allow-private-network / OBSCURA_ALLOW_PRIVATE_NETWORK escape hatch is kept.
Co-authored-by: charlesmamane26-sketch <[email protected]>
DDoS-Guard's view.js challenge calls Element.prototype.insertAdjacentText
and dies with 'TypeError: t.insertAdjacentText is not a function', blocking
any DDoS-Guard-protected site (e.g. pikabu.ru). insertAdjacentElement was
missing too. insertAdjacentHTML was already implemented, so this just lifts
the same insertion pattern for Text-node and Element insertion semantics.
Both methods mirror the spec: case-insensitive position keywords, four
insertion points (beforebegin/afterbegin/beforeend/afterend), and
insertAdjacentElement returns the inserted element (or null when
beforebegin/afterend has no parent). Registered with _markNative so
Function.prototype.toString keeps reporting [native code] — consistent
with the existing Element.prototype.insertAdjacentHTML stealth registration.
The geolocation shim returned a fixed Frankfurt position. With the timezone now
configurable, a deployment in another region would report a location that does
not match its exit IP or timezone, the mismatch issue #228 describes.
Set OBSCURA_GEOLOCATION="lat,lon" to override the reported coordinates; the
value is range-checked and falls back to the Frankfurt default when unset or
malformed. The existing positional jitter is preserved.
* feat(stealth): randomize browser profile per session
Add a pool of 8 realistic browser profiles (Win10/Win11/macOS, Chrome
143-146). One is picked randomly per BrowserContext instead of always
advertising Linux, which is a strong bot signal.
Each profile keeps navigator.platform, userAgentData.platform, and
userAgentData.platformVersion internally consistent with the UA string.
- profiles.rs: static PROFILES pool + random_profile() picker
- context.rs: pick profile on construction, store platform fields
- page.rs: call rt.set_platform() alongside rt.set_user_agent()
- runtime.rs: add set_platform() to inject __obscura_platform globals
- bootstrap.js: read platform/userAgentData from globals (default Win32)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* feat(stealth): match GPU renderer pool to OS profile
Windows profiles get ANGLE Direct3D11 renderers; macOS profiles get
ANGLE Metal renderers. Vendor strings updated to match (Google Inc. (Apple)
for Apple Silicon/Intel Mac, Google Inc. (NVIDIA/AMD/Intel) for Windows).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* fix(stealth): navigator.webdriver returns false instead of undefined
Browsers set navigator.webdriver to false when automation is not active.
Returning undefined is a detectable bot signal — fingerprinting scripts
check typeof navigator.webdriver === 'boolean' as part of headless detection.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* Make stealth profile stable by default, rotation opt-in
Picking a random profile per BrowserContext meant one address would cycle
through different browser identities, which is itself a bot signal, and the
rotated profile carries no matching TLS or timezone fingerprint. Default to a
single stable profile; opt into rotation with OBSCURA_ROTATE_PROFILE, or pin
one with OBSCURA_PROFILE=<index>.
Drive timezone from the process TZ so Intl.DateTimeFormat and Date report one
consistent zone. Default Europe/Berlin; override with OBSCURA_TIMEZONE.
Update navigator.webdriver test to expect false, matching real Chrome.
---------
Co-authored-by: SGavrl <[email protected]>
A slow non-essential ES module could block navigation for ~10s. On real
pages (www.ycombinator.com) obscura spent ~11.5s loading YC's bookface
module, whose top-level eval idle-waits forever, until the phase watchdog
killed it. The page renders fully without it (CPU was ~3s, the rest pure
idle), but wall time was ~2.5x lightpanda.
load_module / load_inline_module now take a per-module budget and bound
both the import-graph fetch and the eval loop with it. page.rs sizes the
budget by whether the body is already rendered: an enhancement module on
a populated page (>50 body descendants) gets a short budget (3s, override
OBSCURA_MODULE_BUDGET_MS); an unmounted SPA shell keeps the full script
budget so the app module still mounts (#205). Fetch concurrency is
unchanged; only the idle eval wait is bounded.
www.ycombinator.com: wall 13.1s -> 6.6s, still 475 links, CPU unchanged.
teragrowth.org (Vite SPA) still renders. Obstacle course 33/33 including
the es-modules / ssr-hydrate / spa stages. Post-drop eval works.
* DOM read fast paths: constant nodeType, fewer ops and allocations
- Element.nodeType is now a constant getter instead of inheriting Node's
dynamic op-backed getter. nodeType is invariant (1) for element wrappers
(wrappers are only built for element nodes and node ids are never freed
and reused), so this drops one JS/Rust op per read on a pervasive path.
Micro-bench: ~600k nodeType reads go from ~380ms to ~5ms.
- attributes getter reads each attribute once instead of twice.
- dataset Proxy is cached per element instead of reallocated on every
access, which also makes el.dataset === el.dataset (spec-correct).
- stylesheet <link> scan borrows the node (with_node) instead of deep
cloning it and compares rel via eq_ignore_ascii_case, dropping a clone
and a lowercased String alloc per link per navigation.
No regression: obstacle course 33/33, WPT dom/nodes Node- subset identical
to baseline (3551/4163 subtests), targeted DOM probe identical.
* DOM op reads: borrow the node instead of deep-cloning it
The scalar read arms in op_dom (node_type, node_name, tag_name,
get_attribute, attribute_names, parent/first/last/next/prev lookups,
has_child_nodes, pi_target, doctype_name, doctype_public_id) and the
node_root / node_ancestors / node_child_index walks called get_node,
which clones the whole Node (for elements a deep clone of the attribute
Vec and its Strings) just to read one field. Switch them to with_node,
which borrows under the RefCell. element_children and the
document_element / document_doctype child scans stay on get_node on
purpose (they iterate, so the borrow story is left untouched).
Output is identical. Traversal/attribute micro-bench ~25-30% faster with
fewer allocations per op. No regression: obstacle course 33/33, WPT
dom/nodes identical to baseline (7057/10620 across 314 files).
* base64 decode: reverse-lookup table instead of indexOf scan
_base64ToUint8Array decodes every fetch/XHR response body. It ran an
O(64) alphabet.indexOf plus a one-char substring alloc per character.
Build a charCode->6-bit reverse table once at module load and index it by
charCodeAt, folding out-of-range (NaN or code >= 128) to -1 so the
bitmath stays byte-identical to the old indexOf path on valid and
malformed input alike.
~20-25% faster decoding a 2MB body; output byte-identical (differential
test across valid, unpadded, '='-position, >127, non-alphabet and
truncated inputs matches the old code exactly). No regression: obstacle
course 33/33.
build_request_client (the client behind every JS fetch(), XHR, and ES-module
load) set no request timeout. A cross-origin XHR with a non-simple header
triggers a CORS preflight OPTIONS; if the server accepts the connection but never
answers it, op_fetch_url's send().await hung forever, the fetch promise never
settled, and the XHR was stuck at readyState 1 with no completion event ever
firing. That stranded Angular HttpClient on live.deutsche-boerse.com.
Give the client a 30s request timeout (matching the other clients in the
workspace); OBSCURA_FETCH_TIMEOUT_MS overrides it. On timeout the request errors,
op_fetch_url propagates it, and the fetch shim turns it into an XHR error/loadend,
so readyState advances to 4 and Angular's error path runs.
Verified: a server that hangs on OPTIONS now makes the XHR fire error at the
timeout (was stuck forever); a CORS-allowed cross-origin XHR still succeeds with
load/200; obstacle course 33/33.
Closes#251
A hung page (a runaway Runtime.evaluate, a synchronous DOM op) holds the
process-wide V8 lock and wedges every other session forever. The one-shot CLI has
a process-level hard deadline for this; the long-running server cannot force-exit,
so terminate just the offending isolate instead.
dispatch() arms a watchdog on the session page's isolate after taking the V8 lock
and disarms it after the handler; on overrun it terminates the isolate and clears
the termination flag so the session keeps working. OBSCURA_CDP_COMMAND_TIMEOUT_MS
tunes the bound (default 60s, 0 disables); navigation already self-bounds well
under it.
The watchdog is a single long-lived shared thread (cdp_watchdog), not a thread
per command: the V8 lock serializes commands so at most one is ever armed, and
arm/disarm are a mutex plus notify with no measurable per-command cost. Also
fixes a lost-wakeup in the per-call watchdog (spawn_watchdog) where disarming a
fast call before the watchdog thread started blocked for the full budget.
Verified: a hung Runtime.evaluate is terminated at the budget and the session
recovers; CDP integration 37/37, obstacle course 33/33; per-command latency
unchanged (watchdog on equals off within noise).
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
* conformance: HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser, fetch UA
Next WPT Core batch, all verified live and benchmarked for no hot-path regression.
DOM:
- compareDocumentPosition rewritten to spec: real PRECEDING/FOLLOWING via the
compare_order op, CONTAINS/CONTAINED_BY, and DISCONNECTED|IMPLEMENTATION_SPECIFIC
for different roots. Adds the DOCUMENT_POSITION_* and missing nodeType constants
on Node and its prototype. WPT getPosition() depends on this, so it cascades into
the Range comparePoint/compareBoundaryPoints/isPointInRange expectations.
- HTML elements now ASCII-lowercase attribute names in getAttribute/setAttribute/
removeAttribute/hasAttribute, so el.setAttribute('accessKey') reflects via
getAttribute('accesskey'). The fold is gated behind a cheap uppercase scan and a
namespaceURI HTML guard (SVG/foreign attributes keep their case), and getAttribute
only folds on a miss, so the all-lowercase hot path is unchanged (benchmarked flat).
- Range comparePoint/isPointInRange ToUint32-coerce the offset (negative throws
IndexSizeError); createTreeWalker/createNodeIterator keep an explicit whatToShow of 0.
URL:
- URLSearchParams uses a proper form-urlencoded byte decoder (partial/invalid escapes
stay literal instead of decodeURIComponent throwing on the whole string), 2-arg
has(name,value)/delete(name,value), and a sequence-pair TypeError. Adds URL.parse
and URL.canParse statics.
Other:
- fetch()/XHR send a default Chrome User-Agent (the navigation path set one but
scripted requests went out with none, so UA-gated servers rejected them); honors an
explicit override.
- DOMParser documents expose URL/compatMode/characterSet/readyState/styleSheets/
defaultView and createTreeWalker/createNodeIterator; XMLDocument is defined globally.
* perf(dom): O(1) querySelector('#id') via the id index
querySelector('#id') went through the full selector engine, scanning descendants
in tree order until the first match (O(n), ~68us per call over a 2000-element
document in the micro-bench). A bare ASCII id selector now resolves through the
existing id index in O(1): ~68us -> ~1.25us per call, about 55x faster, with no
change to any other path (create/getAttribute/setAttribute/querySelectorAll/href
all flat). querySelector by id is one of the hottest scraping operations.
The fast path is conservative: only a bare `#name` selector with an ASCII
letter/underscore-led identifier takes it; escapes, combinators, commas,
whitespace, compounds, and non-ASCII fall through to the full engine unchanged.
For a scoped element.querySelector it returns the indexed element only when it is
a strict descendant of the scope, and falls back to the scan otherwise, so
duplicate-id scoped queries stay correct.
Also makes the id index keep the FIRST element in tree order per id (or_insert at
parse time) instead of the last, so getElementById and querySelector('#id') return
the first-in-tree-order element on duplicate ids, per spec (previously last).
* fix: bound synchronous V8 hangs with a termination watchdog and add op panic safety
Heavy pages could hang obscura fetch indefinitely: tokio::time::timeout only
cancels at await points, so synchronous V8 work (a page's script bootstrap, a
microtask storm) pinned the thread and ran unbounded.
Add a reusable V8 terminate_execution watchdog (arm_watchdog/disarm_watchdog) and
apply it to every place that runs synchronous V8 work: the post-load settle, both
navigation event-loop pumps, the whole script-execution phase, and --eval. A CLI
process-level hard deadline backstops the rest, since a hang inside a Rust op
cannot be cancelled by tokio or terminate_execution.
Also harden against panics: op_dom is wrapped in catch_unwind so a DOM-op panic
degrades to a null result instead of unwinding into V8's FFI frame and aborting
the process; the append_text unwrap is removed; panic = unwind is pinned in the
release profile.
Verified: obstacle course 33/33, no perf regression (per-op deltas within the
measurement noise floor), 6 of 8 previously-hanging real sites now render and the
other 2 exit deterministically at the hard deadline.
* fix(dom): reject cyclic reparenting so descendants() cannot hang the engine
appendChild/insertBefore only guarded the self-append case (parent == child),
never the case where the moved node is an ancestor of the new parent. When page
JS does node.appendChild(ancestorOfNode) the parent/child graph becomes cyclic;
the next querySelector runs descendants() (a DFS with no visited set) and loops
forever, growing a Vec unbounded and pinning the thread in native Rust where
neither tokio nor the V8 watchdog can interrupt it. This hung techcrunch and
kayak.
Reject the cycle at the mutation site per the DOM HierarchyRequestError rule: if
the moved node is an inclusive ancestor of its target parent, treat the move as a
no-op. The walk is gated on the moved node having children, so a fresh/leaf
append (the common case) stays O(1); only relocating a populated subtree pays
O(depth). descendants() also gets a result-length cap as defense in depth so a
cycle from any future path cannot wedge the engine.
techcrunch and kayak now render (37K and 1.26M chars). Verified: new
test_reparent_cycle_is_rejected, obstacle course 33/33, no perf regression.
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
* conformance: HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser, fetch UA
Next WPT Core batch, all verified live and benchmarked for no hot-path regression.
DOM:
- compareDocumentPosition rewritten to spec: real PRECEDING/FOLLOWING via the
compare_order op, CONTAINS/CONTAINED_BY, and DISCONNECTED|IMPLEMENTATION_SPECIFIC
for different roots. Adds the DOCUMENT_POSITION_* and missing nodeType constants
on Node and its prototype. WPT getPosition() depends on this, so it cascades into
the Range comparePoint/compareBoundaryPoints/isPointInRange expectations.
- HTML elements now ASCII-lowercase attribute names in getAttribute/setAttribute/
removeAttribute/hasAttribute, so el.setAttribute('accessKey') reflects via
getAttribute('accesskey'). The fold is gated behind a cheap uppercase scan and a
namespaceURI HTML guard (SVG/foreign attributes keep their case), and getAttribute
only folds on a miss, so the all-lowercase hot path is unchanged (benchmarked flat).
- Range comparePoint/isPointInRange ToUint32-coerce the offset (negative throws
IndexSizeError); createTreeWalker/createNodeIterator keep an explicit whatToShow of 0.
URL:
- URLSearchParams uses a proper form-urlencoded byte decoder (partial/invalid escapes
stay literal instead of decodeURIComponent throwing on the whole string), 2-arg
has(name,value)/delete(name,value), and a sequence-pair TypeError. Adds URL.parse
and URL.canParse statics.
Other:
- fetch()/XHR send a default Chrome User-Agent (the navigation path set one but
scripted requests went out with none, so UA-gated servers rejected them); honors an
explicit override.
- DOMParser documents expose URL/compatMode/characterSet/readyState/styleSheets/
defaultView and createTreeWalker/createNodeIterator; XMLDocument is defined globally.
* perf(dom): O(1) querySelector('#id') via the id index
querySelector('#id') went through the full selector engine, scanning descendants
in tree order until the first match (O(n), ~68us per call over a 2000-element
document in the micro-bench). A bare ASCII id selector now resolves through the
existing id index in O(1): ~68us -> ~1.25us per call, about 55x faster, with no
change to any other path (create/getAttribute/setAttribute/querySelectorAll/href
all flat). querySelector by id is one of the hottest scraping operations.
The fast path is conservative: only a bare `#name` selector with an ASCII
letter/underscore-led identifier takes it; escapes, combinators, commas,
whitespace, compounds, and non-ASCII fall through to the full engine unchanged.
For a scoped element.querySelector it returns the indexed element only when it is
a strict descendant of the scope, and falls back to the scan otherwise, so
duplicate-id scoped queries stay correct.
Also makes the id index keep the FIRST element in tree order per id (or_insert at
parse time) instead of the last, so getElementById and querySelector('#id') return
the first-in-tree-order element on duplicate ids, per spec (previously last).
SGavrl