485497393d4b8b1d082f34acb9cf096a8702b252
43
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrey Mnatsakanov
|
1a5c27a592 |
fix(js): honour NodeFilter.FILTER_REJECT in TreeWalker (#461)
`_accept` collapsed every non-ACCEPT filter result into a single boolean, so FILTER_REJECT behaved exactly like FILTER_SKIP: the node was not returned, but the walk still descended into its subtree. Per DOM 6.1, FILTER_REJECT rejects a node *and* its descendants. This was unreachable until #440 exposed the real NodeFilter constants, which made the canonical prune idiom if (node.tagName === 'SCRIPT') return NodeFilter.FILTER_REJECT; start yielding nodes from inside the pruned subtree instead of failing loudly — sanitizers and text extractors built on TreeWalker silently got extra content. Split the filter into a three-valued `_filter` (keeping `_accept` as the boolean wrapper the sibling/child movers use) and give nextNode a second traversal step for the reject case. That step lives in the DOM layer as `DomTree::next_after_subtree`, mirroring how #447 put `next_in_subtree` there: both share the ancestor climb, and REJECT costs no more JS/native crossings than SKIP. NodeIterator shares this walker but has no subtree pruning (DOM 6.2: FILTER_REJECT behaves as FILTER_SKIP), so `createNodeIterator` marks the walker `_rejectIsSkip` and keeps the plain step. Covered by its own test so the shared implementation cannot drift. |
||
|
SGavrl
|
c12915a1a6 | fix(js): keep TreeWalker traversal in document order (#432) | ||
|
Jim Zhang @ wonderhoi
|
f5dc97b131 |
fix(net): scope passive callbacks to the registering page (#414)
The on_request/on_response Vec lived on the ObscuraHttpClient shared by every page in the context, so a callback registered on one page fired for all pages (#408 cross-page bleed) and outlived its page. Callbacks now live in a per-page CallbackRegistry (Arc shared with the page's JS runtime state). Page-driven fetch paths (navigation, POST, scripts, stylesheets, robots.txt, module loads, JS fetch/XHR in both plain and stealth modes) pass the registry through; the HTTP client itself is callback-free. Adds callbacks_do_not_bleed_across_pages: page A's callback stays silent for page B's requests and stops firing after page A is dropped. |
||
|
SGavrl
|
99ece0e12d |
feat(net): make passive request/response callbacks detachable (#416)
on_request/on_response stored callbacks in append-only Vecs with no way to remove one, so a crawler that installed a response-body capture hook for one phase could never detach it: it fired for the shared client's whole lifetime, re-registering across navigations piled up duplicates, and (because the client is shared per context) it saw sibling pages' requests too. Tag each callback with a stable id. on_request/on_response now return the id and off_request/off_response (remove_on_request/remove_on_response on the client) detach a specific one. Storage is Vec<(u64, Callback)>; all fire sites destructure the tuple. Closes #408. |
||
|
SGavrl
|
d0b6fc4ffd |
fix(cdp): emit Network events for script-initiated requests (#415)
Requests made by page JS (fetch/XHR/dynamic script and stylesheet loads)
run in the V8 op layer and never produced a NetworkEvent, so CDP emitted
Network.requestWillBeSent / responseReceived only for the static
navigation subresources. Puppeteer/Playwright page.on('request') and
on('response') therefore missed every XHR/JSON response, which breaks
response capture (and is the root cause of the Aviasales half of #394).
op_fetch_url now records a JsNetworkEvent keyed by the same fetch-{N} id
under which it already stores the response body. The Page drains these
into its network_events before the CDP layer emits, so they flow through
emit_navigation_events as requestWillBeSent / responseReceived /
loadingFinished alongside the document subresources, and
Network.getResponseBody resolves for them via the shared id. Applied to
both navigation paths (server process_navigation and in-process
do_navigate). The queue is capped at 4096 events per page.
Closes #406.
|
||
|
MrBeanDev
|
ed75730e2d |
feat(js): implement WebCrypto SubtleCrypto secret-key algorithms
crypto.subtle only implemented digest. sign returned a fixed 32-byte buffer, verify always returned true, encrypt and decrypt threw, and generateKey, importKey and deriveBits handed back placeholder data. Pages that use WebCrypto (anti-bot probes, PKCE and other auth flows, client-side crypto) silently got wrong results. Implement the secret-key half of WebCrypto on RustCrypto, following the existing op_subtle_digest pattern: - HMAC sign and verify (SHA-1/256/384/512) - AES-GCM, AES-CBC and AES-CTR encrypt and decrypt - PBKDF2 and HKDF deriveBits and deriveKey - raw and symmetric JWK (oct) importKey and exportKey, generateKey, wrapKey and unwrapKey - getRandomValues and randomUUID now draw from the OS CSPRNG instead of Math.random, so they are uniform across typed-array widths and no longer a fingerprinting tell Unsupported algorithms (RSA, ECDSA, ECDH) and key formats (pkcs8, spki) throw NotSupportedError instead of returning fake data. The crypto runs in stateless Rust ops; the bootstrap.js shim only marshals bytes and normalizes algorithm parameters, so the native fast paths are untouched and there is no cost unless a page calls crypto. Verified against RFC 4231 (HMAC), the NIST GCM vectors, NIST SP800-38A (AES-CTR), RFC 6070 (PBKDF2) and RFC 5869 (HKDF), plus roundtrip and tag/padding tamper-rejection checks. |
||
|
SGavrl
|
bd39512ffb |
api: re-validate intercept URL rewrites against the SSRF gate
A Continue resolution that rewrites the request URL applied the new URL after validate_fetch_url had already run on the original, so a rewrite could reach a private or internal address the original request was blocked from. Run the overridden URL back through validate_fetch_url, matching how redirects are re-validated, and return a blocked response if it fails. Also fix a doc comment that ended up attached to the new fetch_response helper instead of stealth_fetch_all. |
||
|
SGavrl
|
c19a21c142 |
api: honor interception Continue overrides and fix pre-navigation enable
op_fetch_url ignored the url/method/headers/body fields on InterceptResolution::Continue, so the interception channel could pass, mock, or block a request but not rewrite it. Apply them: shadow url/method/body and derive custom_headers from the override after the channel resolves Continue. Also fix enable_interception() called before the first navigation: init_js propagated intercept_tx to the new runtime but not intercept_enabled, so the runtime started with interception disabled and op_fetch_url never intercepted. Propagate intercept_enabled in init_js as well. Add a test that rewrites a fetch's url via Continue and asserts the rewritten response is captured. |
||
|
SGavrl
|
1f5066fc30 |
api: add request/response interception API to Page (#306)
Expose the interception infrastructure that already existed internally but was
private, so embedders can intercept, observe, mock, or block JS fetch()/XHR
without forking or unsafe casts.
On obscura::Page:
- add_preload_script: inject JS before any page <script> (CDP
Page.addScriptToEvaluateOnNewDocument). The field already ran the scripts;
this only adds the method.
- enable_interception: returns a receiver for every JS fetch()/XHR request;
resolve each through its resolver with InterceptResolution::{Continue,
Fulfill, Fail} to pass, mock, or block (CDP Fetch.enable equivalent).
- on_request / on_response: passive callbacks. on_response is the main path for
capturing API response payloads from SPAs.
on_request/on_response previously fired only for navigation; op_fetch_url now
invokes them for JS fetch()/XHR too (both reqwest and stealth paths), gated on
a non-empty callback vector so the common no-callback case pays nothing.
Re-export InterceptedRequest, InterceptResolution, RequestInfo, ResourceType,
RequestCallback, ResponseCallback, Response from the obscura crate.
Integration test tests/interception.rs exercises all three against a local echo
server.
|
||
|
SGavrl
|
8582fcceca |
Merge stealth-overhaul into main
Resolved conflicts: - page.rs: keep both set_blocked_urls (#315) and stealth client wiring - ops.rs: keep SHA-512/224 + SHA-512/256 from stealth branch; keep the unreachable catch-all (vec![]) from #314 instead of silent SHA-256 fallback; updated comment accordingly - bootstrap.js: extend the digest algorithm allowlist to include SHA-512/224 and SHA-512/256 to match the Rust additions |
||
|
SGavrl
|
2e0981ef3f |
stealth: route scripted fetch()/XHR through the wreq client
op_fetch_url backed JS fetch() and XHR with a plain reqwest/rustls client, so in stealth mode scripted requests carried a non-browser TLS fingerprint and none of the Chrome client hints the main navigation sends. Bot managers that score subresource requests (the AWS WAF challenge verify call, Akamai sensors) read that as a script and reject it. Add StealthHttpClient::send_single and a stealth_fetch_all path that op_fetch_url uses when a stealth client is installed, preserving the redirect, SSRF, and CORS handling so JS-level requests share the navigation's Chrome TLS fingerprint and sec-ch-ua client hints. The reqwest path is unchanged for non-stealth. Adds a stealth feature to obscura-js gated on obscura-net/stealth. |
||
|
SGavrl
|
dc780d77e4 |
js: handle SHA-512/224 and SHA-512/256 in crypto.subtle.digest
The digest op fell through to SHA-256 for any unrecognized algorithm name, so SHA-512/224 and SHA-512/256 silently returned a SHA-256 hash. Add the two FIPS 180-4 truncated variants, which the AWS WAF challenge proof-of-work (used by booking.com and other CloudFront/WAF sites) requires. Fixes #314. |
||
|
Gabriel
|
edde67ddd4 |
crypto.subtle.digest: reject unsupported algorithms (#314) (#319)
Unknown algorithm names (SHA-512/224, MD5, etc.) were silently falling
back to SHA-256, producing wrong bytes with no error. Now throws
DOMException('NotSupportedError') to match Chrome behavior.
Supported: SHA-1, SHA-256, SHA-384, SHA-512.
|
||
|
RohitMadhu
|
3bf0da2fae |
Add Network.setBlockedURLs support (#315)
Added Network.setBlockedURLs dispatch handling. Stores blocked URL patterns on each page. Applies blocking to page subresources and JS fetch / XHR requests. Supports wildcard CDP-style patterns like *://*.gstatic.com/*.woff2. |
||
|
SGavrl
|
ab27e9153d |
cli: list fetch()/XHR resources in --dump assets
--dump assets only walked static DOM attributes (script[src], img[src], ...), so resources a page pulled in with fetch() or XHR never showed up (issue #301). Record each absolute URL requested through op_fetch_url in ObscuraState, expose it via the runtime and Page, and append those URLs to --dump assets as type "fetch", deduped against URLs already listed from the DOM. |
||
|
Rohit Madhu
|
570ca1544c | add Network.getResponseBody support | ||
|
SGavrl
|
93b2fba6a1 |
fix: creepjs stealth, AudioBuffer, Crypto, Worker, hasIframeProxy
Worker auto-execution: Worker constructor resolves relative URLs against
location.href, fetches the script, and auto-runs it. WorkerGlobalScope is
exposed so IS_WORKER_SCOPE resolves correctly. Worker data (navigator,
locale, timezone) now arrives via postMessage within the 3s timeout.
AudioBuffer class added to globalThis with getChannelData, copyFromChannel,
copyToChannel. AudioContext.createBuffer returns a real AudioBuffer. Fixes
ReferenceError that blocked the audio fingerprint section.
Crypto class added to globalThis with randomUUID and getRandomValues on its
prototype. Fixes ReferenceError in getPlatformEstimate.
AudioContext.addEventListener was a no-op; startRendering only fired
oncomplete. Add _listeners map and dispatch to stored listeners.
getElementById and querySelector('#id') returned null for elements inserted
via replaceChild(documentFragment, el). id_index only registers nodes at
creation time; .or_insert skipped IDs on cloned fragments; None caused
query_selector_from to short-circuit the full scan.
Fix querySelector: fall through to full scan on index miss or stale entry.
Fix getElementById: verify node is in live document; fall back to attribute
selector scan if not.
contentWindow getter: return null when parentNode is null (iframe was never
inserted into any tree). Previously returning null for all disconnected
iframes broke getBehemothIframe, which caused PHANTOM_DARKNESS to fall back
to self (main window) instead of the phantom iframe. That made DOC point to
the main document, so rect and font measurements wrote fragment nodes into
the main body, and the subsequent full-DOM scan in getElementById could not
find fingerprint-data through nested fragments in time.
Limiting the null return to parentNode===null (never inserted) means
getBehemothIframe gets a valid nested contentWindow as expected. creepjs
now reports hasIframeProxy: false and 0% stealth detection.
DocumentFragment constructor: pass nid through when provided.
Range.getBoundingClientRect / getClientRects: return commonAncestor rect.
|
||
|
SGavrl
|
fa4928faa6 | Trim redundant comments | ||
|
charlesmamane26-sketchandcharlesmamane26-sketch
|
6ba3c63596 |
Security: block SSRF DNS-rebinding and widen the IP deny-set (#279)
Add is_forbidden_ip() (loopback, RFC1918, link-local incl. 169.254.169.254, broadcast, documentation, 0.0.0.0/::, IPv6 unique-local fc00::/7, IPv4-mapped IPv6) and SsrfGuardResolver (impl reqwest::dns::Resolve) that resolves and rejects forbidden addresses at connect time, closing the DNS-rebinding bypass. Wired into the navigation client and the JS fetch()/XHR path. The --allow-private-network / OBSCURA_ALLOW_PRIVATE_NETWORK escape hatch is kept. Co-authored-by: charlesmamane26-sketch <[email protected]> |
||
|
SGavrl
|
5135e3cb4d |
perf: DOM read fast paths (constant nodeType, borrowed op reads, base64 table) (#259)
* DOM read fast paths: constant nodeType, fewer ops and allocations - Element.nodeType is now a constant getter instead of inheriting Node's dynamic op-backed getter. nodeType is invariant (1) for element wrappers (wrappers are only built for element nodes and node ids are never freed and reused), so this drops one JS/Rust op per read on a pervasive path. Micro-bench: ~600k nodeType reads go from ~380ms to ~5ms. - attributes getter reads each attribute once instead of twice. - dataset Proxy is cached per element instead of reallocated on every access, which also makes el.dataset === el.dataset (spec-correct). - stylesheet <link> scan borrows the node (with_node) instead of deep cloning it and compares rel via eq_ignore_ascii_case, dropping a clone and a lowercased String alloc per link per navigation. No regression: obstacle course 33/33, WPT dom/nodes Node- subset identical to baseline (3551/4163 subtests), targeted DOM probe identical. * DOM op reads: borrow the node instead of deep-cloning it The scalar read arms in op_dom (node_type, node_name, tag_name, get_attribute, attribute_names, parent/first/last/next/prev lookups, has_child_nodes, pi_target, doctype_name, doctype_public_id) and the node_root / node_ancestors / node_child_index walks called get_node, which clones the whole Node (for elements a deep clone of the attribute Vec and its Strings) just to read one field. Switch them to with_node, which borrows under the RefCell. element_children and the document_element / document_doctype child scans stay on get_node on purpose (they iterate, so the borrow story is left untouched). Output is identical. Traversal/attribute micro-bench ~25-30% faster with fewer allocations per op. No regression: obstacle course 33/33, WPT dom/nodes identical to baseline (7057/10620 across 314 files). * base64 decode: reverse-lookup table instead of indexOf scan _base64ToUint8Array decodes every fetch/XHR response body. It ran an O(64) alphabet.indexOf plus a one-char substring alloc per character. Build a charCode->6-bit reverse table once at module load and index it by charCodeAt, folding out-of-range (NaN or code >= 128) to -1 so the bitmath stays byte-identical to the old indexOf path on valid and malformed input alike. ~20-25% faster decoding a 2MB body; output byte-identical (differential test across valid, unpadded, '='-position, >127, non-alphabet and truncated inputs matches the old code exactly). No regression: obstacle course 33/33. |
||
|
SGavrl
|
ce67008072 |
fix(js): time out scripted fetch/XHR so a hung request fires error, not nothing (#251) (#252)
build_request_client (the client behind every JS fetch(), XHR, and ES-module load) set no request timeout. A cross-origin XHR with a non-simple header triggers a CORS preflight OPTIONS; if the server accepts the connection but never answers it, op_fetch_url's send().await hung forever, the fetch promise never settled, and the XHR was stuck at readyState 1 with no completion event ever firing. That stranded Angular HttpClient on live.deutsche-boerse.com. Give the client a 30s request timeout (matching the other clients in the workspace); OBSCURA_FETCH_TIMEOUT_MS overrides it. On timeout the request errors, op_fetch_url propagates it, and the fetch shim turns it into an XHR error/loadend, so readyState advances to 4 and Angular's error path runs. Verified: a server that hangs on OPTIONS now makes the XHR fire error at the timeout (was stuck forever); a CORS-allowed cross-origin XHR still succeeds with load/200; obstacle course 33/33. Closes #251 |
||
|
SGavrl
|
1e9e9478fb |
Fix/v8 watchdog panic safety (#239)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
* conformance: HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser, fetch UA
Next WPT Core batch, all verified live and benchmarked for no hot-path regression.
DOM:
- compareDocumentPosition rewritten to spec: real PRECEDING/FOLLOWING via the
compare_order op, CONTAINS/CONTAINED_BY, and DISCONNECTED|IMPLEMENTATION_SPECIFIC
for different roots. Adds the DOCUMENT_POSITION_* and missing nodeType constants
on Node and its prototype. WPT getPosition() depends on this, so it cascades into
the Range comparePoint/compareBoundaryPoints/isPointInRange expectations.
- HTML elements now ASCII-lowercase attribute names in getAttribute/setAttribute/
removeAttribute/hasAttribute, so el.setAttribute('accessKey') reflects via
getAttribute('accesskey'). The fold is gated behind a cheap uppercase scan and a
namespaceURI HTML guard (SVG/foreign attributes keep their case), and getAttribute
only folds on a miss, so the all-lowercase hot path is unchanged (benchmarked flat).
- Range comparePoint/isPointInRange ToUint32-coerce the offset (negative throws
IndexSizeError); createTreeWalker/createNodeIterator keep an explicit whatToShow of 0.
URL:
- URLSearchParams uses a proper form-urlencoded byte decoder (partial/invalid escapes
stay literal instead of decodeURIComponent throwing on the whole string), 2-arg
has(name,value)/delete(name,value), and a sequence-pair TypeError. Adds URL.parse
and URL.canParse statics.
Other:
- fetch()/XHR send a default Chrome User-Agent (the navigation path set one but
scripted requests went out with none, so UA-gated servers rejected them); honors an
explicit override.
- DOMParser documents expose URL/compatMode/characterSet/readyState/styleSheets/
defaultView and createTreeWalker/createNodeIterator; XMLDocument is defined globally.
* perf(dom): O(1) querySelector('#id') via the id index
querySelector('#id') went through the full selector engine, scanning descendants
in tree order until the first match (O(n), ~68us per call over a 2000-element
document in the micro-bench). A bare ASCII id selector now resolves through the
existing id index in O(1): ~68us -> ~1.25us per call, about 55x faster, with no
change to any other path (create/getAttribute/setAttribute/querySelectorAll/href
all flat). querySelector by id is one of the hottest scraping operations.
The fast path is conservative: only a bare `#name` selector with an ASCII
letter/underscore-led identifier takes it; escapes, combinators, commas,
whitespace, compounds, and non-ASCII fall through to the full engine unchanged.
For a scoped element.querySelector it returns the indexed element only when it is
a strict descendant of the scope, and falls back to the scan otherwise, so
duplicate-id scoped queries stay correct.
Also makes the id index keep the FIRST element in tree order per id (or_insert at
parse time) instead of the last, so getElementById and querySelector('#id') return
the first-in-tree-order element on duplicate ids, per spec (previously last).
* fix: bound synchronous V8 hangs with a termination watchdog and add op panic safety
Heavy pages could hang obscura fetch indefinitely: tokio::time::timeout only
cancels at await points, so synchronous V8 work (a page's script bootstrap, a
microtask storm) pinned the thread and ran unbounded.
Add a reusable V8 terminate_execution watchdog (arm_watchdog/disarm_watchdog) and
apply it to every place that runs synchronous V8 work: the post-load settle, both
navigation event-loop pumps, the whole script-execution phase, and --eval. A CLI
process-level hard deadline backstops the rest, since a hang inside a Rust op
cannot be cancelled by tokio or terminate_execution.
Also harden against panics: op_dom is wrapped in catch_unwind so a DOM-op panic
degrades to a null result instead of unwinding into V8's FFI frame and aborting
the process; the append_text unwrap is removed; panic = unwind is pinned in the
release profile.
Verified: obstacle course 33/33, no perf regression (per-op deltas within the
measurement noise floor), 6 of 8 previously-hanging real sites now render and the
other 2 exit deterministically at the hard deadline.
* fix(dom): reject cyclic reparenting so descendants() cannot hang the engine
appendChild/insertBefore only guarded the self-append case (parent == child),
never the case where the moved node is an ancestor of the new parent. When page
JS does node.appendChild(ancestorOfNode) the parent/child graph becomes cyclic;
the next querySelector runs descendants() (a DFS with no visited set) and loops
forever, growing a Vec unbounded and pinning the thread in native Rust where
neither tokio nor the V8 watchdog can interrupt it. This hung techcrunch and
kayak.
Reject the cycle at the mutation site per the DOM HierarchyRequestError rule: if
the moved node is an inclusive ancestor of its target parent, treat the move as a
no-op. The walk is gated on the moved node having children, so a fresh/leaf
append (the common case) stays O(1); only relocating a populated subtree pays
O(depth). descendants() also gets a result-length cap as defense in depth so a
cycle from any future path cannot wedge the engine.
techcrunch and kayak now render (37K and 1.26M chars). Verified: new
test_reparent_cycle_is_rejected, obstacle course 33/33, no perf regression.
|
||
|
SGavrl
|
a703b7ac30 |
WPT conformance + perf: O(1) querySelector('#id'), HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser (#238)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
* conformance: HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser, fetch UA
Next WPT Core batch, all verified live and benchmarked for no hot-path regression.
DOM:
- compareDocumentPosition rewritten to spec: real PRECEDING/FOLLOWING via the
compare_order op, CONTAINS/CONTAINED_BY, and DISCONNECTED|IMPLEMENTATION_SPECIFIC
for different roots. Adds the DOCUMENT_POSITION_* and missing nodeType constants
on Node and its prototype. WPT getPosition() depends on this, so it cascades into
the Range comparePoint/compareBoundaryPoints/isPointInRange expectations.
- HTML elements now ASCII-lowercase attribute names in getAttribute/setAttribute/
removeAttribute/hasAttribute, so el.setAttribute('accessKey') reflects via
getAttribute('accesskey'). The fold is gated behind a cheap uppercase scan and a
namespaceURI HTML guard (SVG/foreign attributes keep their case), and getAttribute
only folds on a miss, so the all-lowercase hot path is unchanged (benchmarked flat).
- Range comparePoint/isPointInRange ToUint32-coerce the offset (negative throws
IndexSizeError); createTreeWalker/createNodeIterator keep an explicit whatToShow of 0.
URL:
- URLSearchParams uses a proper form-urlencoded byte decoder (partial/invalid escapes
stay literal instead of decodeURIComponent throwing on the whole string), 2-arg
has(name,value)/delete(name,value), and a sequence-pair TypeError. Adds URL.parse
and URL.canParse statics.
Other:
- fetch()/XHR send a default Chrome User-Agent (the navigation path set one but
scripted requests went out with none, so UA-gated servers rejected them); honors an
explicit override.
- DOMParser documents expose URL/compatMode/characterSet/readyState/styleSheets/
defaultView and createTreeWalker/createNodeIterator; XMLDocument is defined globally.
* perf(dom): O(1) querySelector('#id') via the id index
querySelector('#id') went through the full selector engine, scanning descendants
in tree order until the first match (O(n), ~68us per call over a 2000-element
document in the micro-bench). A bare ASCII id selector now resolves through the
existing id index in O(1): ~68us -> ~1.25us per call, about 55x faster, with no
change to any other path (create/getAttribute/setAttribute/querySelectorAll/href
all flat). querySelector by id is one of the hottest scraping operations.
The fast path is conservative: only a bare `#name` selector with an ASCII
letter/underscore-led identifier takes it; escapes, combinators, commas,
whitespace, compounds, and non-ASCII fall through to the full engine unchanged.
For a scoped element.querySelector it returns the indexed element only when it is
a strict descendant of the scope, and falls back to the scan otherwise, so
duplicate-id scoped queries stay correct.
Also makes the id index keep the FIRST element in tree order per id (or_insert at
parse time) instead of the last, so getElementById and querySelector('#id') return
the first-in-tree-order element on duplicate ids, per spec (previously last).
|
||
|
SGavrl
|
3b2c6c0deb |
WPT conformance: legacy-charset URL encoding, IDL attribute reflection, FileAPI, Storage, Range (#237)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
|
||
|
SGavrl
|
9c3dd6b21a |
WPT conformance: dom/ranges, FileAPI, detached documents (#236)
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe createHTMLDocument returned a document whose body was null: parsing the skeleton string as innerHTML of <html> collapsed through the fragment parser and kept only <title>. Build head>title and body explicitly instead. The DOMParser doc node also gains appendChild/removeChild/insertBefore and a doctype getter, and createDocument carries its doctype argument. This lets dom/common.js setupRangeTests() finish (it builds fixtures with new Document(), createHTMLDocument(), createDocument()) instead of aborting at harness setup, so the dom/ranges files run. Range-collapse 186/186, Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17. The large O(ranges x points) matrix files (Range-set, comparePoint, compareBoundaryPoints) still time out: the boundary-point comparison walks the tree through per-step DOM ops, which is too slow for those matrices. Speeding that up (a Rust node-position op) is a separate change. Also add globalThis.ProgressEvent (Event subclass with lengthComputable/ loaded/total) and the static Document.parseHTMLUnsafe(html). * perf(dom): compute Range tree-order in Rust ops The Range boundary-point algorithms compared nodes by walking ancestor chains and sibling lists from JS, one DOM op per step. The dom/ranges matrices (testRanges x testPoints, thousands of comparisons) made that time out. Add node_index, compare_order, and node_root ops that do the walk in Rust over the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex, _rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange 5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s where they previously did not finish. Not on any engine hot path (querySelector, mutation, navigation, fetch do not use these). * fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob Blob was a one-line stub that string-joined its parts (wrong size for binary or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size, slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared when non-ASCII), the endings option (native CRLF normalization), and Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and its own toStringTag. FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws InvalidStateError when a read starts while already LOADING, and fires loadstart/progress/load/loadend (and abort) to both on* handlers and addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct results. Request gains blob(). FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice 141/150. |
||
|
SGavrl
|
07c6e735fa |
WPT conformance: WHATWG URL, DOM Range, and post-navigation event-loop settle (#233)
* fix: restore regressions introduced by #227 #227 deleted build.rs (reverting #224's version-from-tag feature), removed DOM cycle guards in append_child/insert_before that prevent infinite loops on self-append/self-insert, dropped the CharacterData textContent fast-path, removed PI/Doctype Rust ops, and cut the env_allows_private_network() check from validate_fetch_url. Restored: - crates/obscura-cli/build.rs (version from tag/env) - OBSCURA_BUILD_VERSION in version string and banner - allow_private_network env var propagation in main.rs - append_child/insert_before self-cycle guards in tree.rs - textContent CharacterData fast-path in tree.rs - create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops in ops.rs - ProcessingInstruction branch in set_text_content op - env_allows_private_network() check in validate_fetch_url * fix(js): Europe/Berlin timezone, plugins iterator recursion Two bugs in the stealth bootstrap code from #227: - 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat silently fell back to UTC. Changed to 'Europe/Berlin'. - plugins[Symbol.iterator] called this[Symbol.iterator]() recursively, blowing the stack on Array.from(navigator.plugins). Fixed to bind Array.prototype[Symbol.iterator] directly to the array. * url: back URL/URLSearchParams with the url crate, add <a>/<area> decomposition The URL global was a regex shim: it mis-parsed userinfo (hostname came back as the username, port as "pass@host"), had no username/password, and its setters were no-ops. Replace it with two ops (op_url_parse, op_url_set) backed by the WHATWG-compliant url crate, and add the HTMLHyperlinkElementUtils members (protocol/host/hostname/port/pathname/search/hash/origin/username/password and their setters) to <a> and <area>, which had none. This was the single largest WPT failure. a.search returned undefined, so the encoding encode-href tests threw "Cannot read properties of undefined (reading 'substr')" about 225k times. With the fix url-constructor passes 831/888, url-setters 192/279, urlsearchparams-sort 17/17, url/a-element 415/887, and the encode-href files run their assertions instead of crashing. op_url_set runs the url crate inside catch_unwind: a few setters panic on the pathological inputs the url-setters tests use, and a panicking setter must be a no-op (the WHATWG "do nothing on invalid value" rule) rather than abort. URLSearchParams also gained getAll/sort/size/keys/values/entries/iterator and a back-reference so mutations write through to the bound URL. * dom: real Range/StaticRange, createDocumentType, XML Document type Range was a no-op stub (setStart/setEnd discarded their arguments, so startContainer/endContainer read back undefined and every dom/ranges helper that walked range.endContainer crashed with "reading 'nodeType' of undefined"). Replace it with a property-storing Range over the live tree: setStart/End and the *Before/*After/selectNode/selectNodeContents/collapse positioning, plus the boundary-point algorithms comparePoint, compareBoundaryPoints, isPointInRange, intersectsNode, commonAncestorContainer, cloneRange and a real stringifier, all built on a correct preorder tree-order comparison. Add StaticRange. new Document() now reports an XML contentType (it has no backing node id), so createCDATASection no longer throws NotSupportedError on it. Add DOMImplementation.createDocumentType with the lenient name validation browsers use and an owner-document backref so DocumentType.ownerDocument is correct. StaticRange-constructor 14/17. This does not fully unblock dom/ranges yet: dom/common.js also needs createHTMLDocument()/createDocument() to return documents with a working body/createComment/etc., which is a separate fix. * fix(browser,cli): settle the event loop after navigation obscura fetch read the page at the load event and never pumped the event loop afterward on the eval/dump path, so --wait was effectively ignored there. Pages that schedule timers, do async work, or only report through a completion callback that runs after load (testharness add_completion_callback, SPAs) were observed frozen at the load event and never produced their result. Add Page::settle, which drives run_event_loop until the loop goes idle or --wait elapses (returns early when idle, so static pages stay fast), and run it after navigation in the fetch path. |
||
|
SGavrl
|
6b3cc1a686 |
Fix/rescue 227 regressions (#232)
* fix: restore regressions introduced by #227 #227 deleted build.rs (reverting #224's version-from-tag feature), removed DOM cycle guards in append_child/insert_before that prevent infinite loops on self-append/self-insert, dropped the CharacterData textContent fast-path, removed PI/Doctype Rust ops, and cut the env_allows_private_network() check from validate_fetch_url. Restored: - crates/obscura-cli/build.rs (version from tag/env) - OBSCURA_BUILD_VERSION in version string and banner - allow_private_network env var propagation in main.rs - append_child/insert_before self-cycle guards in tree.rs - textContent CharacterData fast-path in tree.rs - create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops in ops.rs - ProcessingInstruction branch in set_text_content op - env_allows_private_network() check in validate_fetch_url * fix(js): Europe/Berlin timezone, plugins iterator recursion Two bugs in the stealth bootstrap code from #227: - 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat silently fell back to UTC. Changed to 'Europe/Berlin'. - plugins[Symbol.iterator] called this[Symbol.iterator]() recursively, blowing the stack on Array.from(navigator.plugins). Fixed to bind Array.prototype[Symbol.iterator] directly to the array. |
||
|
Sander Spilleman
|
9ae36764da | feat(stealth): many stealth improvements | ||
|
SGavrl
|
ee10e8ddb5 |
feat(engine): DOM + AbortSignal + storage fixes surfaced by WPT conformance work (#219)
Engine-only changes pulled out of the WPT conformance branch. The tests, runner, and harness vendoring that drove these have moved to a separate benchmark repo (obscura-bench). Real bug fixes: - Tree::insert_before created a self-cycle when the inserted node was its own reference (the spec-conformant `child.after(x)` path that the testharness's ChildNode helpers exercise). The fixup wrote node.prev_sibling = node.next_sibling = own id; every subsequent childNodes / querySelectorAll walk then looped forever and obscura burned RAM until OOM. Per spec the op is a no-op; we now early-return. - Tree::text_content extended to return own data for direct CharacterData access (Text / Comment / ProcessingInstruction). - Element.namespaceURI now reflects the namespace passed to createElementNS instead of collapsing every non-svg namespace to xhtml. Without this, code that creates elements in MathML, custom XML namespaces, or the empty namespace gets misclassified as HTML. - AbortController / AbortSignal: the previous shim was a plain object whose addEventListener was a no-op, so listeners for 'abort' silently never fired and any AbortSignal.timeout / AbortSignal.any chains broke. New impl tracks aborted/reason, has a real addEventListener / removeEventListener / dispatchEvent surface, honors `once`, and implements the static abort / timeout / any factories crawlers use for budgeted fetches. - StorageEvent class added. localStorage / sessionStorage mutations now dispatch 'storage' events on the parent window and every iframe window so cross-frame onstorage listeners fire. - ProcessingInstruction (nodeType 7) and DocumentType (nodeType 10) constructors plus document.createProcessingInstruction and document.implementation.createDocumentType. textContent returns null for Document and DocumentType per spec. - insertAdjacentText / insertAdjacentElement on Element. Without these the testharness output renderer threw in a forEach without try/catch, breaking the completion callback chain for every test with subtests. - Window + Node addEventListener filter non-callable listeners, accept the EventListenerObject shape (object with handleEvent), dedupe by (type, callback, capture), honor the `once` option, and snapshot the listener list before dispatch so a `once: true` removal mid-iteration doesn't corrupt the loop. - Iframe HTML loader now extracts head <script> blocks and body on-handler attributes (`<body onstorage=...>`), evals scripts in the parent realm after the iframe window is built, re-attaches handlers as window listeners. Same iframe element reusing its src no longer double-registers in the iframe registry. - ops.rs validate_fetch_url honors the OBSCURA_ALLOW_PRIVATE_NETWORK env var, and the CLI --allow-private-network flag mirrors itself into that env so the js-side fetch gate matches the http_client gate. Without this, iframe loads of 127.0.0.1 resources were blocked even with the flag set. - ops.rs: create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops added; set_text_content extended to handle PI nodes. |
||
|
SGavrl
|
fd2421940b |
Perf/cdp fetch cache and cookie alloc (#216)
* perf: cache HTTP clients + selector parsing + exponential wait backoff Three low-risk wins discovered while reading the hot paths. 1. op_fetch_url cached its reqwest::Client. Previously every JS-side fetch(), XHR, and dynamic script load built a fresh client, which re-loaded TLS roots and started with an empty connection pool. On an asset-heavy page with 30+ subresources this added ~150ms of wasted setup with zero connection reuse. The cache is a process- wide map keyed by proxy URL (empty string for no-proxy), so the first fetch on each proxy pays the build cost once and every subsequent fetch reuses the same pool. Also bumped pool_idle_timeout to 5 minutes and tcp_keepalive to 60s so connections stay warm for SPA-heavy workloads that hit the same origin dozens of times across a navigation. 2. DomTree::parse_selector now caches parsed selectors in a thread- local HashMap (cap 256). querySelector / querySelectorAll re- parsed the selector string every call before; batch DOM access patterns (an agent scraping a table row-by-row, a framework polling for elements) paid the cssparser cost N times. 3. browser_wait_for / browser_wait_for_text now exponentially back off (5 -> 10 -> 20 -> ... -> 200 ms). The old fixed 200ms tick meant a selector that appeared 30ms after the call still took the next full 200ms tick to be detected. Worst case is still 200ms per check, but the median wait drops sharply for fast appearances. 28/28 integration tests pass. Asset-heavy bench (rust-lang/rust, docs.stripe.com, etc) doesn't regress. * perf: module loader client cache, fetch concurrency cap, snapshot hide list, cookie alloc Round two of perf wins after a deep code audit. 1. ES module loader (module_loader.rs) was building a fresh reqwest::Client on every dynamic import. Modern SPAs (Next, Vite) load 20-50 chunks via dynamic import; each one used to spin up an empty connection pool with fresh TLS init. Now reuses the process-wide FETCH_CLIENT_CACHE that op_fetch_url already uses (the cached_request_client helper). Saves ~30-100ms on chunk- heavy first navigations. 2. Script and CSS fetches in execute_scripts used futures::join_all with unbounded concurrency. A 100-script page would open 100 TCP sockets at once, exhausting the connection pool and the OS ephemeral port range. Capped at 16 in-flight (well above the per- host pool ceiling real Chrome uses). 3. bootstrap.js __obscura_init() ran Object.keys(globalThis).filter() on every page setup to find internals to hide. The set of keys is static at snapshot-build time, so we now precompute it once (top-level statement just below the function) and __obscura_init just iterates the precomputed array. Was 5-40ms per page on heavy SPAs (1000+ globals); now ~0.2ms per page. 4. CookieJar::domain_matches allocated 2 lowercase Strings and a format! per call. The lookup runs per fetch per cookie domain; on a heavily-cookied site with 100 cookies across 30 domains that was thousands of allocations per page. Now zero-alloc with eq_ignore_ascii_case and char-boundary checks. |
||
|
SGavrl
|
b98e457428 |
engine: large compat + reliability pass (#206)
CDP / navigation: - Route all Page.navigate through process_with_interception so the parent dispatcher stays responsive during long navs (fixes Target.createTarget timeouts after a handful of newPage calls). - Default waitUntil to domcontentloaded. Puppeteer and Playwright don't pass it server-side and wait client-side via lifecycleEvent, so the old Load default blocked nearly every nav on slow sites. - 30s navigation hard ceiling, OBSCURA_NAV_TIMEOUT_MS to override. - execute_scripts loop now checks wall-clock deadline per iteration. - Emit Network.requestWillBeSent before Page.frameNavigated so Playwright's FrameManager attaches the navigation response (#189). - Use loaderId as the Document request id so page.goto() resolves to a Response instead of null (#189). - Fresh executionContextId per navigation, no more 'Cannot find context with specified id' after the first nav (#192). - POST navigation params plumbed through the intercept path. DOM / Web API surface: - MutationObserver: subtree filter walks parent chain, fires on textContent / CharacterData.data / innerHTML, type filter respected, oldValue passed. - IntersectionObserver: multi-fire (initial + 120/500/1500ms), disconnect / unobserve / takeRecords, root / rootMargin / thresholds. - ResizeObserver: per-target firing from getBoundingClientRect, drop the no-op override that was clobbering the real class. - crypto.subtle.digest: real SHA-1/256/384/512 via op_subtle_digest. - DOMParser / document.implementation.createHTMLDocument / createDocument: detached doc with body, head, documentElement, querySelector*, getElementById, getElementsByTagName. - WebSocket / EventSource / BroadcastChannel: real EventTarget, open fires on microtask. - indexedDB: request, object store, transaction, database with async onsuccess and in-memory Map storage. - getComputedStyle: width/height/top/left from bbox so React virtualization libs render items. Site compatibility went from 3/40 to 175/175 reachable sites in the sweep. Perf is equal or faster than pristine main on every site tested (DCL default removes the block on long-tail subresources). 28/28 integration tests pass. |
||
|
|
ea89f050cd |
Fix puppeteer exposeFunction end-to-end: real bindings, preload timing, URL coercion (#166)
* fix(js): coerce URL constructor input/base to string
The polyfill at bootstrap.js called `base.match(...)` directly, which
threw `TypeError: base.match is not a function` whenever callers passed
a Location object (e.g. `new URL(href, window.location)`), a URL
instance, or any other non-string with a sensible toString. Per the
WHATWG URL Standard both `url` and `base` are stringified before
parsing.
Coerce explicitly so the existing regex paths keep working and the
constructor matches real-browser behaviour. AngularJS, jQuery, and
anything that resolves a relative URL against `window.location` now
loads under Obscura.
* fix(browser,cdp): run preload scripts before page scripts
`Page.addScriptToEvaluateOnNewDocument` was executed AFTER the page's
own scripts (and even after the load event), which silently broke
every CDP contract that depends on early injection — most visibly
puppeteer's `page.exposeFunction`, whose user-facing wrapper has to
override `globalThis[name]` before any `<script>` tag in the document
runs.
Move the registered preload sources onto `Page` and execute them
inside `navigate_single`, right after `__documentReadyState__ =
'loading'` and before the parser-discovered scripts. The CDP layer
hands the current preload set to the page via `set_preload_scripts`
before kicking off navigation, replacing the two post-navigate
`execute_preload_script` loops in `domains/page.rs` and `server.rs`.
* feat(cdp,js): emit Runtime.bindingCalled for addBinding shims
`Runtime.addBinding` was a no-op stub that installed an empty
`globalThis[name] = function () { return null; }` placeholder. Page JS
calls were silently swallowed and the Node-side callback registered
via puppeteer's `page.exposeFunction` never fired.
Wire the binding all the way through:
1. New `op_binding_called(name, payload)` op forwards every page-side
call into a `pending_binding_calls` queue on `ObscuraState`.
2. `Page::take_pending_binding_calls` drains the queue.
3. `dispatch()` empties every page's queue after each CDP request and
pushes one `Runtime.bindingCalled` event per entry into
`ctx.pending_events`, so the writer task forwards them to the
client like any other CDP event.
4. The `addBinding` handler installs a real shim that calls the op,
registers it as a preload script (keyed `__obscura_binding__<name>`)
so it survives navigations, and evaluates it now so the binding is
usable on the current document.
5. Implement `removeBinding` to mirror `addBinding`.
The shim coerces non-string arguments via `JSON.stringify`, matching
the wire contract that puppeteer's `exposeFunction` wrapper depends
on (`Runtime.bindingCalled.params.payload` is always a string).
* fix(cdp): emit canAccessOpener in all TargetInfo payloads, add aarch64 cross-compile CI
Merges PR #165 and PR #172.
PR #165: The CDP spec marks canAccessOpener as a required boolean on
TargetInfo but Obscura was omitting it. Strict Rust clients like
chromiumoxide reject the event entirely and panic with "Created target
not present" when the field is absent. Added canAccessOpener: false
to every TargetInfo emitted by setDiscoverTargets, getTargets,
createTarget, attachToBrowserTarget, attachToTarget, and getTargetInfo.
PR #172: Added a cross-compile matrix entry for aarch64-unknown-linux-gnu
running on ubuntu-22.04 x86_64, producing obscura-aarch64-linux-cross.
This is a fallback for when GitHub ARM runners are unavailable without
removing the native ARM build.
* fix(cdp): keep HTTP control plane reachable while V8 JS evaluation blocks (#173)
fix(cdp): unblock HTTP control plane during synchronous V8 execution
Fixes #62.
Previously, the TCP accept loop and CDP processor shared a single-threaded
tokio LocalSet. Long synchronous JS evaluations starved the accept loop,
causing incoming connections to hang in the kernel backlog and effectively
killing the HTTP control plane.
This moves the TCP listener and `/json/*` endpoints to a dedicated OS
thread using `std::net::TcpListener`. HTTP requests are now handled
directly via blocking I/O on this thread, completely bypassing V8.
WebSocket connections are handed off to the LocalSet via a bounded mpsc
channel (capacity 128). If V8 stalls and the queue fills, new WS
connections are dropped (ECONNRESET) rather than buffering unboundedly
to prevent OOMs.
Includes a new integration test to verify the control plane stays
responsive during a 5s JS loop, plus inline documentation clarifying
thread lifetime semantics and a minor test TOCTOU race.
* fix(js,dom): add Storage global, CharacterData ChildNode methods, fix remove/isConnected (#176)
fix(dom): add missing DOM/JS shims for modern frameworks
Closes #163.
Adds missing DOM and JS shims required by modern frameworks (Svelte 5,
Vue, Lit) and consent-management SDKs to execute without errors:
- Define the `Storage` global constructor to resolve `ReferenceError`s
and fix `Storage.prototype` access.
- Apply standard ChildNode methods (`before`, `after`, `replaceWith`,
`remove`) to `CharacterData.prototype`. This prevents `TypeError`s when
frameworks call these on Text or Comment nodes during hydration.
- Update `Element.prototype.isConnected` to perform a genuine parent-walk
rather than returning a hardcoded `true`.
- Fix `Element.prototype.remove` to properly clean up the internal ID
index so removed elements are no longer queryable.
- Remove outdated no-op stubs on the `Element` class that were shadowing
the proper polyfill implementations.
* feat(cli): add --dump assets to inventory page sub-resources (124) (#174)
feat: add `--dump assets` to enumerate page sub-resources
Adds a `--dump assets` flag to output all page-referenced sub-resources
(scripts, stylesheets, images, iframes, media, embeds, objects) as NDJSON,
allowing users to easily replay the asset graph.
- Output format: `{"url":"https://example.test/app.js","type":"script"}`
- URLs are resolved against the page origin (matching `--dump links`).
- The `type` tag is derived from the source element. `<link>` elements map
their `rel` attribute to specific types (stylesheet, icon, manifest, preload,
prefetch, modulepreload, dns-prefetch, preconnect, alternate), falling back
to "link" if missing or unknown.
- Adds 7 tests covering URL resolution, rel-mapping, supported selectors,
and empty-attribute skipping.
- Updates README dump-format table and examples.
* fix(cdp,js): playwright actionability stubs and objectId resolve (#178)
Playwright fails locator actions on Obscura for two avoidable reasons
that are separate from the missing layout engine. Both have trivial
fixes that unblock the actionability polling loop.
- Element.checkVisibility(opts) was undefined. Playwright >= 1.25 calls
it before every input event and retries until its action timeout if
the method isn't a function. Add a stub that returns true. Without a
layout pass we can't compute it for real, but a permissive stub still
beats the undefined-function path.
- ARIA reflection properties were missing on Element (role, ariaLabel,
ariaSelected, ariaExpanded, ariaHidden, ariaChecked, ariaDisabled,
ariaRoleDescription). Playwright's getByRole and getByLabel locators
read them. Reflect through to the underlying aria-* attributes for
both read and write.
- DOM.getBoxModel and DOM.getContentQuads ignored the objectId param.
Playwright passes only objectId (from a prior DOM.resolveNode), and
the handlers silently defaulted to node 0, so dispatchMouseEvent
always clicked the wrong element. Extract a resolve_node_id helper
that honors nodeId, backendNodeId, and objectId in that order.
- Page.captureScreenshot and Page.captureSnapshot now return a clear
"not supported, use a real Chromium for the screenshot leg" error
instead of the generic "Unknown Page method" fallback. Matches the
pattern already in place for Page.printToPDF.
Partial fix for #45. The architectural blocker (no layout engine, so
multi-element click coordinates are wrong) is unchanged.
* fix(mcp): allow browser MCP clients through CORS preflight (#179)
The OPTIONS preflight hardcoded Access-Control-Allow-Headers to just
Content-Type. Any browser MCP client sending mcp-protocol-version (per
the MCP spec) or Authorization / X-API-Key (typical for hosted
deployments) failed the preflight and the browser blocked the actual
request.
Add those three to the allow-list and a 24h Max-Age so browsers don't
re-preflight every call. Adds an integration test that drives the
server with a real preflight request.
Closes #175.
* fix(js,dom): scope element queries, real timers, form/prepend/equal (#180)
- Element.querySelector / querySelectorAll scope to the receiver subtree
via a new query_selector_scoped Rust op, not always document root.
Same for DocumentFragment.
- setTimeout respects its delay argument via a new async op_sleep.
setInterval is a real repeating timer instead of an alias to setTimeout.
- document.forms / images / links return live querySelectorAll results
instead of hardcoded [].
- HTMLFormElement subclasses Element with .elements / .length, and form
nodes are wrapped as HTMLFormElement so instanceof works. submit()
stays on Element.prototype to preserve the existing submit-event
dispatch and form-data navigation path.
- Element.prepend has a real implementation.
- Node.isEqualNode does a structural compare (nodeType, nodeName,
nodeValue, attributes, recursive children) instead of nid identity.
- Fix long-standing insert_before arg-order bug: bootstrap.js was
passing (parent, new, ref) but _dom only forwards two args, so ref
was silently dropped. Pass (new, ref) directly. Same fix in
replaceChild.
Adds 3 selector unit tests and 6 runtime regression tests covering
the JS shim fixes.
Closes #105.
* fix(cdp): unblock puppeteer connect, bound Runtime.evaluate, full TargetInfo (#181)
Three independent gaps that hurt browser-based MCP clients connecting
to obscura's CDP endpoint:
- Runtime.enable returned "No page" when called before a session was
attached. puppeteer-extra's FrameManager.initialize fires it on the
browser-level WebSocket BEFORE any page target exists, so the
standard puppeteer.connect / browser.newPage flow died on connect.
Now reply with {} when no session is attached and let the next
Target.attachToTarget emit executionContextCreated.
- Runtime.evaluate had no per-call timeout. A long synchronous JS
expression held the global V8 lock indefinitely and starved every
other CDP command on the session. Honor the CDP `timeout` parameter
and default to 30s (Chrome's protocolTimeout). Returns a clear error
on expiry so client-side deadlines fire cleanly instead of hanging.
- Target.getTargetInfo for the browser target omitted canAccessOpener.
Strict CDP clients (chromiumoxide) panic when the field is missing.
Fill it in (false for the browser target itself).
Partial fix for #122. The remaining items (networkIdle hard-deadline
on SPAs, /json/list hardcoded data, docs on cheap-vs-queued methods)
need bigger or non-code changes and are deferred.
* fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes #113.
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity (#170)
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity
Closes #168.
Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.
Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(cdp): fall back to default cookie jar, add Network.getAllCookies
Addresses @SGavrl review on PR #170:
- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
cookie jar via a shared helper that prefers the session page's jar and
falls back to ctx.default_context.cookie_jar when no target has been
attached. Mirrors how Storage.setCookies already routes when there is
no session, so Puppeteer/Playwright flows that issue cookie methods
before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
existing getCookies handler so both arms return the full jar.
Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).
* fix(net): use bundled webpki roots in stealth client (#184) (#185)
set_default_paths only resolves OpenSSL's compile-time CA dirs, which
exist on Linux but are empty on Windows. The stealth client's cert store
ended up empty there and every TLS handshake failed with
CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc.
CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's
trust list), which behaves the same on every platform and removes the
dependency on a system CA store.
* fix(js): Element.attributes, on* event defaults, per-nid getBoundingClientRect (#45) (#186)
Three DOM-completeness gaps that broke jQuery 1.11.3 init and Playwright
actionability:
1. Element.attributes was undefined, so legacy IE feature-detect code
like div.attributes[eventName].expando threw TypeError. Added a
NamedNodeMap-like backed by a new attribute_names DOM op (length,
numeric index, string name lookup, getNamedItem/setNamedItem,
iterable).
2. Standard on* event handler properties were missing from window, so
('on' + ev) in window returned false and forced jQuery into the
legacy IE branch. Initialised the 79 HTML-spec on* handlers to null.
3. getBoundingClientRect returned the same {x:8,y:8,width:100,height:20}
literal for every element, so Playwright's hit-testing couldn't tell
elements apart. Replaced with a deterministic synthesis from _nid
mapped into a 12-column grid sized to a 1280x720 viewport. Still not
pixel-accurate (no layout engine), but stable across reads and
distinct per node.
* feat: cookie jar persistence + --storage-dir flag + fix HTTP cookie header filtering (#114)
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
* fix(cdp): page.goto returns Response over Puppeteer and Playwright (#189) (#190)
Two correlation invariants real Chrome holds were missing, so the
nav Response was never attached to goto().
requestId === loaderId for the main Document event. Both clients
gate isNavigationRequest() on this; we were emitting a synthetic
per-page id.
Network.requestWillBeSent for the Document fires before
Page.frameNavigated. Playwright's FrameManager needs the request
present as pendingDocument before the frame commits, otherwise
currentDocument lands with request=undefined.
Subresource events and the Document's responseReceived /
loadingFinished stay where they were.
Verified end-to-end on both clients against example.com (200),
google.com (200), httpbin/404 (404).
* fix(browser,cdp): run preloads on about:blank, tighten binding shim arity
Two follow-ups from PR #166 review:
- navigate_single's about: branch returned before any preload sources
ran, so addScriptToEvaluateOnNewDocument scripts and the
Runtime.addBinding shim never installed on about:blank. Puppeteer
hits this every session: browser.newPage() lands on about:blank and
exposeFunction is dead until the first real navigation. Now the
about: branch runs the preload set after init_js(), guarded by the
same 5s timeout as regular preloads.
- The Runtime.addBinding shim used JSON.stringify on non-string args
and silently accepted any arity. Chromium's V8InspectorImpl rejects
calls without exactly one argument and ToString-coerces that arg
before emitting Runtime.bindingCalled. Match that: guard on
arguments.length and use String(arg) for the non-string branch.
Single-string calls (the path puppeteer's exposeFunction wrapper
uses) round-trip unchanged.
* fix(cdp): allocate fresh execution context ids per isolated world (#192) (#200)
Page.createIsolatedWorld and do_navigate both hardcoded context id 100
for every isolated world emission. After the second navigation Playwright
expected a fresh incrementing id (101, 102) while obscura kept re-using
100. Runtime.evaluate then failed with 'Cannot find context with
specified id: 101' because the id was never registered in
valid_context_ids and the client's bookkeeping had diverged from the
server's.
Added next_isolated_context_id counter on CdpContext (starts at 100) and
a next_isolated_context() helper that claims an id, registers it in
valid_context_ids, and increments. Both call sites now use it.
* feat(cli): default V8 heap to 4 GB on 64-bit so heavy SPAs don't OOM (#199) (#202)
V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with
heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/
web-scraping) blow past it and the worker dies with
'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session
with it.
Always set --max-old-space-size=4096 (1024 on 32-bit targets) before
V8 startup, matching headless Chrome's default. User-supplied
--v8-flags are appended after the default so they win (V8 parses
left-to-right and the last value wins), which keeps the existing
escape hatch intact for memory-constrained environments.
---------
Co-authored-by: SGavrl <[email protected]>
Co-authored-by: Ben Younes <[email protected]>
Co-authored-by: zhangyang-crazy-one <[email protected]>
|
||
|
SGavrl
|
d15cac6149 |
fix(js): Element.attributes, on* event defaults, per-nid getBoundingClientRect (#45) (#186)
Three DOM-completeness gaps that broke jQuery 1.11.3 init and Playwright
actionability:
1. Element.attributes was undefined, so legacy IE feature-detect code
like div.attributes[eventName].expando threw TypeError. Added a
NamedNodeMap-like backed by a new attribute_names DOM op (length,
numeric index, string name lookup, getNamedItem/setNamedItem,
iterable).
2. Standard on* event handler properties were missing from window, so
('on' + ev) in window returned false and forced jQuery into the
legacy IE branch. Initialised the 79 HTML-spec on* handlers to null.
3. getBoundingClientRect returned the same {x:8,y:8,width:100,height:20}
literal for every element, so Playwright's hit-testing couldn't tell
elements apart. Replaced with a deterministic synthesis from _nid
mapped into a 12-column grid sized to a 1280x720 viewport. Still not
pixel-accurate (no layout engine), but stable across reads and
distinct per node.
|
||
|
SGavrl
|
71b9b4fbab |
fix(js,dom): scope element queries, real timers, form/prepend/equal (#180)
- Element.querySelector / querySelectorAll scope to the receiver subtree via a new query_selector_scoped Rust op, not always document root. Same for DocumentFragment. - setTimeout respects its delay argument via a new async op_sleep. setInterval is a real repeating timer instead of an alias to setTimeout. - document.forms / images / links return live querySelectorAll results instead of hardcoded []. - HTMLFormElement subclasses Element with .elements / .length, and form nodes are wrapped as HTMLFormElement so instanceof works. submit() stays on Element.prototype to preserve the existing submit-event dispatch and form-data navigation path. - Element.prepend has a real implementation. - Node.isEqualNode does a structural compare (nodeType, nodeName, nodeValue, attributes, recursive children) instead of nid identity. - Fix long-standing insert_before arg-order bug: bootstrap.js was passing (parent, new, ref) but _dom only forwards two args, so ref was silently dropped. Pass (new, ref) directly. Same fix in replaceChild. Adds 3 selector unit tests and 6 runtime regression tests covering the JS shim fixes. Closes #105. |
||
|
Ben Younes
|
183c444a56 |
fix(js,dom): add Storage global, CharacterData ChildNode methods, fix remove/isConnected (#176)
fix(dom): add missing DOM/JS shims for modern frameworks Closes #163. Adds missing DOM and JS shims required by modern frameworks (Svelte 5, Vue, Lit) and consent-management SDKs to execute without errors: - Define the `Storage` global constructor to resolve `ReferenceError`s and fix `Storage.prototype` access. - Apply standard ChildNode methods (`before`, `after`, `replaceWith`, `remove`) to `CharacterData.prototype`. This prevents `TypeError`s when frameworks call these on Text or Comment nodes during hydration. - Update `Element.prototype.isConnected` to perform a genuine parent-walk rather than returning a hardcoded `true`. - Fix `Element.prototype.remove` to properly clean up the internal ID index so removed elements are no longer queryable. - Remove outdated no-op stubs on the `Element` class that were shadowing the proper polyfill implementations. |
||
|
SGavrl
|
e371e942fc |
fix(js): re-validate redirect targets in op_fetch_url
GHSA-8v6v-g4rh-jmcm: op_fetch_url validated the initial URL through validate_fetch_url, but reqwest auto-followed up to 10 redirects with no further checks. An attacker-allowed origin could 302 to http://127.0.0.1 (or any private/internal address) and read the body back to page JS. Real-world impact: AWS instance metadata, kube debug consoles, internal admin endpoints with permissive CORS all become readable from a page hosted anywhere on the open web. Disable reqwest's auto-redirect and follow manually. Apply validate_fetch_url to every redirect target before requesting it. Cap the chain at 10 hops to match prior behaviour. Browser-standard method/body rules apply on cross-method redirects (301/302/303 downgrade to GET with no body, 307/308 preserve). Verified vs unpatched build (pr69): - Patched: attacker-page -> 302 -> http://127.0.0.1:N is blocked with "Redirect to forbidden URL blocked: Access to private IP..." before the internal hop runs. Legitimate 3-hop redirect chain between allowed origins still completes. - Unpatched: internal-service body leaks through to page JS via fetch().then(r => r.text()). |
||
|
Alexandr KondakovandSGavrl
|
6dd1e4e65f |
Fix CDP click submit navigation parity (#69)
* Fix Runtime.evaluate awaitPromise settling * Fix DOM event navigation side effects * Fix CDP click-triggered navigation parity * Strengthen CDP inline script click parity test * Handle positions-like submit URLs in CDP parity Co-authored-by: SGavrl <[email protected]> |
||
|
Ben Younes
|
f3c783f6f5 |
fix(net,js): route op_fetch_url + ES module loader through proxy_url (#139) (#151)
Fixes the remaining two paths from #139: JS-level fetch()/XHR (op_fetch_url) and dynamic ES module imports (ObscuraModuleLoader) both built their reqwest clients with `Client::builder().build()` — no proxy — so every JS-initiated network call silently bypassed BrowserContext.proxy_url. (The third path, StealthHttpClient, was already fixed in main: page.rs:61 uses with_proxy.) Changes: * obscura-net: expose ObscuraHttpClient::proxy_url() so callers outside the crate can route their own reqwest requests through the same upstream proxy. The field was already there — just lacked an accessor. * obscura-js/module_loader: add `proxy_url: Option<String>` and `with_proxy(base_url, proxy_url)`. ObscuraModuleLoader::new keeps the pre-#139 direct-connection behaviour. load() builds a reqwest client per import, threading the proxy and bubbling up "Invalid module proxy" on bad URLs. * obscura-js/ops.rs: replace the process-wide `static SHARED_HTTP_CLIENT: OnceLock<reqwest::Client>` (which could never be reconfigured after first call) with a per-request `build_request_client(proxy_url)` that pulls the configured proxy out of SharedState.http_client.proxy_url() each call. Per-request build cost is ~1ms vs the network round-trip, and we no longer have a global singleton that lies about its config. * obscura-js/runtime: add ObscuraJsRuntime::with_base_url_and_proxy. with_base_url delegates to it with None so existing tests / callers are unaffected. * obscura-browser/page.rs: thread context.proxy_url through to with_base_url_and_proxy when constructing the JS runtime. |
||
|
Adam Ma
|
a8b1ec0968 |
chore: silence unused import/variable/dead-code warnings (#101) (#126)
- obscura-net/cookies.rs: #[allow(dead_code)] on CookieEntry.same_site (parsed today, kept for future SameSite enforcement). - obscura-js/runtime.rs: drop vestigial BOOTSTRAP_JS const; bootstrap.js is already embedded into the V8 snapshot via build.rs. - obscura-js/ops.rs: prefix the four unused fields in InterceptResolution::Continue with _. - obscura-cdp/dispatch.rs: drop unused InterceptResolution import. - obscura-cdp/domains/fetch.rs: drop unused Arc / Mutex / CdpEvent imports. - obscura-cdp/domains/input.rs: prefix _button / _click_count; CDP parses them but the JS click stub is hardcoded for now. - obscura-cdp/server.rs: prefix _processor_handle and _ctx; drop dead initial Ok(())/None for navigate_result/page_back - they are unconditionally overwritten in the only loop break arm. cargo build --workspace: 0 warnings. cargo test --workspace: 113 passed. |
||
|
KuaaMUandKuaaMU
|
86a868bfd6 |
fix: implement CharacterData DOM API for jQuery DataTables compatibility (#73)
jQuery DataTables references the standard CharacterData interface which was undefined in obscura's DOM environment, causing ReferenceError and empty page renders. - Add CharacterData class extending Node with data/length/substringData/ appendData/insertData/deleteData/replaceData - Make Text and Comment extend CharacterData instead of Node - Update _wrap() to create proper Text/Comment instances by nodeType - Add create_comment_node op for Document.createComment() - Fix set_text_content to handle Comment node mutations Fixes #72 Co-authored-by: KuaaMU <[email protected]> |
||
|
rifuki
|
53a04fa905 |
fix(wasm): preserve binary fetch responses (#48)
* fix wasm response handling * fix URL inputs in fetch polyfill * test(wasm): cover binary fetch responses |
||
|
ben1009
|
21136ca819 |
feat: support file:// URL scheme for local HTML testing
Allow loading local HTML/CSS/JS files via the file:// protocol, enabling Obscura to be used for testing static sites without running a local HTTP server. Changes: - Allow "file" scheme in validate_url() and validate_fetch_url() - Add fetch_file_url() to read local files and return synthetic HTTP responses with appropriate content-type headers - Skip private IP / localhost checks for file:// URLs Closes #16 |
||
|
h4ckf0r0day
|
8749725b45 | Initial Release |