07c6e735fa40a6f0718a67371b7a8fa7204dc089
45
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 SGavrl
|
07c6e735fa |
WPT conformance: WHATWG URL, DOM Range, and post-navigation event-loop settle (#233)
* fix: restore regressions introduced by #227 #227 deleted build.rs (reverting #224's version-from-tag feature), removed DOM cycle guards in append_child/insert_before that prevent infinite loops on self-append/self-insert, dropped the CharacterData textContent fast-path, removed PI/Doctype Rust ops, and cut the env_allows_private_network() check from validate_fetch_url. Restored: - crates/obscura-cli/build.rs (version from tag/env) - OBSCURA_BUILD_VERSION in version string and banner - allow_private_network env var propagation in main.rs - append_child/insert_before self-cycle guards in tree.rs - textContent CharacterData fast-path in tree.rs - create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops in ops.rs - ProcessingInstruction branch in set_text_content op - env_allows_private_network() check in validate_fetch_url * fix(js): Europe/Berlin timezone, plugins iterator recursion Two bugs in the stealth bootstrap code from #227: - 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat silently fell back to UTC. Changed to 'Europe/Berlin'. - plugins[Symbol.iterator] called this[Symbol.iterator]() recursively, blowing the stack on Array.from(navigator.plugins). Fixed to bind Array.prototype[Symbol.iterator] directly to the array. * url: back URL/URLSearchParams with the url crate, add <a>/<area> decomposition The URL global was a regex shim: it mis-parsed userinfo (hostname came back as the username, port as "pass@host"), had no username/password, and its setters were no-ops. Replace it with two ops (op_url_parse, op_url_set) backed by the WHATWG-compliant url crate, and add the HTMLHyperlinkElementUtils members (protocol/host/hostname/port/pathname/search/hash/origin/username/password and their setters) to <a> and <area>, which had none. This was the single largest WPT failure. a.search returned undefined, so the encoding encode-href tests threw "Cannot read properties of undefined (reading 'substr')" about 225k times. With the fix url-constructor passes 831/888, url-setters 192/279, urlsearchparams-sort 17/17, url/a-element 415/887, and the encode-href files run their assertions instead of crashing. op_url_set runs the url crate inside catch_unwind: a few setters panic on the pathological inputs the url-setters tests use, and a panicking setter must be a no-op (the WHATWG "do nothing on invalid value" rule) rather than abort. URLSearchParams also gained getAll/sort/size/keys/values/entries/iterator and a back-reference so mutations write through to the bound URL. * dom: real Range/StaticRange, createDocumentType, XML Document type Range was a no-op stub (setStart/setEnd discarded their arguments, so startContainer/endContainer read back undefined and every dom/ranges helper that walked range.endContainer crashed with "reading 'nodeType' of undefined"). Replace it with a property-storing Range over the live tree: setStart/End and the *Before/*After/selectNode/selectNodeContents/collapse positioning, plus the boundary-point algorithms comparePoint, compareBoundaryPoints, isPointInRange, intersectsNode, commonAncestorContainer, cloneRange and a real stringifier, all built on a correct preorder tree-order comparison. Add StaticRange. new Document() now reports an XML contentType (it has no backing node id), so createCDATASection no longer throws NotSupportedError on it. Add DOMImplementation.createDocumentType with the lenient name validation browsers use and an owner-document backref so DocumentType.ownerDocument is correct. StaticRange-constructor 14/17. This does not fully unblock dom/ranges yet: dom/common.js also needs createHTMLDocument()/createDocument() to return documents with a working body/createComment/etc., which is a separate fix. * fix(browser,cli): settle the event loop after navigation obscura fetch read the page at the load event and never pumped the event loop afterward on the eval/dump path, so --wait was effectively ignored there. Pages that schedule timers, do async work, or only report through a completion callback that runs after load (testharness add_completion_callback, SPAs) were observed frozen at the load event and never produced their result. Add Page::settle, which drives run_event_loop until the loop goes idle or --wait elapses (returns early when idle, so static pages stay fast), and run it after navigation in the fetch path. |
||
|
SGavrl
|
6b3cc1a686 |
Fix/rescue 227 regressions (#232)
* fix: restore regressions introduced by #227 #227 deleted build.rs (reverting #224's version-from-tag feature), removed DOM cycle guards in append_child/insert_before that prevent infinite loops on self-append/self-insert, dropped the CharacterData textContent fast-path, removed PI/Doctype Rust ops, and cut the env_allows_private_network() check from validate_fetch_url. Restored: - crates/obscura-cli/build.rs (version from tag/env) - OBSCURA_BUILD_VERSION in version string and banner - allow_private_network env var propagation in main.rs - append_child/insert_before self-cycle guards in tree.rs - textContent CharacterData fast-path in tree.rs - create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops in ops.rs - ProcessingInstruction branch in set_text_content op - env_allows_private_network() check in validate_fetch_url * fix(js): Europe/Berlin timezone, plugins iterator recursion Two bugs in the stealth bootstrap code from #227: - 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat silently fell back to UTC. Changed to 'Europe/Berlin'. - plugins[Symbol.iterator] called this[Symbol.iterator]() recursively, blowing the stack on Array.from(navigator.plugins). Fixed to bind Array.prototype[Symbol.iterator] directly to the array. |
||
|
SGavrl
|
990f2a63c8 |
feat(js): popover, legacy events, CDATA/PI, DOMTokenList, namedItem, attachShadow (#231)
Next batch of WPT dom/shadow-dom conformance fixes in bootstrap.js: - Popover API: popover reflection, showPopover/hidePopover/togglePopover, ToggleEvent, :popover-open - Legacy event init: initEvent/initUIEvent/initMouseEvent/initKeyboardEvent/initCompositionEvent, CompositionEvent - Document.createCDATASection / createProcessingInstruction (XML-gated) - classList as a real DOMTokenList (+ relList/sandbox/sizes/htmlFor), spec-correct namespaceURI - getElementsByClassName ASCII-whitespace tokenization - HTMLCollection.namedItem and named access; createRange on document/iframe/DOMParser - attachShadow mode validation and a real ShadowRoot (cloneNode throws, activeElement, styleSheets) querySelectorAll returns a NodeList; named access on live HTMLCollections is resolved lazily via a Proxy so query hot paths stay at native speed. |
||
|
SGavrl
|
dc2a2e3b52 |
feat(js): DOM, namespace, validity, and parsing APIs surfaced by WPT conformance work (#230)
* feat(js): DOMException, ParentNode methods, composedPath, HTMLCollection/NodeList Add the DOMException global (name, message, code, and the legacy code constants on both the constructor and instances). It was referenced by AbortSignal and others but never defined, so any test touching it failed with 'DOMException is not defined'. ParentNode append/prepend/replaceChildren now live on Element, Document, and DocumentFragment and use the spec node conversion: a non-Node argument (including null and undefined) becomes a Text node instead of being dropped or throwing. replaceChildren was missing entirely; append and prepend were Element-only and skipped null/undefined. Add Event.composedPath() and the HTMLCollection and NodeList globals. Verified with fetch --eval; integration suite still 28/28. * feat(js): setHTMLUnsafe/getHTML, namespace lookups, getElementsByTagNameNS, Attr, form validity, Response.blob Element and ShadowRoot setHTMLUnsafe / getHTML as shims over innerHTML. Node lookupNamespaceURI / lookupPrefix / isDefaultNamespace walking the element ancestry and xmlns declarations, with the built-in xml and xmlns namespaces. Element and Document getElementsByTagNameNS with namespace and localName wildcard support. Attr nodes plus Document createAttribute / createAttributeNS and Element getAttributeNode(NS) / setAttributeNode(NS) / removeAttributeNode, bridged to get and set attribute. Form control constraint validation (ValidityState, validity, willValidate, checkValidity, reportValidity, setCustomValidity, validationMessage) and text selection (selectionStart / End / Direction, setSelectionRange, setRangeText, select) as Element.prototype shims. Patch blob / text / json onto the engine Response prototype so fetch(...).blob() works; the real Response prototype lacked blob. Verified with fetch --eval; integration suite still 28/28. |
||
|
SGavrl
|
c8d98845ad |
fix(build): switch reqwest from native-tls-vendored to rustls-tls (#229)
Building with --features stealth linked both wreq's BoringSSL and reqwest's vendored OpenSSL into the same binary, causing undefined symbol errors at link time on Linux. Switching reqwest to rustls-tls removes the vendored OpenSSL dependency. rustls is pure Rust and coexists cleanly with wreq's BoringSSL. Verified: stealth and non-stealth release builds succeed, 28/28 integration tests pass. |
||
|
SGavrl
|
1b9be08767 |
fix(cdp): exit on Ctrl-C while serving (#225) (#226)
The LocalSet accept loop blocked on ws_rx.recv() forever because the detached accept thread kept a ws_tx clone alive, so the channel never closed and start_with_full_serve_options never returned. cdp_processor caught Ctrl-C and saved cookies, but that only ended its own task. Add a shared shutdown flag and Notify. On Ctrl-C cdp_processor sets the flag and notifies; the accept loop selects on the notify and breaks, so run_until returns and the process exits. The accept thread checks the flag between connections so it stops and drops ws_tx on the slow path. Closes #225 |
||
|
SGavrl
|
5e23903063 |
fix: DocumentFragment flattening in appendChild / insertBefore / replaceChild (closes #217) (#221)
Heavy-JS sites build subtrees off-DOM in a DocumentFragment then
appendChild the fragment to a real parent. The DOM spec says: move
the fragment's children up and leave the fragment empty. Our shim
was passing the fragment's _nid straight to the Rust op, so the
fragment node itself got grafted instead of its children, and
framework code that read the inserted nodes' properties hit
"Cannot read properties of undefined (reading 'length')" the moment
it expected a real Element but found our fragment wrapper.
That is what was killing Google Shopping and Bing Shopping for
@Tarun-004 — Bing's anti-malware shim, Google's result-row renderer,
and a sticky-container helper (n.Setup / n.InitializeStickyContainer
in the stack) all use the off-DOM-fragment pattern.
Same defect existed in insertBefore and replaceChild — all three
now flatten DocumentFragment per spec.
While there, three smaller defenses that the same kind of page
exposes:
- _nid validation. Passing a non-Node truthy value (e.g. undefined,
a plain object) used to silently call op_dom("append_child",
parent, undefined), which the Rust side parses with unwrap_or(0)
and grafts as a child of the document root.
- Self-append guard mirrors the insert_before self-cycle fix:
appendChild(self) is a no-op rather than corrupting the
sibling-pointer linked list.
- Tree::append_child gains a parent_id == child_id early return as
defense in depth, parallel to the insert_before guard.
Sanity test in eval mode: DocumentFragment with two children appended
to a host now produces "<span>A</span><span>B</span>" and the
fragment is empty; insertBefore with a fragment containing C produces
"<span>C</span><span>A</span><span>B</span>"; self-append and
appendChild(null) no longer throw.
Verified against the actual reproducer: `obscura fetch --dump text
https://www.bing.com/shop?q=test` no longer produces the
TypeError / "Inline script error" / "Timer error" pattern from
the issue.
|
||
|
SGavrl
|
ee10e8ddb5 |
feat(engine): DOM + AbortSignal + storage fixes surfaced by WPT conformance work (#219)
Engine-only changes pulled out of the WPT conformance branch. The tests, runner, and harness vendoring that drove these have moved to a separate benchmark repo (obscura-bench). Real bug fixes: - Tree::insert_before created a self-cycle when the inserted node was its own reference (the spec-conformant `child.after(x)` path that the testharness's ChildNode helpers exercise). The fixup wrote node.prev_sibling = node.next_sibling = own id; every subsequent childNodes / querySelectorAll walk then looped forever and obscura burned RAM until OOM. Per spec the op is a no-op; we now early-return. - Tree::text_content extended to return own data for direct CharacterData access (Text / Comment / ProcessingInstruction). - Element.namespaceURI now reflects the namespace passed to createElementNS instead of collapsing every non-svg namespace to xhtml. Without this, code that creates elements in MathML, custom XML namespaces, or the empty namespace gets misclassified as HTML. - AbortController / AbortSignal: the previous shim was a plain object whose addEventListener was a no-op, so listeners for 'abort' silently never fired and any AbortSignal.timeout / AbortSignal.any chains broke. New impl tracks aborted/reason, has a real addEventListener / removeEventListener / dispatchEvent surface, honors `once`, and implements the static abort / timeout / any factories crawlers use for budgeted fetches. - StorageEvent class added. localStorage / sessionStorage mutations now dispatch 'storage' events on the parent window and every iframe window so cross-frame onstorage listeners fire. - ProcessingInstruction (nodeType 7) and DocumentType (nodeType 10) constructors plus document.createProcessingInstruction and document.implementation.createDocumentType. textContent returns null for Document and DocumentType per spec. - insertAdjacentText / insertAdjacentElement on Element. Without these the testharness output renderer threw in a forEach without try/catch, breaking the completion callback chain for every test with subtests. - Window + Node addEventListener filter non-callable listeners, accept the EventListenerObject shape (object with handleEvent), dedupe by (type, callback, capture), honor the `once` option, and snapshot the listener list before dispatch so a `once: true` removal mid-iteration doesn't corrupt the loop. - Iframe HTML loader now extracts head <script> blocks and body on-handler attributes (`<body onstorage=...>`), evals scripts in the parent realm after the iframe window is built, re-attaches handlers as window listeners. Same iframe element reusing its src no longer double-registers in the iframe registry. - ops.rs validate_fetch_url honors the OBSCURA_ALLOW_PRIVATE_NETWORK env var, and the CLI --allow-private-network flag mirrors itself into that env so the js-side fetch gate matches the http_client gate. Without this, iframe loads of 127.0.0.1 resources were blocked even with the flag set. - ops.rs: create_processing_instruction, create_doctype, pi_target, doctype_name, doctype_public_id ops added; set_text_content extended to handle PI nodes. |
||
|
SGavrl
|
fd2421940b |
Perf/cdp fetch cache and cookie alloc (#216)
* perf: cache HTTP clients + selector parsing + exponential wait backoff Three low-risk wins discovered while reading the hot paths. 1. op_fetch_url cached its reqwest::Client. Previously every JS-side fetch(), XHR, and dynamic script load built a fresh client, which re-loaded TLS roots and started with an empty connection pool. On an asset-heavy page with 30+ subresources this added ~150ms of wasted setup with zero connection reuse. The cache is a process- wide map keyed by proxy URL (empty string for no-proxy), so the first fetch on each proxy pays the build cost once and every subsequent fetch reuses the same pool. Also bumped pool_idle_timeout to 5 minutes and tcp_keepalive to 60s so connections stay warm for SPA-heavy workloads that hit the same origin dozens of times across a navigation. 2. DomTree::parse_selector now caches parsed selectors in a thread- local HashMap (cap 256). querySelector / querySelectorAll re- parsed the selector string every call before; batch DOM access patterns (an agent scraping a table row-by-row, a framework polling for elements) paid the cssparser cost N times. 3. browser_wait_for / browser_wait_for_text now exponentially back off (5 -> 10 -> 20 -> ... -> 200 ms). The old fixed 200ms tick meant a selector that appeared 30ms after the call still took the next full 200ms tick to be detected. Worst case is still 200ms per check, but the median wait drops sharply for fast appearances. 28/28 integration tests pass. Asset-heavy bench (rust-lang/rust, docs.stripe.com, etc) doesn't regress. * perf: module loader client cache, fetch concurrency cap, snapshot hide list, cookie alloc Round two of perf wins after a deep code audit. 1. ES module loader (module_loader.rs) was building a fresh reqwest::Client on every dynamic import. Modern SPAs (Next, Vite) load 20-50 chunks via dynamic import; each one used to spin up an empty connection pool with fresh TLS init. Now reuses the process-wide FETCH_CLIENT_CACHE that op_fetch_url already uses (the cached_request_client helper). Saves ~30-100ms on chunk- heavy first navigations. 2. Script and CSS fetches in execute_scripts used futures::join_all with unbounded concurrency. A 100-script page would open 100 TCP sockets at once, exhausting the connection pool and the OS ephemeral port range. Capped at 16 in-flight (well above the per- host pool ceiling real Chrome uses). 3. bootstrap.js __obscura_init() ran Object.keys(globalThis).filter() on every page setup to find internals to hide. The set of keys is static at snapshot-build time, so we now precompute it once (top-level statement just below the function) and __obscura_init just iterates the precomputed array. Was 5-40ms per page on heavy SPAs (1000+ globals); now ~0.2ms per page. 4. CookieJar::domain_matches allocated 2 lowercase Strings and a format! per call. The lookup runs per fetch per cookie domain; on a heavily-cookied site with 100 cookies across 30 domains that was thousands of allocations per page. Now zero-alloc with eq_ignore_ascii_case and char-boundary checks. |
||
|
SGavrl
|
171dba23e7 |
Feat/mcp agent skill tier1 (#215)
* feat(mcp): 10 new agent tools + element refs (12 -> 22 tools)
Brings obscura-mcp from a Playwright-style core (navigate/click/fill/
type/snapshot/wait_for/evaluate/select/press_key/network_requests/
console_messages/close) up to a full agent surface.
New tools:
- browser_markdown: render the page as Markdown (reuses the
HTML_TO_MARKDOWN_JS converter behind `fetch --dump markdown`).
Token-dense alternative to browser_snapshot.
- browser_links: every <a href> with anchor text, JSON-per-line.
Optional internal_only and limit args.
- browser_interactive_elements: list every clickable / typeable
element with a stable ref (e1, e2, ...) and a one-line label.
- browser_back / browser_forward / browser_reload: history nav,
uses the Page history stack from the engine-tier1 commits.
- browser_get_cookies / browser_set_cookie / browser_clear_cookies:
full cookie jar ops so an agent can skip a login flow when it
already has a session token.
- browser_wait_for_text: wait for a substring to appear anywhere
in the visible page text. Complements browser_wait_for which
only checks for selectors.
Element refs (the playwright-mcp pattern, agent UX win):
- BrowserState now keeps an interactive_refs HashMap built on each
browser_snapshot / browser_interactive_elements call.
- Each interactive element gets a data-obscura-ref="eN" attribute
written into the DOM so refs survive across calls.
- browser_click / browser_fill / browser_type accept `ref` (preferred)
or `selector` (fallback). Removes the agent's 'guess a CSS
selector then retry' loop.
- Refs are cleared on every navigation / reload / back / forward / close.
Output size limits:
- browser_snapshot, browser_markdown accept max_chars (default 4000).
- browser_links, browser_interactive_elements accept limit (default 100).
- Truncated responses say '...(truncated, N more chars)' so the agent
can ask for more if needed.
Verified end-to-end via /tmp/mcp-test.sh: tools/list returns 22 tools,
ref-based click on a real element works, markdown converts H1+P+UL
correctly, cookies round-trip. 28/28 integration tests pass.
* feat(mcp): 10 more agent tools, multi-tab, structured extract (22 -> 32 tools)
Tier 2: closes the gaps in agentic flows the Tier 1 surface still
needed multi-call workarounds for (login forms, paginated lists,
multi-step research across multiple destinations).
Multi-tab:
- BrowserState now holds a BTreeMap of tabs (Page per tab) with an
active_tab cursor. page_mut() auto-creates a default tab so every
legacy single-tab tool keeps working.
- browser_tab_new: open and optionally navigate.
- browser_tab_list: list all open tabs with active marker.
- browser_tab_switch: change which tab subsequent tools operate on.
- browser_tab_close: close (default: active tab), promotes another.
Form ergonomics:
- browser_detect_forms: returns each <form> with action, method, and
every field's tag, type, name, current value, label (from <label for>
or aria-label or placeholder), and option list for selects. Agents
call this once to see the whole form shape instead of guessing
selectors field by field.
- browser_fill_form: fill N inputs in one call. Each entry chooses
type='text' (default value-set), 'check'/'uncheck', or 'select'
(matches by value OR visible text). Optional submit_ref clicks
after filling.
Structured extraction:
- browser_extract: pass {field: 'selector'} and get a JSON object back.
Selector suffix '@attr' reads attribute (e.g. 'a@href'). Field
suffix '[]' returns an array (queries all matches). One call returns
the whole structured object instead of an evaluate per field.
- browser_count: existence / pagination probe.
- browser_get_attribute: read href / src / value / data-* etc.
Page control:
- browser_scroll: 'top'|'bottom'|'up'|'down'|'left'|'right' with
optional amount, or scroll an element into view. Dispatches a real
'scroll' event on window AND document so infinite-scroll handlers
fire even though our viewport isn't a real layout.
Other:
- Bumped #![recursion_limit = "512"] because the tools/list JSON
literal is now too big for the default macro recursion limit.
- browser_count: coerce f64 -> u64 since V8 numbers come back as floats.
Verified end-to-end: 32 tools registered, multi-tab open/list/switch/
close all work, detect_forms returns proper JSON, fill_form fills
text/check/select in one call, extract returns the structured object
with arrays and attribute reads. 28/28 integration tests pass.
* feat(mcp): search + storage state export/restore (32 -> 35 tools)
Three additions that close the long-tail agentic flows:
- browser_search: substring search in visible page text, returns each
match with N chars of surrounding context (default 80). Lets an
agent confirm content exists before scraping and locate a section
without pulling the full page into its window. case_sensitive +
limit + context_chars args.
- browser_storage_state: export the full auth surface (cookies +
localStorage + sessionStorage for the current origin) as one JSON
object. Agents stash this between runs to skip a login flow.
- browser_set_storage_state: apply a previously-exported state. Sets
cookies via the cookie jar; writes localStorage / sessionStorage
via evaluate on the current page (so a navigate to the right origin
must come first).
Tool count: 35. 28/28 integration tests pass.
|
||
|
SGavrl
|
89692f3cb7 |
Perf/cdp network and lockfree (#214)
* perf(cdp): TCP_NODELAY + zero-buffer WS writes shave 130ms off newPage CDP traffic during browser.newPage() and Page.navigate is many small (~100-byte) request/response frames. Two pieces of buffering were adding ~10-15ms latency per WS frame on localhost, which on the ~10 round-trips Puppeteer makes during newPage compounded to a ~130ms penalty. 1. TCP_NODELAY on every accepted WS stream. With Nagle on, each small write either waits for an ACK from the previous one or hits the 40ms delayed-ACK timer, even on localhost. 2. WebSocketConfig.write_buffer_size = 0 on accept_async. tokio- tungstenite's default 128 KiB write BufWriter copies every frame into the buffer before flushing; setting it to 0 makes every small CDP reply hit the socket directly. Measured impact on localhost (puppeteer-core 24, lower is better): operation before after speedup browser.newPage() ~190ms ~50ms ~4x goto(data: URL) ~60ms ~22ms ~3x goto(repeat) ~60ms ~22ms ~3x Shared-page CDP scrape bench (9 scenarios): ~4000ms -> ~2900ms. 28/28 integration tests pass. * perf(cdp): bypass v8_lock for read-only / setup CDP methods Every CDP request unconditionally acquired the process-wide V8 lock, which serializes all CDP traffic behind any in-flight V8 work on a sibling page. During Puppeteer's browser.newPage() the ~8 setup calls (Page.enable, Runtime.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, etc.) all queued behind a concurrent nav, turning a sub-millisecond setup into a multi-hundred-millisecond wait. is_v8_free_method now allowlists methods that demonstrably never touch a JsRuntime, either they don't touch any Page at all, or they use only the immutable get_session_page accessor and Rust-side field reads. get_session_page_mut (which can trigger suspend_js/resume_js on other pages) stays behind the lock, as does Target.createTarget (which calls page.navigate). Allowlisted: most of Target/Browser/Storage, plus Page.enable, Page.getFrameTree, Page.addScriptToEvaluateOnNewDocument, Page.createIsolatedWorld, Runtime.enable, Network.* cookie ops, Fetch.continue/fulfill/fail. Measured on the puppeteer-core 24 CDP scrape bench: browser.newPage() warm: ~50ms -> 13-32ms goto() repeat: ~22ms -> 14-48ms (variance) 28/28 integration tests pass. 18/18 realistic scrape scenarios pass. |
||
|
SGavrl
|
282b4646d3 |
perf(js): resolve_promises_until exits as soon as the awaited IIFE finishes (#213)
Every Puppeteer click ran isIntersectingViewport (an actionability check that wraps IntersectionObserver in a promise). Our IO.observe schedules re-fires at 120/500/1500/3500/7000ms to cover infinite-scroll sentinels. Puppeteer's callback disconnects the observer on the first fire, so the re-fires are no-ops, but they keep the deno_core event loop 'not idle' until they all fire. resolve_promises was waiting for the loop to go idle (5s hard cap), so every awaitPromise call paid the full 5s even though the result was ready in <1ms. Effect: page.click() took ~7s instead of ~150ms because clickableBox + isIntersectingViewport each hit the cap. Fix: each evaluate / callFunctionOn now writes a unique sentinel (globalThis.__obscura_done_<counter> = true) in a finally block. The new resolve_promises_until polls that sentinel with a 1ms / 2ms / ... backoff up to 50ms and exits as soon as the IIFE is done, instead of waiting for unrelated timers to drain. Worst-case miss is one tick (50ms). Measured impact (Puppeteer CDP scenarios, lower is better): scenario before after data: click button 7420ms ~530ms (14x) data: addEvt click 14536ms ~680ms (21x) data: form submit 7676ms ~810ms (9x) CDP bench total 35190ms ~7400ms (~5x) 28/28 integration tests still pass. |
||
|
SGavrl
|
50462d41a9 |
fix(js): load_module actually fetches the module source (#205) (#212)
load_module registered an empty string as the module body and called it loaded. Every Vite / Next / SvelteKit external module bundle 'loaded' in 1ms with zero code, so the framework never executed and the SPA shell sat empty (#root.children.length == 0). Now fetches the URL through the page's http_client and passes the real source. getvix.dev/docs (Vite + React) goes from rootChildren=0 / bodyTextLength=10 (SSR shell only) to rootChildren=4 / bodyTextLength=3708 (React app mounted). Closes #205. |
||
|
SGavrl
|
3c3798b197 |
fix(js): HTMLMetaElement.content is read/write (#210) (#211)
`<meta>.content` was a getter-only stub. `next/head` and `next/document` iterate <meta> tags and assign `.content` during hydration, which threw `Cannot set property content`, React caught and retried in a tight loop, the page never mounted, and the error spammed the console thousands of times. Traveloka and other Next.js sites doing meta management were unusable. Now reflects the content attribute with both get and set, per spec. Template's `.content` (DocumentFragment) is unchanged. Closes #210. |
||
|
SGavrl
|
326096e20f |
Engine tier1 compat fixes (#208)
* engine: document.currentScript, data: subresources, dynamic module scripts
document.currentScript:
- ScriptInfo now carries the script element's nid.
- Before each script body runs, set globalThis.__currentScriptNid; clear after.
- Document.currentScript getter wraps the current nid. Next.js / Turbopack
reads document.currentScript.src to derive the chunk base path. Without
this the loader saw null, threw 'chunk path empty', and hydration stalled.
- Also threaded through the dynamic-script appendChild path so synchronously
injected <script> elements see their own element as currentScript.
data: URIs as subresources:
- subresource_allowed permits 'data:' alongside http/https. The bytes are
inline in the URI, no network fetch, no SSRF.
- Script-fetch loop short-circuits data: URLs and decodes them locally
instead of routing through the HTTP client.
- Real Chrome allows this by default; Instagram and most Meta properties
ship their bootstrap as <script src=data:application/x-javascript;base64,...>.
Dynamic <script type=module>:
- appendChild SCRIPT handler now accepts type='module' instead of dropping it.
- For external module src, uses dynamic import() instead of fetch + eval.
- For inline modules, wraps the source in a data: URL and import()s that
so module syntax (import/export) parses correctly.
Before / after on previously broken sites (HTML bytes rendered):
vercel.com 0 -> 960 KB
instagram.com 0 -> 669 KB
nextjs.org 0 -> 290 KB
music.apple.com 0 -> 1.88 MB
tailwindcss.com 0 -> 900 KB
linear.app 0 -> 2.4 MB
github.com PR 0 -> 468 KB
28/28 integration tests pass.
* engine: customElements.define upgrades existing matches + connectedCallback
The old customElements was a registry stub: define() recorded the class,
but never instantiated it on matching elements, never called the lifecycle
callbacks, and whenDefined() resolved synchronously with whatever was
already registered (or undefined forever for late-defined components).
New behavior matches the Web Components spec well enough for lit /
MusicKit / Polymer to wire up:
- define(): on register, querySelectorAll the document for matching tags,
copy the prototype methods onto each element instance, call the class
constructor with the element as 'this' for init logic, then fire
connectedCallback if the element is currently in the document.
- whenDefined(): returns a real Promise that resolves when the class is
later defined, not before.
- upgrade(root): walks the subtree, upgrades any matching unupgraded
element. Used by frameworks that mount components asynchronously.
We don't truly subclass HTMLElement per spec (Element is shared across
the engine), so the upgrade copies prototype descriptors onto the
element instance instead of swapping its [[Prototype]]. Behavior is
identical for the common patterns (custom methods, shadow DOM attach
in constructor, connectedCallback render).
28/28 integration tests pass.
* scraping: fix puppeteer page.$, click, page.$$ + form / radio / checkbox actions
These bugs broke every realistic Puppeteer scraping flow. Repro: type into
a search box, click a button, scrape rows. Before this commit the very
first step (page.$('selector')) returned null on every site.
Runtime.evaluate (and the two related wrappers) put the user expression
on a single line as '({expr})'. Puppeteer's bundled injected.js ends with
a '//# sourceURL=pptr:internal' line comment; the comment swallowed our
closing paren and the wrapped function failed to parse with a SyntaxError
that never propagated back. Result: every page.$, page.$$, type(),
hover(), click(), waitForSelector(), getByRole(), etc. silently returned
null. Fix: newline before the trailing ')' so the comment can't eat the
wrapper. Applied to all three wrap paths (eval, store_object_with_meta,
wrap_expression).
documentElement and body now return viewport dimensions for clientWidth /
clientHeight / scrollWidth / scrollHeight / offsetWidth / offsetHeight and
for getBoundingClientRect itself. Puppeteer's #clickableBox clips each
candidate quad to document.documentElement.clientWidth/Height; with the
old 100x20 stub every synthesized element bbox was clipped to empty and
clickablePoint threw 'Node is either not clickable or not an Element'.
Also aligned innerWidth / innerHeight to 1280x720 to match
Page.getLayoutMetrics.
document.elementFromPoint now does real hit testing against the synthetic
bboxes (depth-first, deepest match wins) instead of always returning
<body>. Input.dispatchMouseEvent uses it to pick the click target, which
removed the dependency on a __obscura_click_target side-channel that
the previous getBoundingClientRect call had to set right before the
dispatch.
Element.dispatchEvent now fires inline event handlers
(onclick='...' content attribute and the IDL onclick property) in
addition to addEventListener registrations. Without this, every site
that uses inline onclick / onsubmit / onchange was a no-op.
Input.dispatchMouseEvent's click path now triggers the right default
action by element kind: submit button -> form.submit(); checkbox /
radio -> toggle checked + dispatch 'change'; anchor with href -> nav.
Previously only anchor was handled, so checking a checkbox or submitting
a form by clicking its button did nothing through CDP.
Runtime.getProperties now allocates a stable child objectId for every
object-valued property and stashes the value in __obscura_objects.
Items whose value has a numeric nodeType also get subtype:'node' +
className so Puppeteer's JSHandle.asElement() wraps them as
ElementHandle. Without this, page.$$('selector') returned an array
of plain JSHandles and checkboxes[0].click() raised
'click is not a function'.
Real-world Puppeteer scrape coverage (test_all.py style scenarios):
- HN top stories scrape: fail -> pass
- GitHub repo file rows scrape: fail -> pass
- Wikipedia search input type: fail -> pass
- httpbin form fill + check: fail -> pass
- Cookies persistence across nav: pass -> pass
28/28 integration tests pass.
* scraping: inline scripts run on DCL, JS nav emits CDP events, hit testing
Three remaining gaps from realistic Puppeteer scraping flows:
1. DCL path skipped execute_scripts entirely. Returning early on
waitUntil=domcontentloaded meant every inline <script> was silently
dropped: addEventListener calls in onload bootstraps never ran,
form submit handlers never registered, page.click() hit no-ops.
Spec: DOMContentLoaded fires AFTER scripts. Now scripts run before
the lifecycle flips to DCL regardless of waitUntil.
To keep heavy SPAs from blowing the 25s Puppeteer goto timeout
now that scripts always run, added a soft deadline on the
script-execution phase (default 10s, OBSCURA_SCRIPT_DEADLINE_MS
to override). Both fetch and per-script execution check it.
2. JS-initiated navigation (form.submit, location.assign) did not
emit CDP frameNavigated / lifecycleEvent. Puppeteer's
waitForNavigation hung forever. Runtime.evaluate and
Runtime.callFunctionOn now drain pending_navigation through a
helper that fires the same post-nav event sequence Page.navigate
does.
3. Document.elementFromPoint walked the tree only descending into
ancestors containing (x,y). Our synthetic per-nid bboxes do not
form a containment hierarchy, so a deep <input> inside <label><p>
was never reached and clicks resolved to <body>. Switched to a
flat sweep, deepest match wins (highest nid as a proxy for tree
depth).
Realistic Puppeteer scrape scenarios: 18 of 18 assertions pass across
HN, Wikipedia, GitHub, httpbin forms, cookie persistence, and inline
JS submit handlers. 28/28 integration tests pass.
* scraping: fetch in page context, textarea, select, basic history
Four more gaps from the broad API sweep:
1. fetch() and XHR from page JS hung forever. set_intercept_tx() in
the runtime auto-set state.intercept_enabled = true as a side
effect, so every navigation silently enabled Fetch interception
even when the CDP client never called Fetch.enable. op_fetch_url
then routed every request through the intercept channel waiting
for a Fetch.continueRequest that the client wasn't going to send.
Split the flag from the channel: set_intercept_tx wires the
channel without enabling; new set_intercept_enabled toggles the
flag. Fetch.enable / Fetch.disable now call both, and the nav
path only wires the channel.
2. page.type into textarea lost everything past the first newline.
The dispatchKeyEvent handler skipped text=='\n' entirely, so
'multi\nline' inserted only 'multi'. Also Enter on a textarea
tried to submit the nearest form instead of inserting a newline.
Now Enter inserts \n inside textareas and only submits forms
from input fields, matching Chrome.
3. <select>.value did not reflect the selected option. Generic value
getter returned the select's value attribute (empty) instead of
the matching option. Added select-aware getter that walks options
for the selected one and falls back to the first; setter accepts
page.select() and dispatches a 'change' event.
4. Basic navigation history. Pages now track a visited-URL stack
and history_index. Page.getNavigationHistory returns the real
history; Page.navigateToHistoryEntry honors entryId and emits the
same CDP nav-event sequence as Page.navigate so the response and
lifecycleEvent reach waiting clients. resetNavigationHistory
clears the stack. Push happens after every successful nav (skips
consecutive duplicates so Page.reload doesn't grow the stack).
Broad API sweep across 13 feature groups: 43/44 PASS.
All categories solid (navigation, selectors, typing, waitForSelector,
evaluate, cookies, keyboard, attributes, events, fetch/XHR, storage,
timers). goBack reaches its target URL but Puppeteer client-side
waitForNavigation still hangs on it; the CDP frameNavigated +
lifecycleEvent fire correctly, so the remaining gap is in event
shape rather than logic. 28/28 integration tests pass.
|
||
|
SGavrl
|
b98e457428 |
engine: large compat + reliability pass (#206)
CDP / navigation: - Route all Page.navigate through process_with_interception so the parent dispatcher stays responsive during long navs (fixes Target.createTarget timeouts after a handful of newPage calls). - Default waitUntil to domcontentloaded. Puppeteer and Playwright don't pass it server-side and wait client-side via lifecycleEvent, so the old Load default blocked nearly every nav on slow sites. - 30s navigation hard ceiling, OBSCURA_NAV_TIMEOUT_MS to override. - execute_scripts loop now checks wall-clock deadline per iteration. - Emit Network.requestWillBeSent before Page.frameNavigated so Playwright's FrameManager attaches the navigation response (#189). - Use loaderId as the Document request id so page.goto() resolves to a Response instead of null (#189). - Fresh executionContextId per navigation, no more 'Cannot find context with specified id' after the first nav (#192). - POST navigation params plumbed through the intercept path. DOM / Web API surface: - MutationObserver: subtree filter walks parent chain, fires on textContent / CharacterData.data / innerHTML, type filter respected, oldValue passed. - IntersectionObserver: multi-fire (initial + 120/500/1500ms), disconnect / unobserve / takeRecords, root / rootMargin / thresholds. - ResizeObserver: per-target firing from getBoundingClientRect, drop the no-op override that was clobbering the real class. - crypto.subtle.digest: real SHA-1/256/384/512 via op_subtle_digest. - DOMParser / document.implementation.createHTMLDocument / createDocument: detached doc with body, head, documentElement, querySelector*, getElementById, getElementsByTagName. - WebSocket / EventSource / BroadcastChannel: real EventTarget, open fires on microtask. - indexedDB: request, object store, transaction, database with async onsuccess and in-memory Map storage. - getComputedStyle: width/height/top/left from bbox so React virtualization libs render items. Site compatibility went from 3/40 to 175/175 reachable sites in the sweep. Perf is equal or faster than pristine main on every site tested (DCL default removes the block on long-tail subresources). 28/28 integration tests pass. |
||
|
SGavrl
|
e7bb0c676d |
feat(cli): --allow-private-network for localhost development (#33) (#203)
Picks up @Timtech4u's PR #33, which has been waiting since April. The design (CLI flag + threading through BrowserContext + http client) is theirs and is correct; this is a clean rewrite on top of current main because repeated rebase rounds had been reverting unrelated work that landed in the meantime. Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is process-wide and clunky in pipelines. Multiple users (@Timtech4u in the original PR, then @duartemvix and @necrogami in the comment thread) reported they cannot use obscura for development against http://localhost:N without it. Adds --allow-private-network as a global CLI flag, works on fetch, serve, and scrape. Threads through BrowserContext::with_storage_and_network and the http client's validate_url. CdpContext gets a new_full builder and obscura_cdp::start_with_full_serve_options entry point so 'obscura serve --allow-private-network' lets Puppeteer/Playwright navigate to localhost too. Default remains blocked. Backward compat: the env var still works (OR'd with the flag inside validate_url). All existing constructors and entry points unchanged. Verified end-to-end against a local http.server on 127.0.0.1: fetch and CDP Page.navigate both blocked by default, both succeed with the flag. Public URLs unaffected. Integration suite 28/28, full workspace cargo test all green. |
||
|
SGavrl
|
1a1253118f |
feat(cli): default V8 heap to 4 GB on 64-bit so heavy SPAs don't OOM (#199) (#202)
V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/ web-scraping) blow past it and the worker dies with 'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session with it. Always set --max-old-space-size=4096 (1024 on 32-bit targets) before V8 startup, matching headless Chrome's default. User-supplied --v8-flags are appended after the default so they win (V8 parses left-to-right and the last value wins), which keeps the existing escape hatch intact for memory-constrained environments. |
||
|
SGavrl
|
682431f766 |
fix(cdp): allocate fresh execution context ids per isolated world (#192) (#200)
Page.createIsolatedWorld and do_navigate both hardcoded context id 100 for every isolated world emission. After the second navigation Playwright expected a fresh incrementing id (101, 102) while obscura kept re-using 100. Runtime.evaluate then failed with 'Cannot find context with specified id: 101' because the id was never registered in valid_context_ids and the client's bookkeeping had diverged from the server's. Added next_isolated_context_id counter on CdpContext (starts at 100) and a next_isolated_context() helper that claims an id, registers it in valid_context_ids, and increments. Both call sites now use it. |
||
|
SGavrl
|
191c5ac7a2 |
fix(cdp): page.goto returns Response over Puppeteer and Playwright (#189) (#190)
Two correlation invariants real Chrome holds were missing, so the nav Response was never attached to goto(). requestId === loaderId for the main Document event. Both clients gate isNavigationRequest() on this; we were emitting a synthetic per-page id. Network.requestWillBeSent for the Document fires before Page.frameNavigated. Playwright's FrameManager needs the request present as pendingDocument before the frame commits, otherwise currentDocument lands with request=undefined. Subresource events and the Document's responseReceived / loadingFinished stay where they were. Verified end-to-end on both clients against example.com (200), google.com (200), httpbin/404 (404). |
||
|
SGavrl
|
d15cac6149 |
fix(js): Element.attributes, on* event defaults, per-nid getBoundingClientRect (#45) (#186)
Three DOM-completeness gaps that broke jQuery 1.11.3 init and Playwright
actionability:
1. Element.attributes was undefined, so legacy IE feature-detect code
like div.attributes[eventName].expando threw TypeError. Added a
NamedNodeMap-like backed by a new attribute_names DOM op (length,
numeric index, string name lookup, getNamedItem/setNamedItem,
iterable).
2. Standard on* event handler properties were missing from window, so
('on' + ev) in window returned false and forced jQuery into the
legacy IE branch. Initialised the 79 HTML-spec on* handlers to null.
3. getBoundingClientRect returned the same {x:8,y:8,width:100,height:20}
literal for every element, so Playwright's hit-testing couldn't tell
elements apart. Replaced with a deterministic synthesis from _nid
mapped into a 12-column grid sized to a 1280x720 viewport. Still not
pixel-accurate (no layout engine), but stable across reads and
distinct per node.
|
||
|
SGavrl
|
8a8c767338 |
fix(net): use bundled webpki roots in stealth client (#184) (#185)
set_default_paths only resolves OpenSSL's compile-time CA dirs, which exist on Linux but are empty on Windows. The stealth client's cert store ended up empty there and every TLS handshake failed with CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc. CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's trust list), which behaves the same on every platform and removes the dependency on a system CA store. |
||
|
SGavrl
|
52fcf2a35a |
fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes #113.
|
||
|
SGavrl
|
f4f49ebab4 |
fix(cdp): unblock puppeteer connect, bound Runtime.evaluate, full TargetInfo (#181)
Three independent gaps that hurt browser-based MCP clients connecting
to obscura's CDP endpoint:
- Runtime.enable returned "No page" when called before a session was
attached. puppeteer-extra's FrameManager.initialize fires it on the
browser-level WebSocket BEFORE any page target exists, so the
standard puppeteer.connect / browser.newPage flow died on connect.
Now reply with {} when no session is attached and let the next
Target.attachToTarget emit executionContextCreated.
- Runtime.evaluate had no per-call timeout. A long synchronous JS
expression held the global V8 lock indefinitely and starved every
other CDP command on the session. Honor the CDP `timeout` parameter
and default to 30s (Chrome's protocolTimeout). Returns a clear error
on expiry so client-side deadlines fire cleanly instead of hanging.
- Target.getTargetInfo for the browser target omitted canAccessOpener.
Strict CDP clients (chromiumoxide) panic when the field is missing.
Fill it in (false for the browser target itself).
Partial fix for #122. The remaining items (networkIdle hard-deadline
on SPAs, /json/list hardcoded data, docs on cheap-vs-queued methods)
need bigger or non-code changes and are deferred.
|
||
|
SGavrl
|
71b9b4fbab |
fix(js,dom): scope element queries, real timers, form/prepend/equal (#180)
- Element.querySelector / querySelectorAll scope to the receiver subtree via a new query_selector_scoped Rust op, not always document root. Same for DocumentFragment. - setTimeout respects its delay argument via a new async op_sleep. setInterval is a real repeating timer instead of an alias to setTimeout. - document.forms / images / links return live querySelectorAll results instead of hardcoded []. - HTMLFormElement subclasses Element with .elements / .length, and form nodes are wrapped as HTMLFormElement so instanceof works. submit() stays on Element.prototype to preserve the existing submit-event dispatch and form-data navigation path. - Element.prepend has a real implementation. - Node.isEqualNode does a structural compare (nodeType, nodeName, nodeValue, attributes, recursive children) instead of nid identity. - Fix long-standing insert_before arg-order bug: bootstrap.js was passing (parent, new, ref) but _dom only forwards two args, so ref was silently dropped. Pass (new, ref) directly. Same fix in replaceChild. Adds 3 selector unit tests and 6 runtime regression tests covering the JS shim fixes. Closes #105. |
||
|
SGavrl
|
d4d749d850 |
fix(mcp): allow browser MCP clients through CORS preflight (#179)
The OPTIONS preflight hardcoded Access-Control-Allow-Headers to just Content-Type. Any browser MCP client sending mcp-protocol-version (per the MCP spec) or Authorization / X-API-Key (typical for hosted deployments) failed the preflight and the browser blocked the actual request. Add those three to the allow-list and a 24h Max-Age so browsers don't re-preflight every call. Adds an integration test that drives the server with a real preflight request. Closes #175. |
||
|
SGavrl
|
a3b21bbb70 |
fix(cdp,js): playwright actionability stubs and objectId resolve (#178)
Playwright fails locator actions on Obscura for two avoidable reasons that are separate from the missing layout engine. Both have trivial fixes that unblock the actionability polling loop. - Element.checkVisibility(opts) was undefined. Playwright >= 1.25 calls it before every input event and retries until its action timeout if the method isn't a function. Add a stub that returns true. Without a layout pass we can't compute it for real, but a permissive stub still beats the undefined-function path. - ARIA reflection properties were missing on Element (role, ariaLabel, ariaSelected, ariaExpanded, ariaHidden, ariaChecked, ariaDisabled, ariaRoleDescription). Playwright's getByRole and getByLabel locators read them. Reflect through to the underlying aria-* attributes for both read and write. - DOM.getBoxModel and DOM.getContentQuads ignored the objectId param. Playwright passes only objectId (from a prior DOM.resolveNode), and the handlers silently defaulted to node 0, so dispatchMouseEvent always clicked the wrong element. Extract a resolve_node_id helper that honors nodeId, backendNodeId, and objectId in that order. - Page.captureScreenshot and Page.captureSnapshot now return a clear "not supported, use a real Chromium for the screenshot leg" error instead of the generic "Unknown Page method" fallback. Matches the pattern already in place for Page.printToPDF. Partial fix for #45. The architectural blocker (no layout engine, so multi-element click coordinates are wrong) is unchanged. |
||
|
SGavrl
|
e0af9fc44f |
fix(cdp): emit canAccessOpener in all TargetInfo payloads, add aarch64 cross-compile CI
Merges PR #165 and PR #172. PR #165: The CDP spec marks canAccessOpener as a required boolean on TargetInfo but Obscura was omitting it. Strict Rust clients like chromiumoxide reject the event entirely and panic with "Created target not present" when the field is absent. Added canAccessOpener: false to every TargetInfo emitted by setDiscoverTargets, getTargets, createTarget, attachToBrowserTarget, attachToTarget, and getTargetInfo. PR #172: Added a cross-compile matrix entry for aarch64-unknown-linux-gnu running on ubuntu-22.04 x86_64, producing obscura-aarch64-linux-cross. This is a fallback for when GitHub ARM runners are unavailable without removing the native ARM build. |
||
|
SGavrl
|
1950048138 |
fix(browser,cdp,js): about:blank navigation, real getBoxModel/getContentQuads, expose _wrap and self
- about: URL scheme now handled in navigate_single instead of hitting validate_url and failing with "Forbidden URL scheme" - DOM.getBoxModel calls getBoundingClientRect via JS instead of returning hardcoded pixel coords - DOM.getContentQuads added (was missing, needed by Playwright click) - globalThis._wrap exposed so CDP eval code can look up nodes by id - globalThis.self = globalThis for webpack/bundler worker detection |
||
|
SGavrl
|
913e6a4865 |
fix(browser,cdp): correct document lifecycle ordering and add missing DOM methods
document.readyState was hardcoded to "complete" so scripts checking
readyState === 'loading' always took the wrong branch. Fixed by using a
JS global (__documentReadyState__) that Rust sets to "loading" before
any scripts execute, "interactive" before DOMContentLoaded fires, and
"complete" after load.
document.addEventListener and document.dispatchEvent on Document were
no-ops, so DOMContentLoaded listeners registered by inline scripts were
never called. Added proper listener storage and dispatch on Document.
Adds document.getElementsByName() (was missing entirely).
Page.reload is now implemented (re-navigates to the current URL),
resolving "Unknown Page method: reload" errors from Playwright.
data: URL navigation now works, fixing Playwright's goto("data:text/html,...")
pattern that previously errored with "invalid format".
Page.navigate logic extracted into do_navigate() so reload can reuse it.
Fixes #137, #146 (SPA bootstrap lifecycle ordering).
|
||
|
SGavrl
|
e371e942fc |
fix(js): re-validate redirect targets in op_fetch_url
GHSA-8v6v-g4rh-jmcm: op_fetch_url validated the initial URL through validate_fetch_url, but reqwest auto-followed up to 10 redirects with no further checks. An attacker-allowed origin could 302 to http://127.0.0.1 (or any private/internal address) and read the body back to page JS. Real-world impact: AWS instance metadata, kube debug consoles, internal admin endpoints with permissive CORS all become readable from a page hosted anywhere on the open web. Disable reqwest's auto-redirect and follow manually. Apply validate_fetch_url to every redirect target before requesting it. Cap the chain at 10 hops to match prior behaviour. Browser-standard method/body rules apply on cross-method redirects (301/302/303 downgrade to GET with no body, 307/308 preserve). Verified vs unpatched build (pr69): - Patched: attacker-page -> 302 -> http://127.0.0.1:N is blocked with "Redirect to forbidden URL blocked: Access to private IP..." before the internal hop runs. Legitimate 3-hop redirect chain between allowed origins still completes. - Unpatched: internal-service body leaks through to page JS via fetch().then(r => r.text()). |
||
|
SGavrl
|
3158a1fd31 |
fix(cdp): gate Target.createTarget file:// like Page.navigate
GHSA-q55h-vfv9-qcr5 patched Page.navigate to refuse file:// URLs
unless --allow-file-access was set, but Target.createTarget accepted
the same scheme and called page.navigate(url) directly, bypassing
the policy. A CDP client could call Target.createTarget {url: file://...}
then Runtime.evaluate to read document.body.textContent and recover
arbitrary local files.
Apply the same gate at the Target.createTarget entrypoint. Pull
the file-scheme detector into a shared util module so both domains
stay in lock-step instead of one drifting from the other.
Verified blocked vs the unpatched build (pr69):
- Patched: Target.createTarget {url: file:///tmp/secret} errors
"Target.createTarget to file:// is disabled. ..." before any
navigation runs.
- Unpatched: same call returns the file contents via
Runtime.evaluate {expression: "document.body.textContent"}.
|
||
|
SGavrl
|
935cb5c0da |
fix(cli,js): add --version flag and script-error-resilience test (#147)
Two improvements while investigating #147: 1. `obscura --version` / `-V` now print the CARGO_PKG_VERSION instead of erroring out. The issue reporter explicitly called out that there was no version flag, and the existing `obscura serve` banner is the only way to discover the version. 2. New regression test in obscura-js verifies that a TypeError thrown from one script does not poison the runtime: subsequent scripts and DOM queries must continue to work. The reporter's symptom was that offside.js crashing collapsed --dump text output to 1 byte; that cascade should never happen and this test pins it down. The classList error itself is offside.js's own bug (it indexes into the querySelectorAll result without checking for an empty array) and only fires when the .slideout-navigation element is absent from the DOM. On current main that element parses correctly and offside.js runs without error against a captured copy of the affected page. |
||
|
SGavrl
|
8cb0cb429b |
fix(cli): refuse --stealth + socks5:// proxy instead of silent breakage (#160)
The previous workaround in page.rs rewrote socks5:// to http:// before handing the proxy to the stealth client. That works for Clash-style mixed-mode proxies, but breaks plain SOCKS5 servers like `ssh -ND`, producing a confusing `TunnelUnexpectedEof` deep in the network layer. The stealth client is backed by wreq, which does not speak SOCKS5, so silently substituting schemes can never be correct in general. Refuse the combination at CLI startup with a clear message that tells the user what to do (use --proxy http://... or drop --stealth) and revert the rewrite in Page::new. SOCKS5 without --stealth still works through the reqwest path. |
||
|
SGavrl
|
b2b41529c4 |
ci: add aarch64-linux release binary (#127)
Uses GitHub's native ubuntu-22.04-arm runner so it's a native build matching the same glibc 2.35 floor as the x86_64 target. README gets the matching install snippet. |
||
|
SGavrl
|
aaeaf8f662 | chore: drop author working notes that landed with #69 | ||
|
SGavrl
|
b07863b5ce | chore: drop author working notes that landed with #69 | ||
|
SGavrl
|
9c6dec4086 | chore: drop author working notes that landed with #69 | ||
|
SGavrl
|
3a6f3e6c22 |
Update README.md
cleaning up |
||
|
SGavrl
|
18dc796df0 |
fix(cdp): gate file:// navigation behind --allow-file-access
GHSA-q55h-vfv9-qcr5. Any process able to reach the CDP port could
send Page.navigate {"url":"file:///etc/shadow"} and read arbitrary
local files (the process's UID's worth of read access). When obscura
runs as root inside Docker or as a systemd service that translates
straight into /etc/shadow, /root/.ssh/id_rsa, /proc/1/environ, etc.
Validate at the CDP boundary, not in obscura-net, so direct CLI
callers ("obscura fetch file://...") keep working. A new
BrowserContext.allow_file_access flag defaults to false; Page.navigate
returns an error when the requested URL is file:// and the flag is
off. CdpContext::new_with_security threads the flag from the server
entry point, and "obscura serve --allow-file-access" opts in for
local-HTML testing workflows.
Tested:
- Default obscura serve: CDP Page.navigate {file:///etc/passwd} ->
refused with descriptive error.
- obscura serve --allow-file-access: same call succeeds.
- obscura fetch file://... CLI path: unchanged, still reads the file.
- All earlier PoCs (GHSA-6pj9 JS->file SOP, GHSA-4789 sub-resource
SSRF, GHSA-4789 CSS template injection) still pass.
- cargo test --workspace clean.
|
||
|
SGavrl
|
ac0efd1798 |
fix(browser): patch sub-resource SSRF, CSS injection, and silent redirect cap
Three findings from GHSA-4789-rq5x-jv3c.
HIGH-1: Sub-resource fetches (<script src>, <link rel=stylesheet>) had
no scheme gate, so an http(s) page could include
<script src="file:///etc/passwd"> and the body would come back as
script source. Added subresource_allowed() that mirrors the top-level
file:// SOP rule: file: sub-resources only load when the page itself
is on file:. Off-origin schemes (data:, javascript:, etc.) are
refused regardless. Private/loopback hosts were already blocked at
the obscura-net layer; this closes the file:// hole on top of that.
HIGH-2: CSS pulled from the network was string-formatted into a JS
template literal with a three-rule escaper (backslash, backtick,
${). That escaper did not handle U+2028 / U+2029 which terminate
ECMAScript template literals, letting attacker CSS break out and
run arbitrary code in the page's V8 realm. Replaced the inline
escape with escape_for_js_template_literal() which also handles
U+2028, U+2029, null, carriage return, and every other C0 control.
LOW-2: navigate_with_wait_post() capped JS-driven navigation chains
at ten but returned Ok(()) when the cap was hit, so callers could
not tell a successful load from a redirect storm. Now returns
PageError::TooManyRedirects(REDIRECT_LIMIT).
Verified with three PoCs (sub-resource SSRF, CSS template literal
break with U+2028, and the original GHSA-6pj9-j389-226p reproducer)
plus negative tests (file:// direct read, http -> http JS chain) all
pass. cargo test --workspace clean.
|
||
|
SGavrl
|
9edde41faf |
fix(browser): block JS-driven http -> file navigation, reset realm per nav
A page on http(s) could set window.onload, assign location.href to a file:// URL, and then read document.body from the local file because two things lined up: obscura-net allows file: URLs, and the JS runtime was reused across navigations so window state from the prior origin survived into the new document's load event. Two fixes: 1. Refuse a JS-initiated chain step when it crosses from a non-file scheme into file:. Direct file:// navigations from the CLI/CDP still work because the gate only fires on the take_pending_nav path. file -> file chains also still work. 2. Drop and rebuild the V8 runtime on every navigation instead of only rebinding globalThis.document. Even within the same origin we should not carry over window.onload, custom properties, or registered event listeners from the previous document. Tested with the PoC from the report: pages on http://127.0.0.1.nip.io no longer leak the contents of file:///tmp/.../pwn.txt. |
||
|
SGavrl
|
aeb1784c32 | docs: add Docker install section to README | ||
|
SGavrl
|
56260822b9 |
ci: auto-build and push Docker image on release tag
Adds a workflow that triggers on v* tag pushes and: - Logs into Docker Hub using DOCKERHUB_USERNAME + DOCKERHUB_TOKEN secrets - Builds the image from the existing Dockerfile - Tags it with both the version (e.g. 0.1.4) and 'latest' - Pushes both tags GitHub Actions cache is used for layers to speed up subsequent builds. |
||
|
SGavrl
|
a23cdc0ae7 |
feat: add Dockerfile + --host flag for containerized deployment
Adds a multi-stage Dockerfile (rust:1-slim-bookworm builder → distroless/cc runtime) plus a --host flag for obscura serve that defaults to 127.0.0.1 for safety but can be set to 0.0.0.0 so the container's port is reachable via docker run -p mapping. Dockerfile cherry-picked from #84. The SPA chunk loading commit from that PR was left out — verified it didn't actually fix the Next.js webpack errors on kiwi.com (#86). |