* fix(js): fire trusted input/change from setFileInputFiles
DOM.setFileInputFiles dispatched plain input/change events, so
event.isTrusted was false. Upload flows and anti-bot code that gate their
change handler on isTrusted ignored them, which silently broke the exact
case the feature targets. The Input domain already marks its synthesized
clicks/keys trusted via __obscura_markTrusted; do the same here.
Fixes#384.
* fix(cdp): bound IO stream buffers so downloads cannot leak
Fetch.takeResponseBodyAsStream moved a response body out of the page's
LRU-bounded cache into CdpContext.io_streams, a server-lifetime map freed only
by an explicit IO.close. A client that opens a stream and disconnects, errors
mid-download, or never closes it leaked the full buffer forever; no teardown
path (WS close, Browser.close, Target.closeTarget) touched io_streams. That
reintroduced the unbounded memory growth streaming was meant to avoid (#360).
Replace the raw map + counter with an IoStreamStore that caps the number of
open streams and their total bytes (OBSCURA_IO_STREAM_MAX_ENTRIES / _BYTES),
evicting the oldest first and always keeping the just-opened stream. Reading an
evicted handle fails cleanly, which is the right trade against an OOM. Read
chunking is unchanged.
Fixes#386.
Cookie path filtering used `request_path.starts_with(cookie_path)`, so a
Path=/admin cookie was sent to sibling paths that merely share the string
prefix, such as /administrator or /admin-backup. Per RFC 6265 5.1.4 a prefix
match also requires the boundary to fall on a '/'.
Add a path_matches helper (identical path, or a prefix ending in '/', or a
prefix followed by '/') and use it in get_cookie_header and
get_js_visible_cookies.
Fixes#382.
DOM.setFileInputFiles dispatched plain input/change events, so
event.isTrusted was false. Upload flows and anti-bot code that gate their
change handler on isTrusted ignored them, which silently broke the exact
case the feature targets. The Input domain already marks its synthesized
clicks/keys trusted via __obscura_markTrusted; do the same here.
Fixes#384.
* fix(stealth): hide internal globals from reflection and mark builtins native
Two anti-fingerprinting leaks let pixelscan's bot-check flag obscura as
automated even with --stealth:
- unusualWindowProperties: internal globals (_fp, _wrap, __obscura_*, ...)
were only made non-enumerable, so they stayed hidden from Object.keys and
for-in but not from Object.getOwnPropertyNames / Reflect.ownKeys, which
fingerprinting scripts use. Filter __obscura_hide_list out of the reflection
APIs (getOwnPropertyNames, ownKeys, keys, getOwnPropertyDescriptors) when
they target the global object.
- tamperedFunctions: DOM/Web platform members reimplemented in JS leaked their
source via toString(), e.g. Element.prototype.nodeType returned
"get nodeType() {...}" instead of "[native code]". Per-member _markNative
calls covered methods but missed property accessors and several constructors.
Sweep every builtin constructor reachable from the global object at snapshot
build time and mark its prototype methods, accessors, and the constructor
itself native. Also name the Function.prototype.toString override so it
reports the correct native form for itself.
Verified against pixelscan.net/bot-check: Webdriver -> Clear (tamperedFunctions
and unusualWindowProperties both Clear).
* fix(stealth): build hide list with getOwnPropertyNames
__obscura_hide_list was built with Object.keys, which omits the internal
globals that _preHideInternals already made non-enumerable. Those names were
therefore missing from the list and stayed visible to the reflection-API
filter and to fingerprinting scripts (e.g. __obscura_ua, _registerIframe).
Use getOwnPropertyNames so the list is complete.
Network.getResponseBody returns the whole body in one base64 blob and the
cache drops anything over 2 MiB, so large downloads (images, PDFs,
archives) either failed or forced the long-running server to hold the
full body in memory (issue #360).
Add Fetch.takeResponseBodyAsStream + IO.read + IO.close. The body is moved
out of the page cache into a stream handle and delivered in bounded chunks
via IO.read, then freed on IO.close, so it is held once and released after
the download. Binary bodies are byte-exact (stored base64). To stream a
large body, raise OBSCURA_NETWORK_BODY_BUFFER_BYTES so it is cached.
This is entirely opt-in: nothing runs unless a client calls the new
methods, so the default fetch/render/scrape path is unchanged (verified:
obstacle course 33/33, obscura-vs-Chrome head-to-head within noise of the
pre-change baseline).
Fixes#360.
load_module and load_inline_module drove run_event_loop to full idle
before reading the module's mod_evaluate result. A page timer keeps the
loop busy forever (Vite's HMR / React-Refresh client installs a
setInterval), so the preamble inline module burned the entire budget
waiting for an idle that never came, and the later module that mounts the
app was skipped, leaving #root empty while the production preview rendered
fine (issue #374).
Drive the event loop and the mod_evaluate result together and return as
soon as the module finishes evaluating, so a busy background timer no
longer starves the remaining modules. Post-eval async work is still
drained by the normal post-load settle.
Fixes#374.
handle_fetch_resolution built InterceptResolution::Continue with url, method,
headers, and body hardcoded to None, so Fetch.continueRequest silently sent the
request unmodified. Playwright route.continue({ headers, url, postData }) and
Puppeteer request.continue reported success while their overrides were dropped.
op_fetch_url already applies all four (and re-validates a rewritten URL through
the SSRF gate); only the CDP layer was discarding them. Read url, method,
headers, and postData from the continueRequest params and pass them through.
Fixes#365.
The cdp_processor decided whether to take the spawn-and-defer navigation path
with `text.contains("Page.navigate")`, which also matches
`Page.navigateToHistoryEntry`. goBack / goForward (both Puppeteer and
Playwright drive Page.navigateToHistoryEntry) were misrouted into the raw
navigate path, which reads an absent `url` param and fails with
"Invalid URL: relative URL without a base", so history navigation never
reached its real handler. Any frame merely embedding the literal
"Page.navigate" (e.g. a Runtime.evaluate expression) was misrouted the same
way.
Match on the parsed CDP method instead, so only an exact Page.navigate takes
the navigation path.
Fixes#363.
The script-execution phase had a 10s soft deadline. On a heavy SPA the
remaining scripts were skipped before the app finished booting, so it
never fired its XHR/fetch calls and page.on('response') captured zero API
responses (issue #361). The knob to extend it, OBSCURA_SCRIPT_DEADLINE_MS,
existed but was undocumented, so it was undiscoverable.
Raise the default to 30s and document the env var. Only pages that keep
running past the deadline are affected; fast pages finish and return well
before it, so the larger budget costs them nothing, and the per-phase
watchdog (armed at deadline + 1s) still bounds a real synchronous hang.
Very heavy SPAs on slow networks can raise it further with
OBSCURA_SCRIPT_DEADLINE_MS, paired with a matching client nav timeout.
Fixes#361.
obscura serve --workers N --proxy ... spawned each worker with the proxy
as a --proxy CLI flag, so the plaintext credential was visible to any
local user via ps / /proc/<pid>/cmdline (issue #366). scrape already
avoids this by passing the proxy in OBSCURA_PROXY, which is only readable
by the owner.
Pass the proxy to serve workers via OBSCURA_PROXY too, and have the serve
command read that env as a fallback when no --proxy flag is given. The
worker argv no longer contains the credential.
Fixes#366.
Puppeteer's ElementHandle.uploadFile and Playwright's setInputFiles drive
an <input type=file> through CDP DOM.setFileInputFiles, which obscura
rejected as an unknown method, so any upload flow failed (issue #359).
Implement it: resolve the node, read each local file, and hand the bytes
(base64) to the JS layer, which builds real File objects (backed by the
bytes, so page code can read them via FileReader or upload them via
fetch/FormData) and fires input+change like a genuine selection. The file
input exposes a FileList via .files, and .value reads C:\fakepath\<name>
to match Chrome.
Fixes#359.
extract_readable_text recursed once per DOM nesting level, so a deeply
nested page overflowed the call stack and aborted the process (exit 134)
on --dump text. This is the CLI counterpart of the serialize/textContent
walks made iterative in obscura-dom; that fix did not cover this function.
Rewrite it as an iterative DFS over an explicit work stack, with a
Newline work item for a block element's trailing newline so output is
byte-for-byte unchanged, plus the same defense-in-depth node cap the DOM
walkers use. Relates to #362.
serialize_node/serialize_children, collect_text_inner (textContent), and
import_node_from recursed once per DOM nesting level with no depth cap, so a
deeply nested tree overflowed the thread stack and aborted the process. A
stack overflow is a hard abort that op_dom's catch_unwind cannot recover, so
one hostile page could take down the worker. descendants() was already made
iterative and capped for exactly this reason; these paths were missed.
Rewrite all three as iterative walks over an explicit heap stack and add the
same defense-in-depth cap so a cyclic graph stops instead of spinning.
Serialized output and traversal order are unchanged. Adds tests that build a
100k-deep tree and exercise each path.
Closes#364.
Regression test for #355: an uncaught `throw` in one inline <script> must
not prevent later independent <script>s from running. This was fixed by
the #147 poison-guard (present on main) but reported again against 0.1.9
with an explicit throw; this pins the exact scenario so it cannot regress.
Closes#355.
el.style (CSSStyleDeclaration) and el.dataset (DOMStringMap) were plain
Proxies with only get/set traps, so `'color' in el.style`, `Object.keys
(el.style)`, `'foo' in el.dataset`, and `Object.keys(el.dataset)` all
failed, and Object.keys(el.style) leaked the internal `_props` (issue
#356). cssText also serialized camelCase names without a trailing
semicolon.
Add has / ownKeys / getOwnPropertyDescriptor traps to both proxies so the
`in` operator and enumeration behave like real CSSOM. CSSStyleDeclaration
now stores properties by their dashed name (matching the spec), so
el.style.fontSize and getPropertyValue('font-size') stay in sync, and
cssText serializes dashed names with a trailing semicolon. A known CSS
property list backs `in`/enumeration so feature detection ('object-fit'
in el.style) works for unset properties.
Closes#356.
Typing over CDP into a React controlled input never fired onChange, so
the value never reached component state (issue #324). Two root causes,
both needed:
1. Value tracker. React/Preact/Vue install a value tracker by redefining
`value` on the element instance to record the last value they wrote.
obscura's CDP typing assigned `el.value = ...` directly, which runs
that wrapper and keeps the tracker in sync, so React saw no change on
the input event. Synthetic value/backspace writes now go through a new
__obscura_setFieldValue helper that writes via the element's prototype
setter, bypassing the tracker. The tracker stays stale, so the input
event reads as a real edit.
2. Event handler feature detection. React decides whether to use the
modern `input` event via `("oninput" in document)`. obscura exposed the
GlobalEventHandlers on* attributes only on window, so that check failed
and React fell back to a legacy change-detection path that ignores
input events. The on* attributes are now also present on Document and
Element, matching the HTML spec (GlobalEventHandlers is mixed into
Window, Document, and HTMLElement).
Verified end to end over CDP against the real React 18 UMD build: typing
into controlled inputs updates state, the component re-renders, and a
submit handler reads the captured values.
Closes#324.
fetch handled one URL at a time, so checking many raw resources meant
wrapping it in an external loop. Add --file <path> to read newline-
delimited URLs (- for stdin; blank lines and # comments skipped) and
--concurrency <N> to fetch them in parallel bounded by a semaphore.
Batch mode is raw HTTP only (--dump original, the default in this mode)
and prints one JSON status line per URL (url, ok, status, content_type,
bytes, elapsed_ms, or error) in input order. Rendered/DOM batch work
still belongs to scrape. A single positional URL is unchanged.
Fixes#349.
new Image() used a bare class shim with no .style, so new Image().style
returned undefined and libraries touching style on a detached image threw
(issue #350). Build the image from document.createElement('img') instead,
which gives a working CSSStyleDeclaration, attribute reflection, and event
dispatch. Assigning .src still emulates a successful decode: it flips
complete and fires load (both the onload property and load listeners) so
lazy loaders and preloaders do not hang.
The DOMSnapshot walk recursed one frame per nesting level. The MAX_NODES
cap bounds the node count but not the depth, so a deeply nested linear
chain (script can build thousands of nested elements) could overflow the
stack before the count guard triggered. Rewrite it with an explicit stack
so depth lives on the heap. Pre-order traversal and the MAX_NODES cap are
preserved.
Fixes#341.
The comment serializer wrote its contents verbatim. The HTML parser can
never produce a comment whose data contains "-->" (it ends the comment),
but script can via document.createComment("a-->b"). Serializing that
verbatim closed the comment early and let the trailing text escape into
markup. Replace "-->" with "-->" so the output round-trips as a single
comment.
Fixes#342.
These Chrome CDP methods logged "Unknown method" warnings. obscura runs
one page per session and does not download to disk, so there is nothing
to do, but clients that call them should get a success ack instead of a
warning. Added as no-op stubs.
Refs #340.
Navigating straight to an image, PDF, or other resource could not read the
main response body over CDP, and Page.frameNavigated always reported
text/html.
Two problems: the main resource's body was stored under its internal
request id, but the client sees it as the navigation loaderId (Chrome's
requestId === loaderId convention), so Network.getResponseBody(loaderId)
missed. And binary bodies were stored as UTF-8-lossy text, corrupting the
bytes. Alias the stored body to the loaderId, store non-text-like content
types (images, PDF, octet-stream) base64, and report the response's real
Content-Type in frameNavigated. HTML pages are unchanged.
Fixes#340.
crypto.subtle only implemented digest. sign returned a fixed 32-byte
buffer, verify always returned true, encrypt and decrypt threw, and
generateKey, importKey and deriveBits handed back placeholder data.
Pages that use WebCrypto (anti-bot probes, PKCE and other auth flows,
client-side crypto) silently got wrong results.
Implement the secret-key half of WebCrypto on RustCrypto, following the
existing op_subtle_digest pattern:
- HMAC sign and verify (SHA-1/256/384/512)
- AES-GCM, AES-CBC and AES-CTR encrypt and decrypt
- PBKDF2 and HKDF deriveBits and deriveKey
- raw and symmetric JWK (oct) importKey and exportKey, generateKey,
wrapKey and unwrapKey
- getRandomValues and randomUUID now draw from the OS CSPRNG instead of
Math.random, so they are uniform across typed-array widths and no
longer a fingerprinting tell
Unsupported algorithms (RSA, ECDSA, ECDH) and key formats (pkcs8, spki)
throw NotSupportedError instead of returning fake data. The crypto runs
in stateless Rust ops; the bootstrap.js shim only marshals bytes and
normalizes algorithm parameters, so the native fast paths are untouched
and there is no cost unless a page calls crypto.
Verified against RFC 4231 (HMAC), the NIST GCM vectors, NIST SP800-38A
(AES-CTR), RFC 6070 (PBKDF2) and RFC 5869 (HKDF), plus roundtrip and
tag/padding tamper-rejection checks.
Two standard-conformance fixes surfaced by #335 (the issue itself is a commercial anti-bot challenge and out of scope; these are the legitimate browser-compatibility gaps from it).
• checkbox/radio value: an input with no value attribute returns "on" in a real browser; obscura returned "". An explicit value attribute still wins.
• NodeList: childNodes returned a plain Array, and NodeList extended Array, so Array.isArray(list) was true and Object.prototype.toString reported "[object Array]". A real browser reports "[object NodeList]" with Array.isArray false. NodeList is now its own type with the array-like surface scripts use (indexed access, length, item, forEach, entries/keys/values, iteration), and childNodes goes through it.
Verified: checkbox/radio default "on" (explicit value preserved), childNodes and querySelectorAll are real NodeLists (constructor "NodeList", "[object NodeList]", Array.isArray false), and spread / for..of / Array.from still work. Obstacle course 33/33, obscura-dom 33/33.
Refs #335.
childNodes returned a plain Array (constructor "Array"), and NodeList
extended Array, so Array.isArray(list) was true and
Object.prototype.toString reported "[object Array]". A real browser
reports "[object NodeList]" and Array.isArray is false, which
feature-detection and fingerprinting scripts check.
NodeList is now its own type with the array-like surface scripts use
(indexed access, length, item, forEach, entries/keys/values, iteration),
and childNodes goes through it. Verified spread, for..of, and Array.from
still work.
Refs #335.
A checkbox or radio input with no value attribute returns "on" in a real
browser; obscura returned the empty string. Page scripts that read
input.value to build form submissions (and browser-environment checks)
saw the wrong default. An explicit value attribute still wins.
Refs #335.
The single-worker serve path honored --host, but run_multi_worker_serve hardcoded the load balancer to 127.0.0.1. With --host 0.0.0.0 and --workers >1 (the common Docker setup) the balancer bound loopback only, so the mapped port was refused from outside the container. Bind the balancer to the requested host. Workers stay on loopback and are reached only by the balancer.
Closes#336.
The single-worker serve path honored --host, but run_multi_worker_serve
hardcoded the load balancer to 127.0.0.1. With --host 0.0.0.0 and
--workers >1 (the common Docker setup) the balancer bound loopback only,
so the mapped port was refused from outside the container. Bind the
balancer to the requested host. Workers stay on loopback and are reached
only by the balancer.
Fixes#336.
serve --storage-dir saved cookies only on Ctrl-C (SIGINT). docker stop
and kill send SIGTERM, which was not caught, so the save_cookies() call
at shutdown never ran and cookies were lost on restart. Watch SIGTERM as
well as Ctrl-C on Unix.
querySelectorAll / --selector silently returned nothing for :has().
Two fixes were needed:
- Enable :has() parsing: the selectors crate gates it on the parser's
parse_has() (default false), so a:has(p) failed to parse and the error
was swallowed into an empty match set.
- Fix OpaqueElement stability: opaque() was keyed off the transient
DomElement (which is Copy and gets a fresh stack address each traversal
step), so the same node produced a different id every call. :has anchor
matching compares the anchor's opaque against the element reached by
walking up the tree, so it never matched. Key opaque() off the node's
stable slot in the tree's Vec instead.
SVG className and href reflected as plain strings, so el.className.animVal
and el.href.animVal were undefined, which made new URL(svgEl.href.animVal)
throw and broke any SVG-aware script. Add an SVGAnimatedString class
(baseVal/animVal both read the live attribute; baseVal writable) and make
the className and href getters return it for SVG-namespace elements. href
falls back to the legacy xlink:href. HTML elements are unchanged.
* Add fetch body serialization support for Blob, ArrayBuffer, and TypedArray
* js: fix Blob fetch body serialization
The Blob branch passed the Blob to _bodyToUint8Array, which did not know
about obscura's Blob (its data lives in _bytes) and fell through to
encoding String(blob), so a Blob body was sent as "[object Blob]".
Teach _bodyToUint8Array about _bytes, and build the binary string with
the loop-based _bytesToBinaryString instead of String.fromCharCode.apply
so large blobs do not overflow the argument limit.
---------
Co-authored-by: SGavrl <277843151+SGavrl@users.noreply.github.com>
CookieJar::set_cookie and set_cookie_from_js started from the request host but
blindly overwrote it with any Domain attribute, so a response from
attacker.test could plant a cookie scoped to victim.test, which was then sent
on later requests to victim.test. The same sink backs document.cookie, HTTP
Set-Cookie, and CookieStore::set.
Add resolve_cookie_domain: with no Domain the cookie is host-only (exact host
only); a Domain attribute is honored only when it domain-matches the origin
(equal or a parent domain) and is not an obvious public suffix, otherwise the
attribute is ignored and the cookie is stored host-only on the origin. Track
host_only on CookieEntry so host-only cookies are not sent to subdomains.
A full public suffix list is not bundled, so multi-label public suffixes
(co.uk, github.io) are not rejected; the domain-match check still blocks the
reported cross-domain attack, and single-label suffixes (com, local) are.
Tests cover the cross-domain rejection for both HTTP and document.cookie,
public-suffix rejection, host-only subdomain scoping, and that legitimate
parent-domain cookies still work.
Co-authored-by: DavidCarliez <DavidCarliez@users.noreply.github.com>
The request interception API (#306) and the global --stealth flag are
merged, so:
- Use-as-a-Rust-library.md: add an "Intercept requests" section covering
on_request/on_response, enable_interception (Continue/Fulfill/Fail),
and add_preload_script, with examples.
- Intercept-and-modify-requests.md: point embedders at the native API.
- AGENTS.md: --stealth is now global (all subcommands incl. scrape, via
OBSCURA_STEALTH for workers); note the interception API on the obscura
crate and that op_fetch_url rewrites must stay behind validate_fetch_url.
- skills/obscura/SKILL.md: fix the build note (stealth is not a default
feature), correct "randomized" to a consistent fingerprint, add the
global --stealth flag, --dump assets/cookies, and an interception note.
- --stealth is now a global flag: document it on every subcommand
including scrape, and that it works before or after the subcommand.
- Correct the stealth description: it presents a consistent Chrome
fingerprint (TLS, headers, JS surfaces), not a randomized one, and
scripted fetch()/XHR share that fingerprint.
- Add --dump cookies to the CLI reference and the extract-data guide.
- Note --dump assets now also lists fetch()/XHR request URLs.
GitHub issue forms that enforce the fields needed to triage: obscura
version, OS and architecture, how it is being used, stealth mode, a
reproduction, expected vs actual, and whether headless Chrome behaves
the same. config.yml routes questions and API proposals to Discussions
and security reports to the private advisory flow.
Andrey Mnatsakanov