The wreq dependency was declared without the gzip, brotli, deflate and zstd
features, so the stealth client returned bodies still compressed while the
reqwest path decoded them. The Chrome145 emulation advertises those encodings,
so origins do compress, and the raw gzip bytes reached the HTML parser as
document text.
Add a regression test that serves one gzip response and asserts the stealth
client decodes it.
Fixes#563.
The stealth (wreq) client gated its custom-trust-store build on
`var_os(...).is_some()`, which is true for a set-but-empty variable. Supplying
a store to wreq's `tls_cert_store` REPLACES the bundled webpki roots, so an
empty `SSL_CERT_FILE` (a common shell accident, e.g. `export SSL_CERT_FILE=$UNSET`)
built a near-empty default-paths store and made every HTTPS request fail with
CERTIFICATE_VERIFY_FAILED -- while the reqwest client, which already filters
empty values in configured_root_paths(), kept working. Same input, opposite
behavior.
Extract a shared `custom_cert_store_requested` predicate in client.rs that
applies the same `!is_empty()` filter, and call it from the stealth client's
guard. The predicate takes the values as arguments so it is unit-tested under
default features (the stealth feature pulls in boringssl and is not built in
the normal test job).
Fixes#551
The #525 fix landed on the reqwest path (`client.rs`: `configured_root_paths()`
-> `add_root_certificate`) but did not cover `wreq_client.rs`. The stealth
client builds its `wreq::Client` without a certificate store, so those
variables are silently ignored whenever `--stealth` is on.
The consequence is inverted: the transport with the best TLS fingerprint is the
only one that cannot reach a host behind a private or incomplete CA chain --
precisely the environments #525 described.
Same binary on current main, same private-CA server, only the flag differs:
SSL_CERT_FILE=ca.pem obscura fetch https://127.0.0.1:8443/ -> ok
SSL_CERT_FILE=ca.pem obscura --stealth fetch https://127.0.0.1:8443/ -> fails
`wreq` already exposes the hook -- `ClientBuilder::tls_cert_store(CertStore)`,
and `CertStore::builder().set_default_paths()` reads both variables. It is
simply never called.
Two constraints, both of which fail silently if got wrong:
* `tls_cert_store`, not `tls_options`. `emulation()` is documented as
"This will overwrite the existing configuration" and internally calls
`self.tls_options(emulation.tls_options)`; setting TLS options here would
discard the Chrome fingerprint, which is the whole point of this client.
`tls_cert_store` is a separate config field and composes with emulation.
* Opt-in only. Supplying a store REPLACES the webpki roots (see
`set_cert_store` in `tls/conn/ext.rs`), it does not add to them. Applying it
unconditionally would break ordinary sites whenever the bundle is
incomplete. With neither variable set, behaviour is byte-for-byte unchanged.
Verified: with the variable, a private-CA host resolves under --stealth;
without it, the same host still fails; with a deliberately EMPTY bundle,
example.com fails -- which is what proves the store is actually applied rather
than the bundle merely happening to be right.
Refs #538, #525
reqwest was pinned to rustls-tls, so HTTPS only trusted the bundled
webpki roots: SSL_CERT_FILE, SSL_CERT_DIR, and the system CA store were
ignored, and obscura-net exposes no add-CA hook. Deployments behind a
TLS-inspecting egress proxy or an internal CA cannot fetch at all, and a
caching proxy in front of --proxy cannot terminate TLS.
Enable rustls-tls-native-roots alongside rustls-tls. reqwest merges the
two root stores into one, so the default trust surface only grows and
certificate verification is otherwise unchanged: unknown CAs are still
rejected. The stealth (wreq) client has its own TLS stack and is out of
scope here.
Tests: an in-process HTTPS fixture with an rcgen-minted private CA,
served via tokio-rustls (pinned to the ring provider so the test binary
has a single rustls CryptoProvider). With SSL_CERT_FILE pointing at the
CA the fetch succeeds; without it the same fixture is rejected.
Fixes#525.
CookieJar stored cookies as domain -> name -> entry, so a second
Set-Cookie with the same name but a different path silently overwrote the
first (RFC 6265 §5.3 identifies a cookie by name+domain+path). The
retrieval side already treated path as part of identity — get_cookie_header
path-matches, and delete_cookies_filtered keys on (name, path) — so only
storage lost cookies.
Rekey the inner map by (name, path). A Max-Age=0 / expired Set-Cookie now
deletes exactly the (name, path) it targets instead of every same-name
cookie; delete_cookie(name) still clears all paths (retain by entry name).
Adds tests: same-name/different-path coexistence and no cross-path leak,
same-name/same-path replacement, and path-scoped deletion leaving a
sibling-path cookie intact.
The on_request/on_response Vec lived on the ObscuraHttpClient shared by
every page in the context, so a callback registered on one page fired for
all pages (#408 cross-page bleed) and outlived its page.
Callbacks now live in a per-page CallbackRegistry (Arc shared with the
page's JS runtime state). Page-driven fetch paths (navigation, POST,
scripts, stylesheets, robots.txt, module loads, JS fetch/XHR in both
plain and stealth modes) pass the registry through; the HTTP client
itself is callback-free.
Adds callbacks_do_not_bleed_across_pages: page A's callback stays silent
for page B's requests and stops firing after page A is dropped.
on_request/on_response stored callbacks in append-only Vecs with no way
to remove one, so a crawler that installed a response-body capture hook
for one phase could never detach it: it fired for the shared client's
whole lifetime, re-registering across navigations piled up duplicates,
and (because the client is shared per context) it saw sibling pages'
requests too.
Tag each callback with a stable id. on_request/on_response now return
the id and off_request/off_response (remove_on_request/remove_on_response
on the client) detach a specific one. Storage is Vec<(u64, Callback)>;
all fire sites destructure the tuple. Closes#408.
* fix(net): default cookie Path to the request directory (RFC 6265 5.1.4)
When a Set-Cookie carried no Path attribute, the jar stored the full request
path as the cookie's path. RFC 6265 5.1.4 default-path is the request URI's
directory — the path up to but not including the right-most '/'. Storing the
full path scoped a session cookie to the exact URL that set it: a cookie set on
GET /app/login was stored with path=/app/login and then, because path-match
(5.1.4, fixed earlier) requires a '/' boundary, was not sent to /app/dashboard.
The user appeared logged out on the very next navigation.
Add default_cookie_path() implementing the RFC algorithm and use it as the
initial path in both set_cookie (HTTP) and set_cookie_from_js (document.cookie).
An explicit Path attribute still overrides it. Tests cover the algorithm and the
end-to-end 'cookie set on /app/login is sent to /app/dashboard' path for both
the HTTP and JS entry points; both failed before this change.
* fix(cdp): apply RFC 6265 default cookie Path to CDP-injected cookies
Network.setCookie/setCookies (the Puppeteer page.setCookie and
Playwright addCookies surface) derived the cookie Path from the full
request URL when the caller omitted path, the same bug the parent
commit fixed for Set-Cookie responses. Reuse default_cookie_path (now
pub) so a cookie set under /v1/things scopes to /v1 and is resent on
sibling paths. Fixes the CDP half of #401.
Correct the test that locked in the old full-path behavior and add a
case confirming an explicit Path still wins.
---------
Co-authored-by: SGavrl <277843151+SGavrl@users.noreply.github.com>
Cookie path filtering used `request_path.starts_with(cookie_path)`, so a
Path=/admin cookie was sent to sibling paths that merely share the string
prefix, such as /administrator or /admin-backup. Per RFC 6265 5.1.4 a prefix
match also requires the boundary to fall on a '/'.
Add a path_matches helper (identical path, or a prefix ending in '/', or a
prefix followed by '/') and use it in get_cookie_header and
get_js_visible_cookies.
Fixes#382.
CookieJar::set_cookie and set_cookie_from_js started from the request host but
blindly overwrote it with any Domain attribute, so a response from
attacker.test could plant a cookie scoped to victim.test, which was then sent
on later requests to victim.test. The same sink backs document.cookie, HTTP
Set-Cookie, and CookieStore::set.
Add resolve_cookie_domain: with no Domain the cookie is host-only (exact host
only); a Domain attribute is honored only when it domain-matches the origin
(equal or a parent domain) and is not an obvious public suffix, otherwise the
attribute is ignored and the cookie is stored host-only on the origin. Track
host_only on CookieEntry so host-only cookies are not sent to subdomains.
A full public suffix list is not bundled, so multi-label public suffixes
(co.uk, github.io) are not rejected; the domain-match check still blocks the
reported cross-domain attack, and single-label suffixes (com, local) are.
Tests cover the cross-domain rejection for both HTTP and document.cookie,
public-suffix rejection, host-only subdomain scoping, and that legitimate
parent-domain cookies still work.
Co-authored-by: DavidCarliez <DavidCarliez@users.noreply.github.com>
Expose the interception infrastructure that already existed internally but was
private, so embedders can intercept, observe, mock, or block JS fetch()/XHR
without forking or unsafe casts.
On obscura::Page:
- add_preload_script: inject JS before any page <script> (CDP
Page.addScriptToEvaluateOnNewDocument). The field already ran the scripts;
this only adds the method.
- enable_interception: returns a receiver for every JS fetch()/XHR request;
resolve each through its resolver with InterceptResolution::{Continue,
Fulfill, Fail} to pass, mock, or block (CDP Fetch.enable equivalent).
- on_request / on_response: passive callbacks. on_response is the main path for
capturing API response payloads from SPAs.
on_request/on_response previously fired only for navigation; op_fetch_url now
invokes them for JS fetch()/XHR too (both reqwest and stealth paths), gated on
a non-empty callback vector so the common no-callback case pays nothing.
Re-export InterceptedRequest, InterceptResolution, RequestInfo, ResourceType,
RequestCallback, ResponseCallback, Response from the obscura crate.
Integration test tests/interception.rs exercises all three against a local echo
server.
The default HTTP path emitted headers with user-agent first and sec-fetch-dest
before sec-fetch-site, an order no real browser produces and a known
fingerprint signal. Reorder the explicit headers to Chrome's top-level
navigation order (sec-ch-ua family, upgrade-insecure-requests, user-agent,
accept, sec-fetch-site/mode/user/dest, accept-language). reqwest still appends
accept-encoding and host after these, so accept-encoding lands after
accept-language instead of before it; the rest matches Chrome exactly.
The non-stealth navigation path hardcoded sec-ch-ua-platform "Linux" and a
fixed sec-ch-ua using Chrome 149's GREASE token, regardless of the selected
profile. With the default Windows Chrome 143 profile that meant a Windows UA
and navigator.userAgentData traveling with a Linux sec-ch-ua-platform header
and mismatched brands, a cross-layer inconsistency a site can check at a
glance. Derive both headers from the profile User-Agent using Chromium's
per-major GREASE algorithm, so the HTTP client hints agree with the JS surfaces.
op_fetch_url backed JS fetch() and XHR with a plain reqwest/rustls client, so
in stealth mode scripted requests carried a non-browser TLS fingerprint and
none of the Chrome client hints the main navigation sends. Bot managers that
score subresource requests (the AWS WAF challenge verify call, Akamai sensors)
read that as a script and reject it. Add StealthHttpClient::send_single and a
stealth_fetch_all path that op_fetch_url uses when a stealth client is
installed, preserving the redirect, SSRF, and CORS handling so JS-level
requests share the navigation's Chrome TLS fingerprint and sec-ch-ua client
hints. The reqwest path is unchanged for non-stealth. Adds a stealth feature to
obscura-js gated on obscura-net/stealth.
Stealth mode forced a Linux navigator UA and platform but reported Windows
in userAgentData.getHighEntropyValues and served Windows Direct3D11 WebGL
renderers, a combination no real Linux Chrome ever produces and that
cross-layer fingerprint checks flag. Present one consistent Windows Chrome
145 identity instead: the wreq emulation uses Platform::Windows, and the
navigator surfaces read the UA and platform from shared constants so the
HTTP User-Agent header, sec-ch-ua-platform, navigator.userAgent, and
userAgentData all agree. userAgentData brands are now derived from the
Chrome major version via Chromium's GREASE algorithm rather than a single
hardcoded token, and a Linux WebGL renderer pool is added so a Linux UA can
never report a Direct3D11 renderer.
wreq upgraded to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the
latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension
order, ALPS, GREASE).
__obscura_stealth global variable introduced and passed from Rust to JS,
allowing stealth-specific JS behaviors to run only when the stealth flag
is enabled.
Tightened hardwareConcurrency and deviceMemory pools in stealth mode
to desktop-realistic values (e.g. min 4GB RAM, 4 cores), eliminating
mobile-tier values that leak the spoof.
navigator.userAgentData and getHighEntropyValues() are now overridden
in stealth mode to perfectly match the sec-ch-ua header sent by
wreq_util::Profile::Chrome145.
Fixed usedJSHeapSize sometimes exceeding totalJSHeapSize.
SameSite is case-insensitive per RFC 6265bis but servers send it in any
casing (lax, LAX, Lax). Normalize to title-case at parse time so
downstream comparisons work regardless. Unknown values fall back to Lax
per spec.
Add is_forbidden_ip() (loopback, RFC1918, link-local incl. 169.254.169.254,
broadcast, documentation, 0.0.0.0/::, IPv6 unique-local fc00::/7, IPv4-mapped
IPv6) and SsrfGuardResolver (impl reqwest::dns::Resolve) that resolves and
rejects forbidden addresses at connect time, closing the DNS-rebinding bypass.
Wired into the navigation client and the JS fetch()/XHR path. The
--allow-private-network / OBSCURA_ALLOW_PRIVATE_NETWORK escape hatch is kept.
Co-authored-by: charlesmamane26-sketch <charlesmamane1826@gmail.com>
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
wreq and wreq-util are pre-release crates whose public API shifts
between rc releases. The caret version requirements let a fresh resolve
pull a newer rc where CertStore moved out of tls, cert_store was renamed
to tls_cert_store, and the Emulation builder changed. That broke
cargo build --features stealth (issue #234).
Pin to =6.0.0-rc.28 and =3.0.0-rc.10, the versions wreq_client.rs is
written against, so the build always resolves to a compatible API.
* perf: cache HTTP clients + selector parsing + exponential wait backoff
Three low-risk wins discovered while reading the hot paths.
1. op_fetch_url cached its reqwest::Client. Previously every JS-side
fetch(), XHR, and dynamic script load built a fresh client, which
re-loaded TLS roots and started with an empty connection pool. On
an asset-heavy page with 30+ subresources this added ~150ms of
wasted setup with zero connection reuse. The cache is a process-
wide map keyed by proxy URL (empty string for no-proxy), so the
first fetch on each proxy pays the build cost once and every
subsequent fetch reuses the same pool.
Also bumped pool_idle_timeout to 5 minutes and tcp_keepalive to
60s so connections stay warm for SPA-heavy workloads that hit
the same origin dozens of times across a navigation.
2. DomTree::parse_selector now caches parsed selectors in a thread-
local HashMap (cap 256). querySelector / querySelectorAll re-
parsed the selector string every call before; batch DOM access
patterns (an agent scraping a table row-by-row, a framework
polling for elements) paid the cssparser cost N times.
3. browser_wait_for / browser_wait_for_text now exponentially back
off (5 -> 10 -> 20 -> ... -> 200 ms). The old fixed 200ms tick
meant a selector that appeared 30ms after the call still took
the next full 200ms tick to be detected. Worst case is still
200ms per check, but the median wait drops sharply for fast
appearances.
28/28 integration tests pass. Asset-heavy bench (rust-lang/rust,
docs.stripe.com, etc) doesn't regress.
* perf: module loader client cache, fetch concurrency cap, snapshot hide list, cookie alloc
Round two of perf wins after a deep code audit.
1. ES module loader (module_loader.rs) was building a fresh
reqwest::Client on every dynamic import. Modern SPAs (Next, Vite)
load 20-50 chunks via dynamic import; each one used to spin up an
empty connection pool with fresh TLS init. Now reuses the
process-wide FETCH_CLIENT_CACHE that op_fetch_url already uses
(the cached_request_client helper). Saves ~30-100ms on chunk-
heavy first navigations.
2. Script and CSS fetches in execute_scripts used futures::join_all
with unbounded concurrency. A 100-script page would open 100 TCP
sockets at once, exhausting the connection pool and the OS
ephemeral port range. Capped at 16 in-flight (well above the per-
host pool ceiling real Chrome uses).
3. bootstrap.js __obscura_init() ran Object.keys(globalThis).filter()
on every page setup to find internals to hide. The set of keys
is static at snapshot-build time, so we now precompute it once
(top-level statement just below the function) and __obscura_init
just iterates the precomputed array. Was 5-40ms per page on heavy
SPAs (1000+ globals); now ~0.2ms per page.
4. CookieJar::domain_matches allocated 2 lowercase Strings and a
format! per call. The lookup runs per fetch per cookie domain;
on a heavily-cookied site with 100 cookies across 30 domains
that was thousands of allocations per page. Now zero-alloc with
eq_ignore_ascii_case and char-boundary checks.
Picks up @Timtech4u's PR #33, which has been waiting since April. The
design (CLI flag + threading through BrowserContext + http client) is
theirs and is correct; this is a clean rewrite on top of current main
because repeated rebase rounds had been reverting unrelated work that
landed in the meantime.
Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent
SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is
process-wide and clunky in pipelines. Multiple users (@Timtech4u in
the original PR, then @duartemvix and @necrogami in the comment
thread) reported they cannot use obscura for development against
http://localhost:N without it.
Adds --allow-private-network as a global CLI flag, works on fetch,
serve, and scrape. Threads through BrowserContext::with_storage_and_network
and the http client's validate_url. CdpContext gets a new_full
builder and obscura_cdp::start_with_full_serve_options entry point so
'obscura serve --allow-private-network' lets Puppeteer/Playwright
navigate to localhost too. Default remains blocked.
Backward compat: the env var still works (OR'd with the flag inside
validate_url). All existing constructors and entry points unchanged.
Verified end-to-end against a local http.server on 127.0.0.1: fetch
and CDP Page.navigate both blocked by default, both succeed with the
flag. Public URLs unaffected. Integration suite 28/28, full workspace
cargo test all green.
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <zhangyang.crazyone@gmail.com>
Co-authored-by: SGavrl <277843151+SGavrl@users.noreply.github.com>
set_default_paths only resolves OpenSSL's compile-time CA dirs, which
exist on Linux but are empty on Windows. The stealth client's cert store
ended up empty there and every TLS handshake failed with
CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc.
CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's
trust list), which behaves the same on every platform and removes the
dependency on a system CA store.
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity
Closes#168.
Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.
Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cdp): fall back to default cookie jar, add Network.getAllCookies
Addresses @SGavrl review on PR #170:
- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
cookie jar via a shared helper that prefers the session page's jar and
falls back to ctx.default_context.cookie_jar when no target has been
attached. Mirrors how Storage.setCookies already routes when there is
no session, so Puppeteer/Playwright flows that issue cookie methods
before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
existing getCookies handler so both arms return the full jar.
Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes#113.
Fixes the remaining two paths from #139: JS-level fetch()/XHR
(op_fetch_url) and dynamic ES module imports (ObscuraModuleLoader) both
built their reqwest clients with `Client::builder().build()` — no
proxy — so every JS-initiated network call silently bypassed
BrowserContext.proxy_url. (The third path, StealthHttpClient, was
already fixed in main: page.rs:61 uses with_proxy.)
Changes:
* obscura-net: expose ObscuraHttpClient::proxy_url() so callers outside
the crate can route their own reqwest requests through the same
upstream proxy. The field was already there — just lacked an accessor.
* obscura-js/module_loader: add `proxy_url: Option<String>` and
`with_proxy(base_url, proxy_url)`. ObscuraModuleLoader::new keeps the
pre-#139 direct-connection behaviour. load() builds a reqwest client
per import, threading the proxy and bubbling up "Invalid module proxy"
on bad URLs.
* obscura-js/ops.rs: replace the process-wide
`static SHARED_HTTP_CLIENT: OnceLock<reqwest::Client>` (which could
never be reconfigured after first call) with a per-request
`build_request_client(proxy_url)` that pulls the configured proxy out
of SharedState.http_client.proxy_url() each call. Per-request build
cost is ~1ms vs the network round-trip, and we no longer have a
global singleton that lies about its config.
* obscura-js/runtime: add ObscuraJsRuntime::with_base_url_and_proxy.
with_base_url delegates to it with None so existing tests / callers
are unaffected.
* obscura-browser/page.rs: thread context.proxy_url through to
with_base_url_and_proxy when constructing the JS runtime.
- obscura-net/cookies.rs: #[allow(dead_code)] on CookieEntry.same_site
(parsed today, kept for future SameSite enforcement).
- obscura-js/runtime.rs: drop vestigial BOOTSTRAP_JS const; bootstrap.js
is already embedded into the V8 snapshot via build.rs.
- obscura-js/ops.rs: prefix the four unused fields in
InterceptResolution::Continue with _.
- obscura-cdp/dispatch.rs: drop unused InterceptResolution import.
- obscura-cdp/domains/fetch.rs: drop unused Arc / Mutex / CdpEvent imports.
- obscura-cdp/domains/input.rs: prefix _button / _click_count; CDP
parses them but the JS click stub is hardcoded for now.
- obscura-cdp/server.rs: prefix _processor_handle and _ctx; drop dead
initial Ok(())/None for navigate_result/page_back - they are
unconditionally overwritten in the only loop break arm.
cargo build --workspace: 0 warnings.
cargo test --workspace: 113 passed.
Allow loading local HTML/CSS/JS files via the file:// protocol,
enabling Obscura to be used for testing static sites without
running a local HTTP server.
Changes:
- Allow "file" scheme in validate_url() and validate_fetch_url()
- Add fetch_file_url() to read local files and return synthetic
HTTP responses with appropriate content-type headers
- Skip private IP / localhost checks for file:// URLs
Closes#16
SGavrl