* fix(net): default cookie Path to the request directory (RFC 6265 5.1.4)
When a Set-Cookie carried no Path attribute, the jar stored the full request
path as the cookie's path. RFC 6265 5.1.4 default-path is the request URI's
directory — the path up to but not including the right-most '/'. Storing the
full path scoped a session cookie to the exact URL that set it: a cookie set on
GET /app/login was stored with path=/app/login and then, because path-match
(5.1.4, fixed earlier) requires a '/' boundary, was not sent to /app/dashboard.
The user appeared logged out on the very next navigation.
Add default_cookie_path() implementing the RFC algorithm and use it as the
initial path in both set_cookie (HTTP) and set_cookie_from_js (document.cookie).
An explicit Path attribute still overrides it. Tests cover the algorithm and the
end-to-end 'cookie set on /app/login is sent to /app/dashboard' path for both
the HTTP and JS entry points; both failed before this change.
* fix(cdp): apply RFC 6265 default cookie Path to CDP-injected cookies
Network.setCookie/setCookies (the Puppeteer page.setCookie and
Playwright addCookies surface) derived the cookie Path from the full
request URL when the caller omitted path, the same bug the parent
commit fixed for Set-Cookie responses. Reuse default_cookie_path (now
pub) so a cookie set under /v1/things scopes to /v1 and is resent on
sibling paths. Fixes the CDP half of #401.
Correct the test that locked in the old full-path behavior and add a
case confirming an explicit Path still wins.
---------
Co-authored-by: SGavrl <[email protected]>
Cookie path filtering used `request_path.starts_with(cookie_path)`, so a
Path=/admin cookie was sent to sibling paths that merely share the string
prefix, such as /administrator or /admin-backup. Per RFC 6265 5.1.4 a prefix
match also requires the boundary to fall on a '/'.
Add a path_matches helper (identical path, or a prefix ending in '/', or a
prefix followed by '/') and use it in get_cookie_header and
get_js_visible_cookies.
Fixes#382.
CookieJar::set_cookie and set_cookie_from_js started from the request host but
blindly overwrote it with any Domain attribute, so a response from
attacker.test could plant a cookie scoped to victim.test, which was then sent
on later requests to victim.test. The same sink backs document.cookie, HTTP
Set-Cookie, and CookieStore::set.
Add resolve_cookie_domain: with no Domain the cookie is host-only (exact host
only); a Domain attribute is honored only when it domain-matches the origin
(equal or a parent domain) and is not an obvious public suffix, otherwise the
attribute is ignored and the cookie is stored host-only on the origin. Track
host_only on CookieEntry so host-only cookies are not sent to subdomains.
A full public suffix list is not bundled, so multi-label public suffixes
(co.uk, github.io) are not rejected; the domain-match check still blocks the
reported cross-domain attack, and single-label suffixes (com, local) are.
Tests cover the cross-domain rejection for both HTTP and document.cookie,
public-suffix rejection, host-only subdomain scoping, and that legitimate
parent-domain cookies still work.
Co-authored-by: DavidCarliez <[email protected]>
SameSite is case-insensitive per RFC 6265bis but servers send it in any
casing (lax, LAX, Lax). Normalize to title-case at parse time so
downstream comparisons work regardless. Unknown values fall back to Lax
per spec.
* perf: cache HTTP clients + selector parsing + exponential wait backoff
Three low-risk wins discovered while reading the hot paths.
1. op_fetch_url cached its reqwest::Client. Previously every JS-side
fetch(), XHR, and dynamic script load built a fresh client, which
re-loaded TLS roots and started with an empty connection pool. On
an asset-heavy page with 30+ subresources this added ~150ms of
wasted setup with zero connection reuse. The cache is a process-
wide map keyed by proxy URL (empty string for no-proxy), so the
first fetch on each proxy pays the build cost once and every
subsequent fetch reuses the same pool.
Also bumped pool_idle_timeout to 5 minutes and tcp_keepalive to
60s so connections stay warm for SPA-heavy workloads that hit
the same origin dozens of times across a navigation.
2. DomTree::parse_selector now caches parsed selectors in a thread-
local HashMap (cap 256). querySelector / querySelectorAll re-
parsed the selector string every call before; batch DOM access
patterns (an agent scraping a table row-by-row, a framework
polling for elements) paid the cssparser cost N times.
3. browser_wait_for / browser_wait_for_text now exponentially back
off (5 -> 10 -> 20 -> ... -> 200 ms). The old fixed 200ms tick
meant a selector that appeared 30ms after the call still took
the next full 200ms tick to be detected. Worst case is still
200ms per check, but the median wait drops sharply for fast
appearances.
28/28 integration tests pass. Asset-heavy bench (rust-lang/rust,
docs.stripe.com, etc) doesn't regress.
* perf: module loader client cache, fetch concurrency cap, snapshot hide list, cookie alloc
Round two of perf wins after a deep code audit.
1. ES module loader (module_loader.rs) was building a fresh
reqwest::Client on every dynamic import. Modern SPAs (Next, Vite)
load 20-50 chunks via dynamic import; each one used to spin up an
empty connection pool with fresh TLS init. Now reuses the
process-wide FETCH_CLIENT_CACHE that op_fetch_url already uses
(the cached_request_client helper). Saves ~30-100ms on chunk-
heavy first navigations.
2. Script and CSS fetches in execute_scripts used futures::join_all
with unbounded concurrency. A 100-script page would open 100 TCP
sockets at once, exhausting the connection pool and the OS
ephemeral port range. Capped at 16 in-flight (well above the per-
host pool ceiling real Chrome uses).
3. bootstrap.js __obscura_init() ran Object.keys(globalThis).filter()
on every page setup to find internals to hide. The set of keys
is static at snapshot-build time, so we now precompute it once
(top-level statement just below the function) and __obscura_init
just iterates the precomputed array. Was 5-40ms per page on heavy
SPAs (1000+ globals); now ~0.2ms per page.
4. CookieJar::domain_matches allocated 2 lowercase Strings and a
format! per call. The lookup runs per fetch per cookie domain;
on a heavily-cookied site with 100 cookies across 30 domains
that was thousands of allocations per page. Now zero-alloc with
eq_ignore_ascii_case and char-boundary checks.
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity
Closes#168.
Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.
Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(cdp): fall back to default cookie jar, add Network.getAllCookies
Addresses @SGavrl review on PR #170:
- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
cookie jar via a shared helper that prefers the session page's jar and
falls back to ctx.default_context.cookie_jar when no target has been
attached. Mirrors how Storage.setCookies already routes when there is
no session, so Puppeteer/Playwright flows that issue cookie methods
before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
existing getCookies handler so both arms return the full jar.
Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).
- obscura-net/cookies.rs: #[allow(dead_code)] on CookieEntry.same_site
(parsed today, kept for future SameSite enforcement).
- obscura-js/runtime.rs: drop vestigial BOOTSTRAP_JS const; bootstrap.js
is already embedded into the V8 snapshot via build.rs.
- obscura-js/ops.rs: prefix the four unused fields in
InterceptResolution::Continue with _.
- obscura-cdp/dispatch.rs: drop unused InterceptResolution import.
- obscura-cdp/domains/fetch.rs: drop unused Arc / Mutex / CdpEvent imports.
- obscura-cdp/domains/input.rs: prefix _button / _click_count; CDP
parses them but the JS click stub is hardcoded for now.
- obscura-cdp/server.rs: prefix _processor_handle and _ctx; drop dead
initial Ok(())/None for navigate_result/page_back - they are
unconditionally overwritten in the only loop break arm.
cargo build --workspace: 0 warnings.
cargo test --workspace: 113 passed.
