* perf: cache HTTP clients + selector parsing + exponential wait backoff
Three low-risk wins discovered while reading the hot paths.
1. op_fetch_url cached its reqwest::Client. Previously every JS-side
fetch(), XHR, and dynamic script load built a fresh client, which
re-loaded TLS roots and started with an empty connection pool. On
an asset-heavy page with 30+ subresources this added ~150ms of
wasted setup with zero connection reuse. The cache is a process-
wide map keyed by proxy URL (empty string for no-proxy), so the
first fetch on each proxy pays the build cost once and every
subsequent fetch reuses the same pool.
Also bumped pool_idle_timeout to 5 minutes and tcp_keepalive to
60s so connections stay warm for SPA-heavy workloads that hit
the same origin dozens of times across a navigation.
2. DomTree::parse_selector now caches parsed selectors in a thread-
local HashMap (cap 256). querySelector / querySelectorAll re-
parsed the selector string every call before; batch DOM access
patterns (an agent scraping a table row-by-row, a framework
polling for elements) paid the cssparser cost N times.
3. browser_wait_for / browser_wait_for_text now exponentially back
off (5 -> 10 -> 20 -> ... -> 200 ms). The old fixed 200ms tick
meant a selector that appeared 30ms after the call still took
the next full 200ms tick to be detected. Worst case is still
200ms per check, but the median wait drops sharply for fast
appearances.
28/28 integration tests pass. Asset-heavy bench (rust-lang/rust,
docs.stripe.com, etc) doesn't regress.
* perf: module loader client cache, fetch concurrency cap, snapshot hide list, cookie alloc
Round two of perf wins after a deep code audit.
1. ES module loader (module_loader.rs) was building a fresh
reqwest::Client on every dynamic import. Modern SPAs (Next, Vite)
load 20-50 chunks via dynamic import; each one used to spin up an
empty connection pool with fresh TLS init. Now reuses the
process-wide FETCH_CLIENT_CACHE that op_fetch_url already uses
(the cached_request_client helper). Saves ~30-100ms on chunk-
heavy first navigations.
2. Script and CSS fetches in execute_scripts used futures::join_all
with unbounded concurrency. A 100-script page would open 100 TCP
sockets at once, exhausting the connection pool and the OS
ephemeral port range. Capped at 16 in-flight (well above the per-
host pool ceiling real Chrome uses).
3. bootstrap.js __obscura_init() ran Object.keys(globalThis).filter()
on every page setup to find internals to hide. The set of keys
is static at snapshot-build time, so we now precompute it once
(top-level statement just below the function) and __obscura_init
just iterates the precomputed array. Was 5-40ms per page on heavy
SPAs (1000+ globals); now ~0.2ms per page.
4. CookieJar::domain_matches allocated 2 lowercase Strings and a
format! per call. The lookup runs per fetch per cookie domain;
on a heavily-cookied site with 100 cookies across 30 domains
that was thousands of allocations per page. Now zero-alloc with
eq_ignore_ascii_case and char-boundary checks.
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity
Closes#168.
Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.
Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(cdp): fall back to default cookie jar, add Network.getAllCookies
Addresses @SGavrl review on PR #170:
- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
cookie jar via a shared helper that prefers the session page's jar and
falls back to ctx.default_context.cookie_jar when no target has been
attached. Mirrors how Storage.setCookies already routes when there is
no session, so Puppeteer/Playwright flows that issue cookie methods
before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
existing getCookies handler so both arms return the full jar.
Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).
- obscura-net/cookies.rs: #[allow(dead_code)] on CookieEntry.same_site
(parsed today, kept for future SameSite enforcement).
- obscura-js/runtime.rs: drop vestigial BOOTSTRAP_JS const; bootstrap.js
is already embedded into the V8 snapshot via build.rs.
- obscura-js/ops.rs: prefix the four unused fields in
InterceptResolution::Continue with _.
- obscura-cdp/dispatch.rs: drop unused InterceptResolution import.
- obscura-cdp/domains/fetch.rs: drop unused Arc / Mutex / CdpEvent imports.
- obscura-cdp/domains/input.rs: prefix _button / _click_count; CDP
parses them but the JS click stub is hardcoded for now.
- obscura-cdp/server.rs: prefix _processor_handle and _ctx; drop dead
initial Ok(())/None for navigate_result/page_back - they are
unconditionally overwritten in the only loop break arm.
cargo build --workspace: 0 warnings.
cargo test --workspace: 113 passed.
SGavrl