* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes#113.
- about: URL scheme now handled in navigate_single instead of
hitting validate_url and failing with "Forbidden URL scheme"
- DOM.getBoxModel calls getBoundingClientRect via JS instead of
returning hardcoded pixel coords
- DOM.getContentQuads added (was missing, needed by Playwright click)
- globalThis._wrap exposed so CDP eval code can look up nodes by id
- globalThis.self = globalThis for webpack/bundler worker detection
document.readyState was hardcoded to "complete" so scripts checking
readyState === 'loading' always took the wrong branch. Fixed by using a
JS global (__documentReadyState__) that Rust sets to "loading" before
any scripts execute, "interactive" before DOMContentLoaded fires, and
"complete" after load.
document.addEventListener and document.dispatchEvent on Document were
no-ops, so DOMContentLoaded listeners registered by inline scripts were
never called. Added proper listener storage and dispatch on Document.
Adds document.getElementsByName() (was missing entirely).
Page.reload is now implemented (re-navigates to the current URL),
resolving "Unknown Page method: reload" errors from Playwright.
data: URL navigation now works, fixing Playwright's goto("data:text/html,...")
pattern that previously errored with "invalid format".
Page.navigate logic extracted into do_navigate() so reload can reuse it.
Fixes#137, #146 (SPA bootstrap lifecycle ordering).
The previous workaround in page.rs rewrote socks5:// to http:// before
handing the proxy to the stealth client. That works for Clash-style
mixed-mode proxies, but breaks plain SOCKS5 servers like `ssh -ND`,
producing a confusing `TunnelUnexpectedEof` deep in the network layer.
The stealth client is backed by wreq, which does not speak SOCKS5,
so silently substituting schemes can never be correct in general.
Refuse the combination at CLI startup with a clear message that tells
the user what to do (use --proxy http://... or drop --stealth) and
revert the rewrite in Page::new. SOCKS5 without --stealth still works
through the reqwest path.
GHSA-q55h-vfv9-qcr5. Any process able to reach the CDP port could
send Page.navigate {"url":"file:///etc/shadow"} and read arbitrary
local files (the process's UID's worth of read access). When obscura
runs as root inside Docker or as a systemd service that translates
straight into /etc/shadow, /root/.ssh/id_rsa, /proc/1/environ, etc.
Validate at the CDP boundary, not in obscura-net, so direct CLI
callers ("obscura fetch file://...") keep working. A new
BrowserContext.allow_file_access flag defaults to false; Page.navigate
returns an error when the requested URL is file:// and the flag is
off. CdpContext::new_with_security threads the flag from the server
entry point, and "obscura serve --allow-file-access" opts in for
local-HTML testing workflows.
Tested:
- Default obscura serve: CDP Page.navigate {file:///etc/passwd} ->
refused with descriptive error.
- obscura serve --allow-file-access: same call succeeds.
- obscura fetch file://... CLI path: unchanged, still reads the file.
- All earlier PoCs (GHSA-6pj9 JS->file SOP, GHSA-4789 sub-resource
SSRF, GHSA-4789 CSS template injection) still pass.
- cargo test --workspace clean.
Three findings from GHSA-4789-rq5x-jv3c.
HIGH-1: Sub-resource fetches (<script src>, <link rel=stylesheet>) had
no scheme gate, so an http(s) page could include
<script src="file:///etc/passwd"> and the body would come back as
script source. Added subresource_allowed() that mirrors the top-level
file:// SOP rule: file: sub-resources only load when the page itself
is on file:. Off-origin schemes (data:, javascript:, etc.) are
refused regardless. Private/loopback hosts were already blocked at
the obscura-net layer; this closes the file:// hole on top of that.
HIGH-2: CSS pulled from the network was string-formatted into a JS
template literal with a three-rule escaper (backslash, backtick,
${). That escaper did not handle U+2028 / U+2029 which terminate
ECMAScript template literals, letting attacker CSS break out and
run arbitrary code in the page's V8 realm. Replaced the inline
escape with escape_for_js_template_literal() which also handles
U+2028, U+2029, null, carriage return, and every other C0 control.
LOW-2: navigate_with_wait_post() capped JS-driven navigation chains
at ten but returned Ok(()) when the cap was hit, so callers could
not tell a successful load from a redirect storm. Now returns
PageError::TooManyRedirects(REDIRECT_LIMIT).
Verified with three PoCs (sub-resource SSRF, CSS template literal
break with U+2028, and the original GHSA-6pj9-j389-226p reproducer)
plus negative tests (file:// direct read, http -> http JS chain) all
pass. cargo test --workspace clean.
A page on http(s) could set window.onload, assign location.href to a
file:// URL, and then read document.body from the local file because
two things lined up: obscura-net allows file: URLs, and the JS
runtime was reused across navigations so window state from the prior
origin survived into the new document's load event.
Two fixes:
1. Refuse a JS-initiated chain step when it crosses from a non-file
scheme into file:. Direct file:// navigations from the CLI/CDP
still work because the gate only fires on the take_pending_nav
path. file -> file chains also still work.
2. Drop and rebuild the V8 runtime on every navigation instead of
only rebinding globalThis.document. Even within the same origin
we should not carry over window.onload, custom properties, or
registered event listeners from the previous document.
Tested with the PoC from the report: pages on http://127.0.0.1.nip.io
no longer leak the contents of file:///tmp/.../pwn.txt.
Fixes the remaining two paths from #139: JS-level fetch()/XHR
(op_fetch_url) and dynamic ES module imports (ObscuraModuleLoader) both
built their reqwest clients with `Client::builder().build()` — no
proxy — so every JS-initiated network call silently bypassed
BrowserContext.proxy_url. (The third path, StealthHttpClient, was
already fixed in main: page.rs:61 uses with_proxy.)
Changes:
* obscura-net: expose ObscuraHttpClient::proxy_url() so callers outside
the crate can route their own reqwest requests through the same
upstream proxy. The field was already there — just lacked an accessor.
* obscura-js/module_loader: add `proxy_url: Option<String>` and
`with_proxy(base_url, proxy_url)`. ObscuraModuleLoader::new keeps the
pre-#139 direct-connection behaviour. load() builds a reqwest client
per import, threading the proxy and bubbling up "Invalid module proxy"
on bad URLs.
* obscura-js/ops.rs: replace the process-wide
`static SHARED_HTTP_CLIENT: OnceLock<reqwest::Client>` (which could
never be reconfigured after first call) with a per-request
`build_request_client(proxy_url)` that pulls the configured proxy out
of SharedState.http_client.proxy_url() each call. Per-request build
cost is ~1ms vs the network round-trip, and we no longer have a
global singleton that lies about its config.
* obscura-js/runtime: add ObscuraJsRuntime::with_base_url_and_proxy.
with_base_url delegates to it with None so existing tests / callers
are unaffected.
* obscura-browser/page.rs: thread context.proxy_url through to
with_base_url_and_proxy when constructing the JS runtime.
Surfaces the existing LP.getMarkdown CDP machinery as a CLI flag so
text-pipeline users (LLM agents, scrapers feeding markdown to other
tools) don't need a separate HTML-to-markdown converter step.
obscura fetch --dump markdown 'https://example.com'
Layout:
- crates/obscura-js/src/markdown.rs holds HTML_TO_MARKDOWN_JS as a
pub const so every layer can share the same DOM-walker. obscura-js
is at the bottom of the dep DAG (obscura-browser -> obscura-js,
obscura-cdp -> obscura-js, obscura-cli -> obscura-js) so a single
source of truth works without a circular dependency.
- crates/obscura-browser/src/lib.rs re-exports it for ergonomics.
- crates/obscura-cdp/src/domains/lp.rs drops its inline duplicate
and calls the shared const.
- crates/obscura-cli/src/main.rs adds the `Markdown` DumpFormat
variant + dump_markdown(&mut Page) which page.evaluate's the
shared script and prints the result.
Net diff: a new 75-line markdown.rs, a 70+ line deletion in lp.rs
that's reborn as 3 lines, and ~10 lines added across the other four
files. No behavior change for `--dump html|text|links`, and
LP.getMarkdown over CDP keeps the exact same JSON shape.
Four new unit tests in obscura-js cover headings, inline emphasis +
links, ordered/unordered lists, and that <script> / <style> contents
do not leak into the output.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
When --stealth is enabled, StealthHttpClient was created without
the proxy URL from --proxy, causing all stealth-mode requests to
bypass the proxy.
Also convert SOCKS5 proxy URL to HTTP when passing to wreq-based
StealthHttpClient, because wreq does not support SOCKS5 protocol
(Clash mixed port supports both protocols on the same port).
Fixes proxy connectivity for 'obscura serve --stealth --proxy socks5://...'
Co-authored-by: handsome.cheng <[email protected]>
obscura serve --user-agent <ua> only logged the value via tracing::info!
and never passed it to the CDP server, so every navigation kept using
the default Chrome UA string and httpbin.org/user-agent reported the
default rather than the override.
Plumb the option end-to-end:
- BrowserContext::with_full_options(id, proxy, stealth, user_agent)
initializes both BrowserContext::user_agent and the underlying
ObscuraHttpClient::user_agent RwLock at construction time so the
first navigation already sees the override.
- CdpContext::new_with_full_options forwards the UA to the default
BrowserContext.
- obscura_cdp::start_with_full_options accepts the UA and passes it
through cdp_processor.
- main.rs Serve arm calls start_with_full_options with the parsed UA.
- run_multi_worker_serve forwards --user-agent <ua> when spawning
worker subprocesses so multi-worker mode honors the flag too.
Existing two-arg start_with_options and three-arg with_options stay
as thin shims (call the full variant with None) so older callers and
the public crate API remain source-compatible.
Three new unit tests in obscura-browser cover the propagation, the
Chrome default fallback, and the legacy with_options path.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
Brings Obscura's CDP server in line with what Playwright's
`chromium.connectOverCDP()` expects. Each fix is small and surgical;
together they make `connect → newContext → newPage → goto → page.title()`
work end-to-end against `https://example.com`.
1. **server.rs — emit pending events before command response.**
Chromium CDP semantics require side-effect events (e.g.
`Target.targetCreated` + `Target.attachedToTarget` from
`Target.createTarget`) to arrive *before* the originating command's
response. Playwright's wire reader assumes this ordering — it
resolves the response promise on the response and then synchronously
looks up the freshly-created session. Reversing the order makes that
lookup miss and the page hang at `newPage()`.
2. **domains/browser.rs — accept Browser.setWindowBounds as a no-op.**
Playwright calls `Browser.setWindowBounds` during teardown. Returning
"Unknown Browser method" causes Playwright to treat the target as
broken and tear it down mid-test, surfacing as "Frame has been
detached" on the next `page.title()`.
3. **obscura-browser/page.rs — frame_id == target_id for main frame.**
Was `format!("{id}.1", id)`. Playwright's frame manager indexes
frames by frame id and looks up the main frame by target id, so the
`.1` suffix made every main-frame lookup miss.
4. **domains/page.rs + dispatch.rs — track isolated worlds and re-emit
them post-navigation.** Playwright registers a utility world via
`Page.createIsolatedWorld` and expects `Runtime.executionContextCreated`
for each registered world after every navigation. Without re-emission,
the utility-world handle goes stale post-`goto` and `page.title()`
hangs waiting for the world's execution context. Falls back to the
legacy `__puppeteer_utility_world__24.40.0` name when no worlds have
been registered, preserving prior Puppeteer support.
5. **obscura-js/runtime.rs — strip trailing semicolons before wrapping.**
Playwright's utility-script expression is an IIFE ending in `})();`.
Both `wrap_expression` (CDP `Runtime.evaluate`) and `evaluate_for_cdp`
(the `meta_code` IIFE) embed the user expression inside `({EXPR})` /
`return ({EXPR});`. With the trailing `;` left in, this produces
`({...;})` — a SyntaxError. Parse errors are not catchable, so the
surrounding try/catch never fires and the call silently returns
`undefined`, surfacing as `page.title() === ""`.
Verified end-to-end against `https://example.com`:
[probe] connect OK
[probe] newContext OK
[probe] newPage OK
[probe] goto OK
[probe] title = Example Domain
