Commit Graph
555 Commits
Author SHA1 Message Date
h4ckf0r0day c9f32aa2f0 v0.1.6 2026-05-29 20:22:19 +02:00
ea89f050cd Fix puppeteer exposeFunction end-to-end: real bindings, preload timing, URL coercion (#166)
* fix(js): coerce URL constructor input/base to string

The polyfill at bootstrap.js called `base.match(...)` directly, which
threw `TypeError: base.match is not a function` whenever callers passed
a Location object (e.g. `new URL(href, window.location)`), a URL
instance, or any other non-string with a sensible toString. Per the
WHATWG URL Standard both `url` and `base` are stringified before
parsing.

Coerce explicitly so the existing regex paths keep working and the
constructor matches real-browser behaviour. AngularJS, jQuery, and
anything that resolves a relative URL against `window.location` now
loads under Obscura.

* fix(browser,cdp): run preload scripts before page scripts

`Page.addScriptToEvaluateOnNewDocument` was executed AFTER the page's
own scripts (and even after the load event), which silently broke
every CDP contract that depends on early injection — most visibly
puppeteer's `page.exposeFunction`, whose user-facing wrapper has to
override `globalThis[name]` before any `<script>` tag in the document
runs.

Move the registered preload sources onto `Page` and execute them
inside `navigate_single`, right after `__documentReadyState__ =
'loading'` and before the parser-discovered scripts. The CDP layer
hands the current preload set to the page via `set_preload_scripts`
before kicking off navigation, replacing the two post-navigate
`execute_preload_script` loops in `domains/page.rs` and `server.rs`.

* feat(cdp,js): emit Runtime.bindingCalled for addBinding shims

`Runtime.addBinding` was a no-op stub that installed an empty
`globalThis[name] = function () { return null; }` placeholder. Page JS
calls were silently swallowed and the Node-side callback registered
via puppeteer's `page.exposeFunction` never fired.

Wire the binding all the way through:

1. New `op_binding_called(name, payload)` op forwards every page-side
   call into a `pending_binding_calls` queue on `ObscuraState`.
2. `Page::take_pending_binding_calls` drains the queue.
3. `dispatch()` empties every page's queue after each CDP request and
   pushes one `Runtime.bindingCalled` event per entry into
   `ctx.pending_events`, so the writer task forwards them to the
   client like any other CDP event.
4. The `addBinding` handler installs a real shim that calls the op,
   registers it as a preload script (keyed `__obscura_binding__<name>`)
   so it survives navigations, and evaluates it now so the binding is
   usable on the current document.
5. Implement `removeBinding` to mirror `addBinding`.

The shim coerces non-string arguments via `JSON.stringify`, matching
the wire contract that puppeteer's `exposeFunction` wrapper depends
on (`Runtime.bindingCalled.params.payload` is always a string).

* fix(cdp): emit canAccessOpener in all TargetInfo payloads, add aarch64 cross-compile CI

Merges PR #165 and PR #172.

PR #165: The CDP spec marks canAccessOpener as a required boolean on
TargetInfo but Obscura was omitting it. Strict Rust clients like
chromiumoxide reject the event entirely and panic with "Created target
not present" when the field is absent. Added canAccessOpener: false
to every TargetInfo emitted by setDiscoverTargets, getTargets,
createTarget, attachToBrowserTarget, attachToTarget, and getTargetInfo.

PR #172: Added a cross-compile matrix entry for aarch64-unknown-linux-gnu
running on ubuntu-22.04 x86_64, producing obscura-aarch64-linux-cross.
This is a fallback for when GitHub ARM runners are unavailable without
removing the native ARM build.

* fix(cdp): keep HTTP control plane reachable while V8 JS evaluation blocks (#173)

fix(cdp): unblock HTTP control plane during synchronous V8 execution

Fixes #62.

Previously, the TCP accept loop and CDP processor shared a single-threaded
tokio LocalSet. Long synchronous JS evaluations starved the accept loop,
causing incoming connections to hang in the kernel backlog and effectively
killing the HTTP control plane.

This moves the TCP listener and `/json/*` endpoints to a dedicated OS
thread using `std::net::TcpListener`. HTTP requests are now handled
directly via blocking I/O on this thread, completely bypassing V8.

WebSocket connections are handed off to the LocalSet via a bounded mpsc
channel (capacity 128). If V8 stalls and the queue fills, new WS
connections are dropped (ECONNRESET) rather than buffering unboundedly
to prevent OOMs.

Includes a new integration test to verify the control plane stays
responsive during a 5s JS loop, plus inline documentation clarifying
thread lifetime semantics and a minor test TOCTOU race.

* fix(js,dom): add Storage global, CharacterData ChildNode methods, fix remove/isConnected (#176)

fix(dom): add missing DOM/JS shims for modern frameworks

Closes #163.

Adds missing DOM and JS shims required by modern frameworks (Svelte 5, 
Vue, Lit) and consent-management SDKs to execute without errors:

- Define the `Storage` global constructor to resolve `ReferenceError`s 
  and fix `Storage.prototype` access.
- Apply standard ChildNode methods (`before`, `after`, `replaceWith`, 
  `remove`) to `CharacterData.prototype`. This prevents `TypeError`s when 
  frameworks call these on Text or Comment nodes during hydration.
- Update `Element.prototype.isConnected` to perform a genuine parent-walk 
  rather than returning a hardcoded `true`.
- Fix `Element.prototype.remove` to properly clean up the internal ID 
  index so removed elements are no longer queryable.
- Remove outdated no-op stubs on the `Element` class that were shadowing 
  the proper polyfill implementations.

* feat(cli): add --dump assets to inventory page sub-resources (124) (#174)

feat: add `--dump assets` to enumerate page sub-resources

Adds a `--dump assets` flag to output all page-referenced sub-resources 
(scripts, stylesheets, images, iframes, media, embeds, objects) as NDJSON, 
allowing users to easily replay the asset graph.

- Output format: `{"url":"https://example.test/app.js","type":"script"}`
- URLs are resolved against the page origin (matching `--dump links`).
- The `type` tag is derived from the source element. `<link>` elements map 
  their `rel` attribute to specific types (stylesheet, icon, manifest, preload, 
  prefetch, modulepreload, dns-prefetch, preconnect, alternate), falling back 
  to "link" if missing or unknown.
- Adds 7 tests covering URL resolution, rel-mapping, supported selectors, 
  and empty-attribute skipping.
- Updates README dump-format table and examples.

* fix(cdp,js): playwright actionability stubs and objectId resolve (#178)

Playwright fails locator actions on Obscura for two avoidable reasons
that are separate from the missing layout engine. Both have trivial
fixes that unblock the actionability polling loop.

- Element.checkVisibility(opts) was undefined. Playwright >= 1.25 calls
  it before every input event and retries until its action timeout if
  the method isn't a function. Add a stub that returns true. Without a
  layout pass we can't compute it for real, but a permissive stub still
  beats the undefined-function path.

- ARIA reflection properties were missing on Element (role, ariaLabel,
  ariaSelected, ariaExpanded, ariaHidden, ariaChecked, ariaDisabled,
  ariaRoleDescription). Playwright's getByRole and getByLabel locators
  read them. Reflect through to the underlying aria-* attributes for
  both read and write.

- DOM.getBoxModel and DOM.getContentQuads ignored the objectId param.
  Playwright passes only objectId (from a prior DOM.resolveNode), and
  the handlers silently defaulted to node 0, so dispatchMouseEvent
  always clicked the wrong element. Extract a resolve_node_id helper
  that honors nodeId, backendNodeId, and objectId in that order.

- Page.captureScreenshot and Page.captureSnapshot now return a clear
  "not supported, use a real Chromium for the screenshot leg" error
  instead of the generic "Unknown Page method" fallback. Matches the
  pattern already in place for Page.printToPDF.

Partial fix for #45. The architectural blocker (no layout engine, so
multi-element click coordinates are wrong) is unchanged.

* fix(mcp): allow browser MCP clients through CORS preflight (#179)

The OPTIONS preflight hardcoded Access-Control-Allow-Headers to just
Content-Type. Any browser MCP client sending mcp-protocol-version (per
the MCP spec) or Authorization / X-API-Key (typical for hosted
deployments) failed the preflight and the browser blocked the actual
request.

Add those three to the allow-list and a 24h Max-Age so browsers don't
re-preflight every call. Adds an integration test that drives the
server with a real preflight request.

Closes #175.

* fix(js,dom): scope element queries, real timers, form/prepend/equal (#180)

- Element.querySelector / querySelectorAll scope to the receiver subtree
  via a new query_selector_scoped Rust op, not always document root.
  Same for DocumentFragment.
- setTimeout respects its delay argument via a new async op_sleep.
  setInterval is a real repeating timer instead of an alias to setTimeout.
- document.forms / images / links return live querySelectorAll results
  instead of hardcoded [].
- HTMLFormElement subclasses Element with .elements / .length, and form
  nodes are wrapped as HTMLFormElement so instanceof works. submit()
  stays on Element.prototype to preserve the existing submit-event
  dispatch and form-data navigation path.
- Element.prepend has a real implementation.
- Node.isEqualNode does a structural compare (nodeType, nodeName,
  nodeValue, attributes, recursive children) instead of nid identity.
- Fix long-standing insert_before arg-order bug: bootstrap.js was
  passing (parent, new, ref) but _dom only forwards two args, so ref
  was silently dropped. Pass (new, ref) directly. Same fix in
  replaceChild.

Adds 3 selector unit tests and 6 runtime regression tests covering
the JS shim fixes.

Closes #105.

* fix(cdp): unblock puppeteer connect, bound Runtime.evaluate, full TargetInfo (#181)

Three independent gaps that hurt browser-based MCP clients connecting
to obscura's CDP endpoint:

- Runtime.enable returned "No page" when called before a session was
  attached. puppeteer-extra's FrameManager.initialize fires it on the
  browser-level WebSocket BEFORE any page target exists, so the
  standard puppeteer.connect / browser.newPage flow died on connect.
  Now reply with {} when no session is attached and let the next
  Target.attachToTarget emit executionContextCreated.

- Runtime.evaluate had no per-call timeout. A long synchronous JS
  expression held the global V8 lock indefinitely and starved every
  other CDP command on the session. Honor the CDP `timeout` parameter
  and default to 30s (Chrome's protocolTimeout). Returns a clear error
  on expiry so client-side deadlines fire cleanly instead of hanging.

- Target.getTargetInfo for the browser target omitted canAccessOpener.
  Strict CDP clients (chromiumoxide) panic when the field is missing.
  Fill it in (false for the browser target itself).

Partial fix for #122. The remaining items (networkIdle hard-deadline
on SPAs, /json/list hardcoded data, docs on cheap-vs-queued methods)
need bigger or non-code changes and are deferred.

* fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)

obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.

Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:

  1. HTTP Content-Type charset=... parameter (authoritative when present)
  2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
     first 1KB of the body (HTML only)
  3. UTF-8 fallback

Use it in:
  - obscura-net Response::text (signature changed: String, not Result)
  - obscura-browser page.rs HTML body decode (the primary scrape path)
  - obscura-browser page.rs JS body decode
  - obscura-browser page.rs CSS body decode

For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.

Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.

Closes #113.

* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity (#170)

* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity

Closes #168.

Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.

Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(cdp): fall back to default cookie jar, add Network.getAllCookies

Addresses @SGavrl review on PR #170:

- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
  cookie jar via a shared helper that prefers the session page's jar and
  falls back to ctx.default_context.cookie_jar when no target has been
  attached. Mirrors how Storage.setCookies already routes when there is
  no session, so Puppeteer/Playwright flows that issue cookie methods
  before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
  existing getCookies handler so both arms return the full jar.

Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).

* fix(net): use bundled webpki roots in stealth client (#184) (#185)

set_default_paths only resolves OpenSSL's compile-time CA dirs, which
  exist on Linux but are empty on Windows. The stealth client's cert store
  ended up empty there and every TLS handshake failed with
  CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc.

  CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's
  trust list), which behaves the same on every platform and removes the
  dependency on a system CA store.

* fix(js): Element.attributes, on* event defaults, per-nid getBoundingClientRect (#45) (#186)

Three DOM-completeness gaps that broke jQuery 1.11.3 init and Playwright
  actionability:

  1. Element.attributes was undefined, so legacy IE feature-detect code
     like div.attributes[eventName].expando threw TypeError. Added a
     NamedNodeMap-like backed by a new attribute_names DOM op (length,
     numeric index, string name lookup, getNamedItem/setNamedItem,
     iterable).

  2. Standard on* event handler properties were missing from window, so
     ('on' + ev) in window returned false and forced jQuery into the
     legacy IE branch. Initialised the 79 HTML-spec on* handlers to null.

  3. getBoundingClientRect returned the same {x:8,y:8,width:100,height:20}
     literal for every element, so Playwright's hit-testing couldn't tell
     elements apart. Replaced with a deterministic synthesis from _nid
     mapped into a 12-column grid sized to a 1280x720 viewport. Still not
     pixel-accurate (no layout engine), but stable across reads and
     distinct per node.

* feat: cookie jar persistence + --storage-dir flag + fix HTTP cookie header filtering (#114)

* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)

* fix: resolve build errors, add storage_dir param throughout server chain

* fix: storage_dir end-to-end, plus two test-literal field additions

  Five issues found while testing PR 114 end-to-end.

  Build fixes (two test literals out of sync with new struct fields):
  - crates/obscura-net/src/cookies.rs: CookieInfo literal in
    test_domain_matches_subdomain_without_leading_dot was missing the
    same_site and expires fields added in #170.
  - crates/obscura-cli/src/main.rs: Command::Fetch literal in
    matcher_still_uses_fetch_variant was missing the new storage_dir
    field. Without these cargo test --workspace would not compile.

  run_fetch was branching between BrowserContext::with_storage (which
  hardcodes proxy=None, stealth=false, user_agent=None) and
  with_options (which has no storage_dir slot), so --storage-dir +
  --proxy + --stealth silently dropped the proxy and stealth flags.
  Same bug SGavrl flagged on dispatch.rs in the previous round, just on
  the other code path. Switched to with_storage_full which carries all
  four. Verified the fetch now errors out against an invalid proxy.

  run_fetch --eval short-circuited the function before
  context.save_cookies() ran, so cookies set during an eval session
  were never persisted to {storage_dir}/cookies.json. Moved the save
  above the return.

  obscura serve loaded cookies on startup but never saved them
  anywhere, so a session that set cookies via Network.setCookie or
  navigation dropped them on shutdown. Subscribed to Ctrl-C once in
  put a single save_cookies() after the loop so both the Ctrl-C path
  and the channel-closed path persist cookies. Verified end-to-end.

---------

Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>

* fix(cdp): page.goto returns Response over Puppeteer and Playwright (#189) (#190)

Two correlation invariants real Chrome holds were missing, so the
  nav Response was never attached to goto().

  requestId === loaderId for the main Document event. Both clients
  gate isNavigationRequest() on this; we were emitting a synthetic
  per-page id.

  Network.requestWillBeSent for the Document fires before
  Page.frameNavigated. Playwright's FrameManager needs the request
  present as pendingDocument before the frame commits, otherwise
  currentDocument lands with request=undefined.

  Subresource events and the Document's responseReceived /
  loadingFinished stay where they were.

  Verified end-to-end on both clients against example.com (200),
  google.com (200), httpbin/404 (404).

* fix(browser,cdp): run preloads on about:blank, tighten binding shim arity

Two follow-ups from PR #166 review:

- navigate_single's about: branch returned before any preload sources
  ran, so addScriptToEvaluateOnNewDocument scripts and the
  Runtime.addBinding shim never installed on about:blank. Puppeteer
  hits this every session: browser.newPage() lands on about:blank and
  exposeFunction is dead until the first real navigation. Now the
  about: branch runs the preload set after init_js(), guarded by the
  same 5s timeout as regular preloads.

- The Runtime.addBinding shim used JSON.stringify on non-string args
  and silently accepted any arity. Chromium's V8InspectorImpl rejects
  calls without exactly one argument and ToString-coerces that arg
  before emitting Runtime.bindingCalled. Match that: guard on
  arguments.length and use String(arg) for the non-string branch.
  Single-string calls (the path puppeteer's exposeFunction wrapper
  uses) round-trip unchanged.

* fix(cdp): allocate fresh execution context ids per isolated world (#192) (#200)

Page.createIsolatedWorld and do_navigate both hardcoded context id 100
  for every isolated world emission. After the second navigation Playwright
  expected a fresh incrementing id (101, 102) while obscura kept re-using
  100. Runtime.evaluate then failed with 'Cannot find context with
  specified id: 101' because the id was never registered in
  valid_context_ids and the client's bookkeeping had diverged from the
  server's.

  Added next_isolated_context_id counter on CdpContext (starts at 100) and
  a next_isolated_context() helper that claims an id, registers it in
  valid_context_ids, and increments. Both call sites now use it.

* feat(cli): default V8 heap to 4 GB on 64-bit so heavy SPAs don't OOM (#199) (#202)

V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with
  heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/
  web-scraping) blow past it and the worker dies with
  'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session
  with it.

  Always set --max-old-space-size=4096 (1024 on 32-bit targets) before
  V8 startup, matching headless Chrome's default. User-supplied
  --v8-flags are appended after the default so they win (V8 parses
  left-to-right and the last value wins), which keeps the existing
  escape hatch intact for memory-constrained environments.

---------

Co-authored-by: SGavrl <[email protected]>
Co-authored-by: Ben Younes <[email protected]>
Co-authored-by: zhangyang-crazy-one <[email protected]>
2026-05-29 18:20:46 +00:00
SGavrl e7bb0c676d feat(cli): --allow-private-network for localhost development (#33) (#203)
Picks up @Timtech4u's PR #33, which has been waiting since April. The
  design (CLI flag + threading through BrowserContext + http client) is
  theirs and is correct; this is a clean rewrite on top of current main
  because repeated rebase rounds had been reverting unrelated work that
  landed in the meantime.

  Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent
  SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is
  process-wide and clunky in pipelines. Multiple users (@Timtech4u in
  the original PR, then @duartemvix and @necrogami in the comment
  thread) reported they cannot use obscura for development against
  http://localhost:N without it.

  Adds --allow-private-network as a global CLI flag, works on fetch,
  serve, and scrape. Threads through BrowserContext::with_storage_and_network
  and the http client's validate_url. CdpContext gets a new_full
  builder and obscura_cdp::start_with_full_serve_options entry point so
  'obscura serve --allow-private-network' lets Puppeteer/Playwright
  navigate to localhost too. Default remains blocked.

  Backward compat: the env var still works (OR'd with the flag inside
  validate_url). All existing constructors and entry points unchanged.

  Verified end-to-end against a local http.server on 127.0.0.1: fetch
  and CDP Page.navigate both blocked by default, both succeed with the
  flag. Public URLs unaffected. Integration suite 28/28, full workspace
  cargo test all green.
2026-05-29 17:53:23 +00:00
SGavrl 1a1253118f feat(cli): default V8 heap to 4 GB on 64-bit so heavy SPAs don't OOM (#199) (#202)
V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with
  heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/
  web-scraping) blow past it and the worker dies with
  'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session
  with it.

  Always set --max-old-space-size=4096 (1024 on 32-bit targets) before
  V8 startup, matching headless Chrome's default. User-supplied
  --v8-flags are appended after the default so they win (V8 parses
  left-to-right and the last value wins), which keeps the existing
  escape hatch intact for memory-constrained environments.
2026-05-29 17:16:02 +00:00
SGavrl 682431f766 fix(cdp): allocate fresh execution context ids per isolated world (#192) (#200)
Page.createIsolatedWorld and do_navigate both hardcoded context id 100
  for every isolated world emission. After the second navigation Playwright
  expected a fresh incrementing id (101, 102) while obscura kept re-using
  100. Runtime.evaluate then failed with 'Cannot find context with
  specified id: 101' because the id was never registered in
  valid_context_ids and the client's bookkeeping had diverged from the
  server's.

  Added next_isolated_context_id counter on CdpContext (starts at 100) and
  a next_isolated_context() helper that claims an id, registers it in
  valid_context_ids, and increments. Both call sites now use it.
2026-05-29 17:11:10 +00:00
SGavrl 191c5ac7a2 fix(cdp): page.goto returns Response over Puppeteer and Playwright (#189) (#190)
Two correlation invariants real Chrome holds were missing, so the
  nav Response was never attached to goto().

  requestId === loaderId for the main Document event. Both clients
  gate isNavigationRequest() on this; we were emitting a synthetic
  per-page id.

  Network.requestWillBeSent for the Document fires before
  Page.frameNavigated. Playwright's FrameManager needs the request
  present as pendingDocument before the frame commits, otherwise
  currentDocument lands with request=undefined.

  Subresource events and the Document's responseReceived /
  loadingFinished stay where they were.

  Verified end-to-end on both clients against example.com (200),
  google.com (200), httpbin/404 (404).
2026-05-27 20:58:29 +00:00
906dcfcde1 feat: cookie jar persistence + --storage-dir flag + fix HTTP cookie header filtering (#114)
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)

* fix: resolve build errors, add storage_dir param throughout server chain

* fix: storage_dir end-to-end, plus two test-literal field additions

  Five issues found while testing PR 114 end-to-end.

  Build fixes (two test literals out of sync with new struct fields):
  - crates/obscura-net/src/cookies.rs: CookieInfo literal in
    test_domain_matches_subdomain_without_leading_dot was missing the
    same_site and expires fields added in #170.
  - crates/obscura-cli/src/main.rs: Command::Fetch literal in
    matcher_still_uses_fetch_variant was missing the new storage_dir
    field. Without these cargo test --workspace would not compile.

  run_fetch was branching between BrowserContext::with_storage (which
  hardcodes proxy=None, stealth=false, user_agent=None) and
  with_options (which has no storage_dir slot), so --storage-dir +
  --proxy + --stealth silently dropped the proxy and stealth flags.
  Same bug SGavrl flagged on dispatch.rs in the previous round, just on
  the other code path. Switched to with_storage_full which carries all
  four. Verified the fetch now errors out against an invalid proxy.

  run_fetch --eval short-circuited the function before
  context.save_cookies() ran, so cookies set during an eval session
  were never persisted to {storage_dir}/cookies.json. Moved the save
  above the return.

  obscura serve loaded cookies on startup but never saved them
  anywhere, so a session that set cookies via Network.setCookie or
  navigation dropped them on shutdown. Subscribed to Ctrl-C once in
  put a single save_cookies() after the loop so both the Ctrl-C path
  and the channel-closed path persist cookies. Verified end-to-end.

---------

Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
2026-05-26 11:09:10 +02:00
SGavrl d15cac6149 fix(js): Element.attributes, on* event defaults, per-nid getBoundingClientRect (#45) (#186)
Three DOM-completeness gaps that broke jQuery 1.11.3 init and Playwright
  actionability:

  1. Element.attributes was undefined, so legacy IE feature-detect code
     like div.attributes[eventName].expando threw TypeError. Added a
     NamedNodeMap-like backed by a new attribute_names DOM op (length,
     numeric index, string name lookup, getNamedItem/setNamedItem,
     iterable).

  2. Standard on* event handler properties were missing from window, so
     ('on' + ev) in window returned false and forced jQuery into the
     legacy IE branch. Initialised the 79 HTML-spec on* handlers to null.

  3. getBoundingClientRect returned the same {x:8,y:8,width:100,height:20}
     literal for every element, so Playwright's hit-testing couldn't tell
     elements apart. Replaced with a deterministic synthesis from _nid
     mapped into a 12-column grid sized to a 1280x720 viewport. Still not
     pixel-accurate (no layout engine), but stable across reads and
     distinct per node.
2026-05-25 22:08:49 +02:00
SGavrl 8a8c767338 fix(net): use bundled webpki roots in stealth client (#184) (#185)
set_default_paths only resolves OpenSSL's compile-time CA dirs, which
  exist on Linux but are empty on Windows. The stealth client's cert store
  ended up empty there and every TLS handshake failed with
  CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc.

  CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's
  trust list), which behaves the same on every platform and removes the
  dependency on a system CA store.
2026-05-25 19:24:56 +02:00
Ben Younes 1da6032639 feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity (#170)
* feat(cdp): implement Network.setCookie + deleteCookies, full field fidelity

Closes #168.

Adds the missing Network.setCookie (singular) and Network.deleteCookies
methods and upgrades the existing Network.setCookies/getCookies handlers
so sameSite, expires, httpOnly, secure, and the url-fallback parameter
all round-trip correctly.

Network and Storage cookie handlers share a single parse_cdp_cookie /
parse_delete_cookies_params helper to avoid drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(cdp): fall back to default cookie jar, add Network.getAllCookies

Addresses @SGavrl review on PR #170:

- Network.{set,get,delete}Cookie(s)/clearBrowserCookies now resolve the
  cookie jar via a shared helper that prefers the session page's jar and
  falls back to ctx.default_context.cookie_jar when no target has been
  attached. Mirrors how Storage.setCookies already routes when there is
  no session, so Puppeteer/Playwright flows that issue cookie methods
  before Target.attachToTarget no longer hit "No page".
- Add Network.getAllCookies (separate CDP method per spec); aliases the
  existing getCookies handler so both arms return the full jar.

Tests: 6 new #[tokio::test] cases in domains::network::tests cover the
no-session fallback for set/get/setMany/delete/clear and the new
getAllCookies arm. RED on prod-reverted file (6/6 fail with "No page" /
"Unknown Network method: getAllCookies"), GREEN on patched
(22 -> 30 obscura-cdp lib tests, obscura-net 22/22 baseline preserved).
2026-05-25 17:58:48 +02:00
SGavrl 52fcf2a35a fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.

Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:

  1. HTTP Content-Type charset=... parameter (authoritative when present)
  2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
     first 1KB of the body (HTML only)
  3. UTF-8 fallback

Use it in:
  - obscura-net Response::text (signature changed: String, not Result)
  - obscura-browser page.rs HTML body decode (the primary scrape path)
  - obscura-browser page.rs JS body decode
  - obscura-browser page.rs CSS body decode

For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.

Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.

Closes #113.
2026-05-24 20:14:53 +00:00
SGavrl f4f49ebab4 fix(cdp): unblock puppeteer connect, bound Runtime.evaluate, full TargetInfo (#181)
Three independent gaps that hurt browser-based MCP clients connecting
to obscura's CDP endpoint:

- Runtime.enable returned "No page" when called before a session was
  attached. puppeteer-extra's FrameManager.initialize fires it on the
  browser-level WebSocket BEFORE any page target exists, so the
  standard puppeteer.connect / browser.newPage flow died on connect.
  Now reply with {} when no session is attached and let the next
  Target.attachToTarget emit executionContextCreated.

- Runtime.evaluate had no per-call timeout. A long synchronous JS
  expression held the global V8 lock indefinitely and starved every
  other CDP command on the session. Honor the CDP `timeout` parameter
  and default to 30s (Chrome's protocolTimeout). Returns a clear error
  on expiry so client-side deadlines fire cleanly instead of hanging.

- Target.getTargetInfo for the browser target omitted canAccessOpener.
  Strict CDP clients (chromiumoxide) panic when the field is missing.
  Fill it in (false for the browser target itself).

Partial fix for #122. The remaining items (networkIdle hard-deadline
on SPAs, /json/list hardcoded data, docs on cheap-vs-queued methods)
need bigger or non-code changes and are deferred.
2026-05-24 20:14:35 +00:00
SGavrl 71b9b4fbab fix(js,dom): scope element queries, real timers, form/prepend/equal (#180)
- Element.querySelector / querySelectorAll scope to the receiver subtree
  via a new query_selector_scoped Rust op, not always document root.
  Same for DocumentFragment.
- setTimeout respects its delay argument via a new async op_sleep.
  setInterval is a real repeating timer instead of an alias to setTimeout.
- document.forms / images / links return live querySelectorAll results
  instead of hardcoded [].
- HTMLFormElement subclasses Element with .elements / .length, and form
  nodes are wrapped as HTMLFormElement so instanceof works. submit()
  stays on Element.prototype to preserve the existing submit-event
  dispatch and form-data navigation path.
- Element.prepend has a real implementation.
- Node.isEqualNode does a structural compare (nodeType, nodeName,
  nodeValue, attributes, recursive children) instead of nid identity.
- Fix long-standing insert_before arg-order bug: bootstrap.js was
  passing (parent, new, ref) but _dom only forwards two args, so ref
  was silently dropped. Pass (new, ref) directly. Same fix in
  replaceChild.

Adds 3 selector unit tests and 6 runtime regression tests covering
the JS shim fixes.

Closes #105.
2026-05-24 20:14:17 +00:00
SGavrl d4d749d850 fix(mcp): allow browser MCP clients through CORS preflight (#179)
The OPTIONS preflight hardcoded Access-Control-Allow-Headers to just
Content-Type. Any browser MCP client sending mcp-protocol-version (per
the MCP spec) or Authorization / X-API-Key (typical for hosted
deployments) failed the preflight and the browser blocked the actual
request.

Add those three to the allow-list and a 24h Max-Age so browsers don't
re-preflight every call. Adds an integration test that drives the
server with a real preflight request.

Closes #175.
2026-05-24 20:14:01 +00:00
SGavrl a3b21bbb70 fix(cdp,js): playwright actionability stubs and objectId resolve (#178)
Playwright fails locator actions on Obscura for two avoidable reasons
that are separate from the missing layout engine. Both have trivial
fixes that unblock the actionability polling loop.

- Element.checkVisibility(opts) was undefined. Playwright >= 1.25 calls
  it before every input event and retries until its action timeout if
  the method isn't a function. Add a stub that returns true. Without a
  layout pass we can't compute it for real, but a permissive stub still
  beats the undefined-function path.

- ARIA reflection properties were missing on Element (role, ariaLabel,
  ariaSelected, ariaExpanded, ariaHidden, ariaChecked, ariaDisabled,
  ariaRoleDescription). Playwright's getByRole and getByLabel locators
  read them. Reflect through to the underlying aria-* attributes for
  both read and write.

- DOM.getBoxModel and DOM.getContentQuads ignored the objectId param.
  Playwright passes only objectId (from a prior DOM.resolveNode), and
  the handlers silently defaulted to node 0, so dispatchMouseEvent
  always clicked the wrong element. Extract a resolve_node_id helper
  that honors nodeId, backendNodeId, and objectId in that order.

- Page.captureScreenshot and Page.captureSnapshot now return a clear
  "not supported, use a real Chromium for the screenshot leg" error
  instead of the generic "Unknown Page method" fallback. Matches the
  pattern already in place for Page.printToPDF.

Partial fix for #45. The architectural blocker (no layout engine, so
multi-element click coordinates are wrong) is unchanged.
2026-05-24 20:13:43 +00:00
Ben Younes 3d12f76d8e feat(cli): add --dump assets to inventory page sub-resources (124) (#174)
feat: add `--dump assets` to enumerate page sub-resources

Adds a `--dump assets` flag to output all page-referenced sub-resources 
(scripts, stylesheets, images, iframes, media, embeds, objects) as NDJSON, 
allowing users to easily replay the asset graph.

- Output format: `{"url":"https://example.test/app.js","type":"script"}`
- URLs are resolved against the page origin (matching `--dump links`).
- The `type` tag is derived from the source element. `<link>` elements map 
  their `rel` attribute to specific types (stylesheet, icon, manifest, preload, 
  prefetch, modulepreload, dns-prefetch, preconnect, alternate), falling back 
  to "link" if missing or unknown.
- Adds 7 tests covering URL resolution, rel-mapping, supported selectors, 
  and empty-attribute skipping.
- Updates README dump-format table and examples.
2026-05-24 17:39:30 +00:00
Ben Younes 183c444a56 fix(js,dom): add Storage global, CharacterData ChildNode methods, fix remove/isConnected (#176)
fix(dom): add missing DOM/JS shims for modern frameworks

Closes #163.

Adds missing DOM and JS shims required by modern frameworks (Svelte 5, 
Vue, Lit) and consent-management SDKs to execute without errors:

- Define the `Storage` global constructor to resolve `ReferenceError`s 
  and fix `Storage.prototype` access.
- Apply standard ChildNode methods (`before`, `after`, `replaceWith`, 
  `remove`) to `CharacterData.prototype`. This prevents `TypeError`s when 
  frameworks call these on Text or Comment nodes during hydration.
- Update `Element.prototype.isConnected` to perform a genuine parent-walk 
  rather than returning a hardcoded `true`.
- Fix `Element.prototype.remove` to properly clean up the internal ID 
  index so removed elements are no longer queryable.
- Remove outdated no-op stubs on the `Element` class that were shadowing 
  the proper polyfill implementations.
2026-05-24 17:36:19 +00:00
Ben Younes 10810e250c fix(cdp): keep HTTP control plane reachable while V8 JS evaluation blocks (#173)
fix(cdp): unblock HTTP control plane during synchronous V8 execution

Fixes #62.

Previously, the TCP accept loop and CDP processor shared a single-threaded
tokio LocalSet. Long synchronous JS evaluations starved the accept loop,
causing incoming connections to hang in the kernel backlog and effectively
killing the HTTP control plane.

This moves the TCP listener and `/json/*` endpoints to a dedicated OS
thread using `std::net::TcpListener`. HTTP requests are now handled
directly via blocking I/O on this thread, completely bypassing V8.

WebSocket connections are handed off to the LocalSet via a bounded mpsc
channel (capacity 128). If V8 stalls and the queue fills, new WS
connections are dropped (ECONNRESET) rather than buffering unboundedly
to prevent OOMs.

Includes a new integration test to verify the control plane stays
responsive during a 5s JS loop, plus inline documentation clarifying
thread lifetime semantics and a minor test TOCTOU race.
2026-05-24 17:34:55 +00:00
SGavrl e0af9fc44f fix(cdp): emit canAccessOpener in all TargetInfo payloads, add aarch64 cross-compile CI
Merges PR #165 and PR #172.

PR #165: The CDP spec marks canAccessOpener as a required boolean on
TargetInfo but Obscura was omitting it. Strict Rust clients like
chromiumoxide reject the event entirely and panic with "Created target
not present" when the field is absent. Added canAccessOpener: false
to every TargetInfo emitted by setDiscoverTargets, getTargets,
createTarget, attachToBrowserTarget, attachToTarget, and getTargetInfo.

PR #172: Added a cross-compile matrix entry for aarch64-unknown-linux-gnu
running on ubuntu-22.04 x86_64, producing obscura-aarch64-linux-cross.
This is a fallback for when GitHub ARM runners are unavailable without
removing the native ARM build.
2026-05-24 15:22:30 +00:00
SGavrl 1950048138 fix(browser,cdp,js): about:blank navigation, real getBoxModel/getContentQuads, expose _wrap and self
- about: URL scheme now handled in navigate_single instead of
  hitting validate_url and failing with "Forbidden URL scheme"
- DOM.getBoxModel calls getBoundingClientRect via JS instead of
  returning hardcoded pixel coords
- DOM.getContentQuads added (was missing, needed by Playwright click)
- globalThis._wrap exposed so CDP eval code can look up nodes by id
- globalThis.self = globalThis for webpack/bundler worker detection
2026-05-18 10:11:54 +00:00
SGavrl 913e6a4865 fix(browser,cdp): correct document lifecycle ordering and add missing DOM methods
document.readyState was hardcoded to "complete" so scripts checking
readyState === 'loading' always took the wrong branch. Fixed by using a
JS global (__documentReadyState__) that Rust sets to "loading" before
any scripts execute, "interactive" before DOMContentLoaded fires, and
"complete" after load.

document.addEventListener and document.dispatchEvent on Document were
no-ops, so DOMContentLoaded listeners registered by inline scripts were
never called. Added proper listener storage and dispatch on Document.

Adds document.getElementsByName() (was missing entirely).

Page.reload is now implemented (re-navigates to the current URL),
resolving "Unknown Page method: reload" errors from Playwright.

data: URL navigation now works, fixing Playwright's goto("data:text/html,...")
pattern that previously errored with "invalid format".

Page.navigate logic extracted into do_navigate() so reload can reuse it.

Fixes #137, #146 (SPA bootstrap lifecycle ordering).
2026-05-18 09:50:23 +00:00
SGavrl e371e942fc fix(js): re-validate redirect targets in op_fetch_url
GHSA-8v6v-g4rh-jmcm: op_fetch_url validated the initial URL through
validate_fetch_url, but reqwest auto-followed up to 10 redirects with
no further checks. An attacker-allowed origin could 302 to
http://127.0.0.1 (or any private/internal address) and read the body
back to page JS. Real-world impact: AWS instance metadata, kube
debug consoles, internal admin endpoints with permissive CORS all
become readable from a page hosted anywhere on the open web.

Disable reqwest's auto-redirect and follow manually. Apply
validate_fetch_url to every redirect target before requesting it.
Cap the chain at 10 hops to match prior behaviour. Browser-standard
method/body rules apply on cross-method redirects (301/302/303
downgrade to GET with no body, 307/308 preserve).

Verified vs unpatched build (pr69):
  - Patched: attacker-page -> 302 -> http://127.0.0.1:N is blocked
    with "Redirect to forbidden URL blocked: Access to private IP..."
    before the internal hop runs. Legitimate 3-hop redirect chain
    between allowed origins still completes.
  - Unpatched: internal-service body leaks through to page JS via
    fetch().then(r => r.text()).
2026-05-16 15:26:58 +00:00
SGavrl 3158a1fd31 fix(cdp): gate Target.createTarget file:// like Page.navigate
GHSA-q55h-vfv9-qcr5 patched Page.navigate to refuse file:// URLs
unless --allow-file-access was set, but Target.createTarget accepted
the same scheme and called page.navigate(url) directly, bypassing
the policy. A CDP client could call Target.createTarget {url: file://...}
then Runtime.evaluate to read document.body.textContent and recover
arbitrary local files.

Apply the same gate at the Target.createTarget entrypoint. Pull
the file-scheme detector into a shared util module so both domains
stay in lock-step instead of one drifting from the other.

Verified blocked vs the unpatched build (pr69):
  - Patched: Target.createTarget {url: file:///tmp/secret} errors
    "Target.createTarget to file:// is disabled. ..." before any
    navigation runs.
  - Unpatched: same call returns the file contents via
    Runtime.evaluate {expression: "document.body.textContent"}.
2026-05-16 15:26:58 +00:00
SGavrl 935cb5c0da fix(cli,js): add --version flag and script-error-resilience test (#147)
Two improvements while investigating #147:

1. `obscura --version` / `-V` now print the CARGO_PKG_VERSION instead of
   erroring out. The issue reporter explicitly called out that there was
   no version flag, and the existing `obscura serve` banner is the only
   way to discover the version.

2. New regression test in obscura-js verifies that a TypeError thrown
   from one script does not poison the runtime: subsequent scripts and
   DOM queries must continue to work. The reporter's symptom was that
   offside.js crashing collapsed --dump text output to 1 byte; that
   cascade should never happen and this test pins it down.

The classList error itself is offside.js's own bug (it indexes into the
querySelectorAll result without checking for an empty array) and only
fires when the .slideout-navigation element is absent from the DOM.
On current main that element parses correctly and offside.js runs
without error against a captured copy of the affected page.
2026-05-16 15:11:18 +00:00
SGavrl 8cb0cb429b fix(cli): refuse --stealth + socks5:// proxy instead of silent breakage (#160)
The previous workaround in page.rs rewrote socks5:// to http:// before
handing the proxy to the stealth client. That works for Clash-style
mixed-mode proxies, but breaks plain SOCKS5 servers like `ssh -ND`,
producing a confusing `TunnelUnexpectedEof` deep in the network layer.

The stealth client is backed by wreq, which does not speak SOCKS5,
so silently substituting schemes can never be correct in general.

Refuse the combination at CLI startup with a clear message that tells
the user what to do (use --proxy http://... or drop --stealth) and
revert the rewrite in Page::new. SOCKS5 without --stealth still works
through the reqwest path.
2026-05-15 21:23:53 +00:00
SGavrl b2b41529c4 ci: add aarch64-linux release binary (#127)
Uses GitHub's native ubuntu-22.04-arm runner so it's a native build
matching the same glibc 2.35 floor as the x86_64 target. README gets
the matching install snippet.
2026-05-15 21:10:29 +00:00
SGavrl aaeaf8f662 chore: drop author working notes that landed with #69 2026-05-15 20:44:46 +00:00
SGavrl b07863b5ce chore: drop author working notes that landed with #69 2026-05-15 20:44:32 +00:00
SGavrl 9c6dec4086 chore: drop author working notes that landed with #69 2026-05-15 20:44:21 +00:00
Alexandr KondakovandSGavrl 6dd1e4e65f Fix CDP click submit navigation parity (#69)
* Fix Runtime.evaluate awaitPromise settling

* Fix DOM event navigation side effects

* Fix CDP click-triggered navigation parity

* Strengthen CDP inline script click parity test

* Handle positions-like submit URLs in CDP parity


Co-authored-by: SGavrl <[email protected]>
2026-05-15 20:38:54 +00:00
SGavrl 3a6f3e6c22 Update README.md
cleaning up
2026-05-15 19:51:59 +00:00
Adam Ma 597fe5ef8d docs(readme): document --v8-flags and add Tuning V8 section (#152) (#153)
Adds a Global flags subsection to CLI Reference (covering --verbose,
--proxy, --user-agent, --obey-robots, --v8-flags) and a Tuning V8
section with the OOM-on-large-pages motivating example, so users
hitting JavaScript heap out of memory can find --v8-flags from the
use-case side rather than only from --help.

Closes #152. Depends on #131.
2026-05-15 19:51:09 +00:00
Adam Ma cb701fc2d8 feat(cli): add --v8-flags plumbing to configure V8 at startup (#130) (#131)
* feat(cli): add --v8-flags plumbing to configure V8 at startup (#130)

Obscura had no way to pass flags to V8, so users hitting OOM on pages
larger than ~5MB had no escape hatch. Adds a generic passthrough rather
than a one-off heap knob, so future V8 tuning needs no new code.

- New `obscura_js::set_v8_flags` (Once-guarded, empty-string no-op).
- Global `--v8-flags <FLAGS>` arg with `allow_hyphen_values = true`,
  applied once in `main` before any isolate is created.
- Usage: `obscura --v8-flags "--max-old-space-size=4096" fetch <url>`.

Closes #130.

* test(cli): expand --v8-flags coverage with normalize helper

Extract normalize_v8_flags() from main() so the trim/empty-check logic
is unit-testable without needing to invoke V8. Adds parse-coverage tests
for --v8-flags with serve/scrape subcommands and empty-string values,
plus pure-function tests for the normalizer.
2026-05-15 19:46:03 +00:00
h4ckf0r0day 7b209dc655 v0.1.5 2026-05-14 23:52:41 +02:00
SGavrl 18dc796df0 fix(cdp): gate file:// navigation behind --allow-file-access
GHSA-q55h-vfv9-qcr5. Any process able to reach the CDP port could
send Page.navigate {"url":"file:///etc/shadow"} and read arbitrary
local files (the process's UID's worth of read access). When obscura
runs as root inside Docker or as a systemd service that translates
straight into /etc/shadow, /root/.ssh/id_rsa, /proc/1/environ, etc.

Validate at the CDP boundary, not in obscura-net, so direct CLI
callers ("obscura fetch file://...") keep working. A new
BrowserContext.allow_file_access flag defaults to false; Page.navigate
returns an error when the requested URL is file:// and the flag is
off. CdpContext::new_with_security threads the flag from the server
entry point, and "obscura serve --allow-file-access" opts in for
local-HTML testing workflows.

Tested:
- Default obscura serve: CDP Page.navigate {file:///etc/passwd} ->
  refused with descriptive error.
- obscura serve --allow-file-access: same call succeeds.
- obscura fetch file://... CLI path: unchanged, still reads the file.
- All earlier PoCs (GHSA-6pj9 JS->file SOP, GHSA-4789 sub-resource
  SSRF, GHSA-4789 CSS template injection) still pass.
- cargo test --workspace clean.
2026-05-14 23:46:35 +02:00
SGavrl ac0efd1798 fix(browser): patch sub-resource SSRF, CSS injection, and silent redirect cap
Three findings from GHSA-4789-rq5x-jv3c.

HIGH-1: Sub-resource fetches (<script src>, <link rel=stylesheet>) had
no scheme gate, so an http(s) page could include
<script src="file:///etc/passwd"> and the body would come back as
script source. Added subresource_allowed() that mirrors the top-level
file:// SOP rule: file: sub-resources only load when the page itself
is on file:. Off-origin schemes (data:, javascript:, etc.) are
refused regardless. Private/loopback hosts were already blocked at
the obscura-net layer; this closes the file:// hole on top of that.

HIGH-2: CSS pulled from the network was string-formatted into a JS
template literal with a three-rule escaper (backslash, backtick,
${). That escaper did not handle U+2028 / U+2029 which terminate
ECMAScript template literals, letting attacker CSS break out and
run arbitrary code in the page's V8 realm. Replaced the inline
escape with escape_for_js_template_literal() which also handles
U+2028, U+2029, null, carriage return, and every other C0 control.

LOW-2: navigate_with_wait_post() capped JS-driven navigation chains
at ten but returned Ok(()) when the cap was hit, so callers could
not tell a successful load from a redirect storm. Now returns
PageError::TooManyRedirects(REDIRECT_LIMIT).

Verified with three PoCs (sub-resource SSRF, CSS template literal
break with U+2028, and the original GHSA-6pj9-j389-226p reproducer)
plus negative tests (file:// direct read, http -> http JS chain) all
pass. cargo test --workspace clean.
2026-05-14 23:45:26 +02:00
SGavrl 9edde41faf fix(browser): block JS-driven http -> file navigation, reset realm per nav
A page on http(s) could set window.onload, assign location.href to a
file:// URL, and then read document.body from the local file because
two things lined up: obscura-net allows file: URLs, and the JS
runtime was reused across navigations so window state from the prior
origin survived into the new document's load event.

Two fixes:

1. Refuse a JS-initiated chain step when it crosses from a non-file
   scheme into file:. Direct file:// navigations from the CLI/CDP
   still work because the gate only fires on the take_pending_nav
   path. file -> file chains also still work.

2. Drop and rebuild the V8 runtime on every navigation instead of
   only rebinding globalThis.document. Even within the same origin
   we should not carry over window.onload, custom properties, or
   registered event listeners from the previous document.

Tested with the PoC from the report: pages on http://127.0.0.1.nip.io
no longer leak the contents of file:///tmp/.../pwn.txt.
2026-05-14 23:15:37 +02:00
Ben Younes 09df3050f7 feat(cli): add --dump original for binary-safe raw response output (#117) (#148)
Fixes #117. Adds an `Original` variant to `--dump` that streams the raw
HTTP response body verbatim to stdout (or `--output`), bypassing the
browser/JS/DOM stack entirely. Useful for binary payloads (jpg, png,
fonts) and any non-HTML resource where parsing through the DOM layer
would corrupt or discard data.

The new path uses ObscuraHttpClient directly, honours --proxy and
--user-agent, and writes bytes through tokio::io::stdout so binary
payloads (including embedded NUL bytes) round-trip cleanly.
2026-05-14 10:00:07 +00:00
Ben Younes 2ff0f8fe1d fix(cdp): read and validate contextId in Runtime.evaluate / callFunctionOn (#51) (#149)
Fixes #51. `Runtime.evaluate` and `Runtime.callFunctionOn` previously
ignored the `contextId` / `executionContextId` parameter entirely, so
clients targeting an isolated world (e.g. Playwright's locator path,
which calls `Page.createIsolatedWorld` → contextId 100 → evaluates the
UtilityScript against that id) silently fell through to the default
context, breaking the contract advertised via
Runtime.executionContextCreated.

Track the set of advertised execution context ids on CdpContext
(pre-seeded with 1 and 2 — the default-frame contexts emitted by
Runtime.enable and post-navigation) and extend it from
Page.createIsolatedWorld. Both handlers now reject unknown ids with
`Cannot find context with specified id: <id>` — matching the wording
real Chrome returns — and log accepted ids at debug for observability.

Obscura still routes every JS call against the page's single V8 isolate,
so this PR is spec-compliance scaffolding rather than full multi-world
execution. Follow-up work to physically isolate worlds can build on the
same valid_context_ids registry.
2026-05-14 09:15:19 +00:00
Ben Younes f3c783f6f5 fix(net,js): route op_fetch_url + ES module loader through proxy_url (#139) (#151)
Fixes the remaining two paths from #139: JS-level fetch()/XHR
(op_fetch_url) and dynamic ES module imports (ObscuraModuleLoader) both
built their reqwest clients with `Client::builder().build()` — no
proxy — so every JS-initiated network call silently bypassed
BrowserContext.proxy_url. (The third path, StealthHttpClient, was
already fixed in main: page.rs:61 uses with_proxy.)

Changes:

* obscura-net: expose ObscuraHttpClient::proxy_url() so callers outside
  the crate can route their own reqwest requests through the same
  upstream proxy. The field was already there — just lacked an accessor.

* obscura-js/module_loader: add `proxy_url: Option<String>` and
  `with_proxy(base_url, proxy_url)`. ObscuraModuleLoader::new keeps the
  pre-#139 direct-connection behaviour. load() builds a reqwest client
  per import, threading the proxy and bubbling up "Invalid module proxy"
  on bad URLs.

* obscura-js/ops.rs: replace the process-wide
  `static SHARED_HTTP_CLIENT: OnceLock<reqwest::Client>` (which could
  never be reconfigured after first call) with a per-request
  `build_request_client(proxy_url)` that pulls the configured proxy out
  of SharedState.http_client.proxy_url() each call. Per-request build
  cost is ~1ms vs the network round-trip, and we no longer have a
  global singleton that lies about its config.

* obscura-js/runtime: add ObscuraJsRuntime::with_base_url_and_proxy.
  with_base_url delegates to it with None so existing tests / callers
  are unaffected.

* obscura-browser/page.rs: thread context.proxy_url through to
  with_base_url_and_proxy when constructing the JS runtime.
2026-05-14 08:25:55 +00:00
h4ckf0r0day a750797d15 v0.1.4 Release 2026-05-14 00:34:22 +02:00
SGavrl aeb1784c32 docs: add Docker install section to README 2026-05-13 22:31:44 +00:00
SGavrl 56260822b9 ci: auto-build and push Docker image on release tag
Adds a workflow that triggers on v* tag pushes and:
- Logs into Docker Hub using DOCKERHUB_USERNAME + DOCKERHUB_TOKEN secrets
- Builds the image from the existing Dockerfile
- Tags it with both the version (e.g. 0.1.4) and 'latest'
- Pushes both tags

GitHub Actions cache is used for layers to speed up subsequent builds.
2026-05-13 22:22:31 +00:00
SGavrl a23cdc0ae7 feat: add Dockerfile + --host flag for containerized deployment
Adds a multi-stage Dockerfile (rust:1-slim-bookworm builder → distroless/cc
runtime) plus a --host flag for obscura serve that defaults to 127.0.0.1
for safety but can be set to 0.0.0.0 so the container's port is reachable
via docker run -p mapping.

Dockerfile cherry-picked from #84. The SPA chunk loading commit from that
PR was left out — verified it didn't actually fix the Next.js webpack
errors on kiwi.com (#86).
2026-05-13 22:01:22 +00:00
Jordi Llonch bce055fd2a MCP support (#65)
* add mcp support

* add MCP docs to README

* wire up mcp subcommand in cli

* fix(stealth): propagate stealth feature to obscura-mcp
2026-05-13 21:38:10 +00:00
h4ckf0r0day f1cbd58894 chore(cli): bump banner version to v0.1.3 2026-05-13 23:03:52 +02:00
h4ckf0r0day 7be84b0fb7 ci: cross-compile x86_64-apple-darwin from macos-latest 2026-05-13 23:02:59 +02:00
h4ckf0r0day 27cd10156b fix(cli): use obscura-worker.exe on Windows (#140) 2026-05-13 22:56:09 +02:00
Timothy Olaleke deb5be90ee docs(skills): add agent skill for Obscura (#34)
* docs(skills): add agent skill for Obscura

Adds skills/obscura/SKILL.md following the agent-skill convention used
by addyosmani/agent-skills and vercel-labs/agent-browser. AI coding
agents that scan a repo for SKILL.md files will discover Obscura and
know when to reach for it instead of Chrome.

Covers: build (with and without stealth), CLI fetch, CDP server with
Puppeteer/Playwright examples, localhost / private-network opt-in,
scaling profile, and known limits.

* docs(skills): address review feedback

- Remove --allow-private-network and OBSCURA_ALLOW_PRIVATE_NETWORK examples (not on main)
- Drop "screenshot" trigger word; Obscura doesn't support screenshots (#37, #123)
- Reword Known Limits: screenshots are not implemented, not low-fidelity
2026-05-13 22:37:56 +02:00
Ben YounesandOra Studio d8b7c5fad1 feat(cli): add --dump markdown output mode (#54) (#78)
Surfaces the existing LP.getMarkdown CDP machinery as a CLI flag so
text-pipeline users (LLM agents, scrapers feeding markdown to other
tools) don't need a separate HTML-to-markdown converter step.

  obscura fetch --dump markdown 'https://example.com'

Layout:
- crates/obscura-js/src/markdown.rs holds HTML_TO_MARKDOWN_JS as a
  pub const so every layer can share the same DOM-walker. obscura-js
  is at the bottom of the dep DAG (obscura-browser -> obscura-js,
  obscura-cdp -> obscura-js, obscura-cli -> obscura-js) so a single
  source of truth works without a circular dependency.
- crates/obscura-browser/src/lib.rs re-exports it for ergonomics.
- crates/obscura-cdp/src/domains/lp.rs drops its inline duplicate
  and calls the shared const.
- crates/obscura-cli/src/main.rs adds the `Markdown` DumpFormat
  variant + dump_markdown(&mut Page) which page.evaluate's the
  shared script and prints the result.

Net diff: a new 75-line markdown.rs, a 70+ line deletion in lp.rs
that's reborn as 3 lines, and ~10 lines added across the other four
files. No behavior change for `--dump html|text|links`, and
LP.getMarkdown over CDP keeps the exact same JSON shape.

Four new unit tests in obscura-js cover headings, inline emphasis +
links, ordered/unordered lists, and that <script> / <style> contents
do not leak into the output.

Generated by Ora Studio
Vibe coded by ousamabenyounes

Co-authored-by: Ora Studio <[email protected]>
2026-05-13 22:36:17 +02:00