wreq upgraded to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the
latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension
order, ALPS, GREASE).
__obscura_stealth global variable introduced and passed from Rust to JS,
allowing stealth-specific JS behaviors to run only when the stealth flag
is enabled.
Tightened hardwareConcurrency and deviceMemory pools in stealth mode
to desktop-realistic values (e.g. min 4GB RAM, 4 cores), eliminating
mobile-tier values that leak the spoof.
navigator.userAgentData and getHighEntropyValues() are now overridden
in stealth mode to perfectly match the sec-ch-ua header sent by
wreq_util::Profile::Chrome145.
Fixed usedJSHeapSize sometimes exceeding totalJSHeapSize.
* feat(api): add obscura-api crate with Browser + Page API
Adds a Rust library crate exposing Obscura as a programmable browser:
- Browser: build with config (proxy, stealth, UA, storage_dir)
- Page: goto, url, evaluate, content
- BrowserBuilder with fluent API
- examples/basic.rs
Part of #169
* feat(api): add Element API + CookieStore + wait_for_selector
- Element: text(), attribute(), click() via JS evaluation
- CookieStore: set/get_all/get_for_url/save_to_file/load_from_file
- Page::wait_for_selector with 100ms polling
- Page::query_selector for single element lookup
- Updated examples/basic.rs with full interaction demo
Part of #169
* feat(api): add Browser::cookies() method
* refactor(api): move to obscura crate with feature flag
- Renamed obscura-api → obscura (per maintainer feedback #169)
- API enabled by default via 'api' feature flag
- Usage: cargo add obscura (default=api) or cargo add obscura --features api
- Removed separate obscura-api crate
* chore: exclude docker workflow from fork
* fix(api): make the obscura crate build, run, and document install correctly
- examples/basic.rs and the lib.rs doctest imported `obscura_api`, but the crate
is named `obscura`, so both failed to compile. Use the real crate name.
- wait_for_selector / query_selector read the node id with `Value::as_u64`, but
`evaluate` returns JS numbers as f64, so it was always None and every
wait_for_selector timed out. Accept an integer or a non-negative finite float.
- restore .github/workflows/docker.yml, which the PR removed (a fork-specific
change that should not land upstream).
- add crates/obscura/README.md with honest install instructions: the crate is not
on crates.io, so depend on it via git (and it builds the embedded V8 from
source). The `cargo add obscura` form does not work yet.
Verified: workspace + example build, the lib doctest passes, the example runs end
to end (goto, wait_for_selector, text, attribute), and the integration suite is
28/28 with no regressions.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
Building with --features stealth linked both wreq's BoringSSL and
reqwest's vendored OpenSSL into the same binary, causing undefined
symbol errors at link time on Linux. Switching reqwest to rustls-tls
removes the vendored OpenSSL dependency. rustls is pure Rust and
coexists cleanly with wreq's BoringSSL.
Verified: stealth and non-stealth release builds succeed, 28/28
integration tests pass.
Engine-only changes pulled out of the WPT conformance branch. The
tests, runner, and harness vendoring that drove these have moved to a
separate benchmark repo (obscura-bench).
Real bug fixes:
- Tree::insert_before created a self-cycle when the inserted node was
its own reference (the spec-conformant `child.after(x)` path that
the testharness's ChildNode helpers exercise). The fixup wrote
node.prev_sibling = node.next_sibling = own id; every subsequent
childNodes / querySelectorAll walk then looped forever and obscura
burned RAM until OOM. Per spec the op is a no-op; we now early-return.
- Tree::text_content extended to return own data for direct
CharacterData access (Text / Comment / ProcessingInstruction).
- Element.namespaceURI now reflects the namespace passed to
createElementNS instead of collapsing every non-svg namespace to
xhtml. Without this, code that creates elements in MathML, custom
XML namespaces, or the empty namespace gets misclassified as HTML.
- AbortController / AbortSignal: the previous shim was a plain object
whose addEventListener was a no-op, so listeners for 'abort'
silently never fired and any AbortSignal.timeout / AbortSignal.any
chains broke. New impl tracks aborted/reason, has a real
addEventListener / removeEventListener / dispatchEvent surface,
honors `once`, and implements the static abort / timeout / any
factories crawlers use for budgeted fetches.
- StorageEvent class added. localStorage / sessionStorage mutations
now dispatch 'storage' events on the parent window and every iframe
window so cross-frame onstorage listeners fire.
- ProcessingInstruction (nodeType 7) and DocumentType (nodeType 10)
constructors plus document.createProcessingInstruction and
document.implementation.createDocumentType. textContent returns
null for Document and DocumentType per spec.
- insertAdjacentText / insertAdjacentElement on Element. Without
these the testharness output renderer threw in a forEach without
try/catch, breaking the completion callback chain for every test
with subtests.
- Window + Node addEventListener filter non-callable listeners,
accept the EventListenerObject shape (object with handleEvent),
dedupe by (type, callback, capture), honor the `once` option, and
snapshot the listener list before dispatch so a `once: true`
removal mid-iteration doesn't corrupt the loop.
- Iframe HTML loader now extracts head <script> blocks and body
on-handler attributes (`<body onstorage=...>`), evals scripts in
the parent realm after the iframe window is built, re-attaches
handlers as window listeners. Same iframe element reusing its src
no longer double-registers in the iframe registry.
- ops.rs validate_fetch_url honors the OBSCURA_ALLOW_PRIVATE_NETWORK
env var, and the CLI --allow-private-network flag mirrors itself
into that env so the js-side fetch gate matches the http_client
gate. Without this, iframe loads of 127.0.0.1 resources were
blocked even with the flag set.
- ops.rs: create_processing_instruction, create_doctype, pi_target,
doctype_name, doctype_public_id ops added; set_text_content extended
to handle PI nodes.
CDP / navigation:
- Route all Page.navigate through process_with_interception so the
parent dispatcher stays responsive during long navs (fixes
Target.createTarget timeouts after a handful of newPage calls).
- Default waitUntil to domcontentloaded. Puppeteer and Playwright
don't pass it server-side and wait client-side via lifecycleEvent,
so the old Load default blocked nearly every nav on slow sites.
- 30s navigation hard ceiling, OBSCURA_NAV_TIMEOUT_MS to override.
- execute_scripts loop now checks wall-clock deadline per iteration.
- Emit Network.requestWillBeSent before Page.frameNavigated so
Playwright's FrameManager attaches the navigation response (#189).
- Use loaderId as the Document request id so page.goto() resolves
to a Response instead of null (#189).
- Fresh executionContextId per navigation, no more 'Cannot find
context with specified id' after the first nav (#192).
- POST navigation params plumbed through the intercept path.
DOM / Web API surface:
- MutationObserver: subtree filter walks parent chain, fires on
textContent / CharacterData.data / innerHTML, type filter respected,
oldValue passed.
- IntersectionObserver: multi-fire (initial + 120/500/1500ms),
disconnect / unobserve / takeRecords, root / rootMargin / thresholds.
- ResizeObserver: per-target firing from getBoundingClientRect, drop
the no-op override that was clobbering the real class.
- crypto.subtle.digest: real SHA-1/256/384/512 via op_subtle_digest.
- DOMParser / document.implementation.createHTMLDocument /
createDocument: detached doc with body, head, documentElement,
querySelector*, getElementById, getElementsByTagName.
- WebSocket / EventSource / BroadcastChannel: real EventTarget,
open fires on microtask.
- indexedDB: request, object store, transaction, database with
async onsuccess and in-memory Map storage.
- getComputedStyle: width/height/top/left from bbox so React
virtualization libs render items.
Site compatibility went from 3/40 to 175/175 reachable sites in
the sweep. Perf is equal or faster than pristine main on every
site tested (DCL default removes the block on long-tail subresources).
28/28 integration tests pass.
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <[email protected]>
Co-authored-by: SGavrl <[email protected]>
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes#113.
document.readyState was hardcoded to "complete" so scripts checking
readyState === 'loading' always took the wrong branch. Fixed by using a
JS global (__documentReadyState__) that Rust sets to "loading" before
any scripts execute, "interactive" before DOMContentLoaded fires, and
"complete" after load.
document.addEventListener and document.dispatchEvent on Document were
no-ops, so DOMContentLoaded listeners registered by inline scripts were
never called. Added proper listener storage and dispatch on Document.
Adds document.getElementsByName() (was missing entirely).
Page.reload is now implemented (re-navigates to the current URL),
resolving "Unknown Page method: reload" errors from Playwright.
data: URL navigation now works, fixing Playwright's goto("data:text/html,...")
pattern that previously errored with "invalid format".
Page.navigate logic extracted into do_navigate() so reload can reuse it.
Fixes#137, #146 (SPA bootstrap lifecycle ordering).
SGavrl