Colors: oklch()/oklab() and color-mix(); linear-gradient backgrounds.
Grid: named-line placement, repeat() expansion, drop whitespace-only
grid items; upgrade taffy 0.7 -> 0.12 so aspect-ratio images in
auto-height rows no longer collapse to slivers.
CSS: @layer, nesting, @property, :is()/:where(), clamp(), @media not all;
source/track/param/area and svg desc/metadata as display:none.
Images: webp + intrinsic sizing, srcset/picture, browser UA + Accept for
CDN fetches, grey placeholder on failure; width/height HTML attributes
rank below author CSS with aspect-ratio preserved for auto axes.
The render layer had no CSS Grid support, so Vector 2022's page layout
(three nested grids) collapsed to a vertical stack with the article
landing tens of thousands of pixels down the page. Wikipedia rendered
as a near-blank page and Hacker News only worked through per-site
hardcoded CSS overrides.
Removed the HN-specific hardcoding and fixed the general engine instead:
- CSS Grid layout (grid-template-areas/columns/rows via taffy's grid mode)
- Indexed, specificity-correct cascade in a new css.rs, replacing the
old test-every-selector-against-every-node scan. Cascade time on
Wikipedia went from 9.5s to under 200ms.
- Ancestor bloom-filter fast-rejection for selector matching in
obscura-dom, needed to make the indexed cascade actually fast on
descendant combinators.
- :link/:visited pseudo-classes were previously unparseable, so any
rule using them (a:link{color:...}, extremely common) silently
dropped the whole rule from the cascade.
- overflow:hidden clipping, fixing the pervasive visually-hidden-but-
accessible pattern used for skip links and screen-reader-only labels.
- @media parser robustness: it did a literal substring search for
max-width: and silently dropped the constraint on max-width : 750px
(space before colon) or max-width:calc(750px - 1px), treating
mobile-only or narrow-viewport rules as always active. The calc()
case was hiding Wikipedia's TOC sidebar via a rule meant only for
narrow screens.
- Table auto-layout approximation: the trailing auto-width td in a row
absorbs leftover space, so a width:100% table's unsized last column
reaches the right edge instead of shrinking to content.
- Inline formatting context approximation: a block holding inline-level
children (a paragraph or list item with links) is promoted to a
wrapping flex row so text and links read as continuous prose instead
of one fragment per line.
- Whitespace-only text nodes no longer get a full line-height box. They
used to stack up as blank rows, turning Wikipedia's collapsed TOC
subsections and the page header into hundreds of pixels of dead space.
- SVG rendering via resvg/usvg, plus resolving relative image URLs
against the page's base URL.
- background-image support including background-size/background-position
for correctly sized and positioned icons.
- Per-screenshot image fetch cache.
- Fixed the layout-only (no paint feature) build, which didn't compile
due to an unconditional call into paint-gated code.
Adds a paint feature to obscura-render (tiny-skia: pure Rust, CPU,
deterministic, no system deps). paint_dom lays out the DOM and fills
each element's border box with its parsed background-color over a white
page, in tree order so descendants paint over ancestors.
Style parsing now also extracts background-color (hex #rgb/#rgba/
later enhancements.
getBoundingClientRect, offsetWidth/Height, and offsetTop/Left now read
real box geometry from the obscura-render layout cache when the render
feature is compiled in. elementFromPoint inherits correct hit-testing
because it already delegates to getBoundingClientRect per element.
Adds a render feature on obscura-js (propagated through obscura-browser
and obscura-cli), a lazily-computed layout cache and viewport in
ObscuraState (feature-gated, cleared on navigation), and an
op_layout_geometry op. bootstrap.js probes the op with typeof, so the
default scraping build is unchanged and falls through to the synthetic
rect.
With --features render, block siblings stack and flex children place
horizontally at correct positions, and elementFromPoint resolves the
right element. Default build returns the synthetic rect unchanged.
Obstacle course 33/33; obscura-js unit failures identical to main.
New optional crate obscura-render, the scoped render layer for Obscura.
Phase 1 adds the layout core: a LayoutNode tree plus a viewport is laid
out with taffy (pure Rust) and border-box geometry is returned per node,
mirroring the input tree.
The crate is a workspace member but not yet a dependency of any other
crate, so the default scraping build is unchanged. Geometry-from-DOM,
JS integration, text, paint, and screenshot capture land in later
phases.
Unit tests cover block stacking, flex row layout, and padding.
The wreq dependency was declared without the gzip, brotli, deflate and zstd
features, so the stealth client returned bodies still compressed while the
reqwest path decoded them. The Chrome145 emulation advertises those encodings,
so origins do compress, and the raw gzip bytes reached the HTML parser as
document text.
Add a regression test that serves one gzip response and asserts the stealth
client decodes it.
Fixes#563.
reqwest was pinned to rustls-tls, so HTTPS only trusted the bundled
webpki roots: SSL_CERT_FILE, SSL_CERT_DIR, and the system CA store were
ignored, and obscura-net exposes no add-CA hook. Deployments behind a
TLS-inspecting egress proxy or an internal CA cannot fetch at all, and a
caching proxy in front of --proxy cannot terminate TLS.
Enable rustls-tls-native-roots alongside rustls-tls. reqwest merges the
two root stores into one, so the default trust surface only grows and
certificate verification is otherwise unchanged: unknown CAs are still
rejected. The stealth (wreq) client has its own TLS stack and is out of
scope here.
Tests: an in-process HTTPS fixture with an rcgen-minted private CA,
served via tokio-rustls (pinned to the ring provider so the test binary
has a single rustls CryptoProvider). With SSL_CERT_FILE pointing at the
CA the fetch succeeds; without it the same fixture is rejected.
Fixes#525.
glibc hands each new thread its own 64 MiB arena (up to 8x cores). With one
thread per connection that arena is the dominant per-connection memory term,
and freed chunks stay reserved in it rather than returning to the OS: resident
memory tracks the *peak* number of concurrent connections and never comes back
down, which reads as a leak although nothing is leaked and no thread is stuck.
Measured with reliability/conn-scale.py in obscura-benchmark. 100 connections
each running JS reserve 90.0 GiB of address space uncapped versus 83.5 GiB
capped; per idle connection, 65 MiB of reserved address space drops to 2.0 MiB.
`main` at that same 100-connection JS workload reserves 83.6 GiB, so with the
cap this server is level with it -- most of the total is V8's process-wide
sandbox, which `main` pays too the moment it runs any JS.
The resident-set effect is the one that bites in production. Measured in the
container image against Google Maps, four concurrent connections per round:
350 / 619 / 826 MiB over three rounds uncapped and still climbing linearly,
versus 166 / 235 / 269 MiB capped, on a decelerating curve.
Two arenas cost no measurable throughput here -- 8 concurrent connections x 12
navigations, 1.53s uncapped against 1.50s capped over three runs. V8 allocates
the JS heap through its own allocator and the Rust side is dominated by network
I/O rather than malloc traffic. Applied only from `serve`, which owns the
process, and skipped when the caller has already set MALLOC_ARENA_MAX.
Refs #430.
Network.getResponseBody returns the whole body in one base64 blob and the
cache drops anything over 2 MiB, so large downloads (images, PDFs,
archives) either failed or forced the long-running server to hold the
full body in memory (issue #360).
Add Fetch.takeResponseBodyAsStream + IO.read + IO.close. The body is moved
out of the page cache into a stream handle and delivered in bounded chunks
via IO.read, then freed on IO.close, so it is held once and released after
the download. Binary bodies are byte-exact (stored base64). To stream a
large body, raise OBSCURA_NETWORK_BODY_BUFFER_BYTES so it is cached.
This is entirely opt-in: nothing runs unless a client calls the new
methods, so the default fetch/render/scrape path is unchanged (verified:
obstacle course 33/33, obscura-vs-Chrome head-to-head within noise of the
pre-change baseline).
Fixes#360.
crypto.subtle only implemented digest. sign returned a fixed 32-byte
buffer, verify always returned true, encrypt and decrypt threw, and
generateKey, importKey and deriveBits handed back placeholder data.
Pages that use WebCrypto (anti-bot probes, PKCE and other auth flows,
client-side crypto) silently got wrong results.
Implement the secret-key half of WebCrypto on RustCrypto, following the
existing op_subtle_digest pattern:
- HMAC sign and verify (SHA-1/256/384/512)
- AES-GCM, AES-CBC and AES-CTR encrypt and decrypt
- PBKDF2 and HKDF deriveBits and deriveKey
- raw and symmetric JWK (oct) importKey and exportKey, generateKey,
wrapKey and unwrapKey
- getRandomValues and randomUUID now draw from the OS CSPRNG instead of
Math.random, so they are uniform across typed-array widths and no
longer a fingerprinting tell
Unsupported algorithms (RSA, ECDSA, ECDH) and key formats (pkcs8, spki)
throw NotSupportedError instead of returning fake data. The crypto runs
in stateless Rust ops; the bootstrap.js shim only marshals bytes and
normalizes algorithm parameters, so the native fast paths are untouched
and there is no cost unless a page calls crypto.
Verified against RFC 4231 (HMAC), the NIST GCM vectors, NIST SP800-38A
(AES-CTR), RFC 6070 (PBKDF2) and RFC 5869 (HKDF), plus roundtrip and
tag/padding tamper-rejection checks.
cargo install --features stealth failed on Windows with E0119 in cookie
0.18.1: its blanket From<T: Into<Option<OffsetDateTime>>> impl fails the
coherence check on some toolchains (issue #295). obscura never uses reqwest's
cookie store; it manages cookies through its own CookieJar and sets the
Cookie/Set-Cookie headers directly. The cookies feature only pulled in the
cookie/cookie_store crates (and the whole time subtree).
Drop it. Removes 12 packages including cookie, cookie_store, and time, so the
failing crate is no longer compiled in any build. No behavior change: the
cookie jar unit tests and the obstacle-course cookies stage still pass.
wreq upgraded to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the
latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension
order, ALPS, GREASE).
__obscura_stealth global variable introduced and passed from Rust to JS,
allowing stealth-specific JS behaviors to run only when the stealth flag
is enabled.
Tightened hardwareConcurrency and deviceMemory pools in stealth mode
to desktop-realistic values (e.g. min 4GB RAM, 4 cores), eliminating
mobile-tier values that leak the spoof.
navigator.userAgentData and getHighEntropyValues() are now overridden
in stealth mode to perfectly match the sec-ch-ua header sent by
wreq_util::Profile::Chrome145.
Fixed usedJSHeapSize sometimes exceeding totalJSHeapSize.
* feat(api): add obscura-api crate with Browser + Page API
Adds a Rust library crate exposing Obscura as a programmable browser:
- Browser: build with config (proxy, stealth, UA, storage_dir)
- Page: goto, url, evaluate, content
- BrowserBuilder with fluent API
- examples/basic.rs
Part of #169
* feat(api): add Element API + CookieStore + wait_for_selector
- Element: text(), attribute(), click() via JS evaluation
- CookieStore: set/get_all/get_for_url/save_to_file/load_from_file
- Page::wait_for_selector with 100ms polling
- Page::query_selector for single element lookup
- Updated examples/basic.rs with full interaction demo
Part of #169
* feat(api): add Browser::cookies() method
* refactor(api): move to obscura crate with feature flag
- Renamed obscura-api → obscura (per maintainer feedback #169)
- API enabled by default via 'api' feature flag
- Usage: cargo add obscura (default=api) or cargo add obscura --features api
- Removed separate obscura-api crate
* chore: exclude docker workflow from fork
* fix(api): make the obscura crate build, run, and document install correctly
- examples/basic.rs and the lib.rs doctest imported `obscura_api`, but the crate
is named `obscura`, so both failed to compile. Use the real crate name.
- wait_for_selector / query_selector read the node id with `Value::as_u64`, but
`evaluate` returns JS numbers as f64, so it was always None and every
wait_for_selector timed out. Accept an integer or a non-negative finite float.
- restore .github/workflows/docker.yml, which the PR removed (a fork-specific
change that should not land upstream).
- add crates/obscura/README.md with honest install instructions: the crate is not
on crates.io, so depend on it via git (and it builds the embedded V8 from
source). The `cargo add obscura` form does not work yet.
Verified: workspace + example build, the lib doctest passes, the example runs end
to end (goto, wait_for_selector, text, attribute), and the integration suite is
28/28 with no regressions.
---------
Co-authored-by: zhangyang-crazy-one <zhangyang.crazyone@gmail.com>
Co-authored-by: SGavrl <277843151+SGavrl@users.noreply.github.com>
Building with --features stealth linked both wreq's BoringSSL and
reqwest's vendored OpenSSL into the same binary, causing undefined
symbol errors at link time on Linux. Switching reqwest to rustls-tls
removes the vendored OpenSSL dependency. rustls is pure Rust and
coexists cleanly with wreq's BoringSSL.
Verified: stealth and non-stealth release builds succeed, 28/28
integration tests pass.
Engine-only changes pulled out of the WPT conformance branch. The
tests, runner, and harness vendoring that drove these have moved to a
separate benchmark repo (obscura-bench).
Real bug fixes:
- Tree::insert_before created a self-cycle when the inserted node was
its own reference (the spec-conformant `child.after(x)` path that
the testharness's ChildNode helpers exercise). The fixup wrote
node.prev_sibling = node.next_sibling = own id; every subsequent
childNodes / querySelectorAll walk then looped forever and obscura
burned RAM until OOM. Per spec the op is a no-op; we now early-return.
- Tree::text_content extended to return own data for direct
CharacterData access (Text / Comment / ProcessingInstruction).
- Element.namespaceURI now reflects the namespace passed to
createElementNS instead of collapsing every non-svg namespace to
xhtml. Without this, code that creates elements in MathML, custom
XML namespaces, or the empty namespace gets misclassified as HTML.
- AbortController / AbortSignal: the previous shim was a plain object
whose addEventListener was a no-op, so listeners for 'abort'
silently never fired and any AbortSignal.timeout / AbortSignal.any
chains broke. New impl tracks aborted/reason, has a real
addEventListener / removeEventListener / dispatchEvent surface,
honors `once`, and implements the static abort / timeout / any
factories crawlers use for budgeted fetches.
- StorageEvent class added. localStorage / sessionStorage mutations
now dispatch 'storage' events on the parent window and every iframe
window so cross-frame onstorage listeners fire.
- ProcessingInstruction (nodeType 7) and DocumentType (nodeType 10)
constructors plus document.createProcessingInstruction and
document.implementation.createDocumentType. textContent returns
null for Document and DocumentType per spec.
- insertAdjacentText / insertAdjacentElement on Element. Without
these the testharness output renderer threw in a forEach without
try/catch, breaking the completion callback chain for every test
with subtests.
- Window + Node addEventListener filter non-callable listeners,
accept the EventListenerObject shape (object with handleEvent),
dedupe by (type, callback, capture), honor the `once` option, and
snapshot the listener list before dispatch so a `once: true`
removal mid-iteration doesn't corrupt the loop.
- Iframe HTML loader now extracts head <script> blocks and body
on-handler attributes (`<body onstorage=...>`), evals scripts in
the parent realm after the iframe window is built, re-attaches
handlers as window listeners. Same iframe element reusing its src
no longer double-registers in the iframe registry.
- ops.rs validate_fetch_url honors the OBSCURA_ALLOW_PRIVATE_NETWORK
env var, and the CLI --allow-private-network flag mirrors itself
into that env so the js-side fetch gate matches the http_client
gate. Without this, iframe loads of 127.0.0.1 resources were
blocked even with the flag set.
- ops.rs: create_processing_instruction, create_doctype, pi_target,
doctype_name, doctype_public_id ops added; set_text_content extended
to handle PI nodes.
CDP / navigation:
- Route all Page.navigate through process_with_interception so the
parent dispatcher stays responsive during long navs (fixes
Target.createTarget timeouts after a handful of newPage calls).
- Default waitUntil to domcontentloaded. Puppeteer and Playwright
don't pass it server-side and wait client-side via lifecycleEvent,
so the old Load default blocked nearly every nav on slow sites.
- 30s navigation hard ceiling, OBSCURA_NAV_TIMEOUT_MS to override.
- execute_scripts loop now checks wall-clock deadline per iteration.
- Emit Network.requestWillBeSent before Page.frameNavigated so
Playwright's FrameManager attaches the navigation response (#189).
- Use loaderId as the Document request id so page.goto() resolves
to a Response instead of null (#189).
- Fresh executionContextId per navigation, no more 'Cannot find
context with specified id' after the first nav (#192).
- POST navigation params plumbed through the intercept path.
DOM / Web API surface:
- MutationObserver: subtree filter walks parent chain, fires on
textContent / CharacterData.data / innerHTML, type filter respected,
oldValue passed.
- IntersectionObserver: multi-fire (initial + 120/500/1500ms),
disconnect / unobserve / takeRecords, root / rootMargin / thresholds.
- ResizeObserver: per-target firing from getBoundingClientRect, drop
the no-op override that was clobbering the real class.
- crypto.subtle.digest: real SHA-1/256/384/512 via op_subtle_digest.
- DOMParser / document.implementation.createHTMLDocument /
createDocument: detached doc with body, head, documentElement,
querySelector*, getElementById, getElementsByTagName.
- WebSocket / EventSource / BroadcastChannel: real EventTarget,
open fires on microtask.
- indexedDB: request, object store, transaction, database with
async onsuccess and in-memory Map storage.
- getComputedStyle: width/height/top/left from bbox so React
virtualization libs render items.
Site compatibility went from 3/40 to 175/175 reachable sites in
the sweep. Perf is equal or faster than pristine main on every
site tested (DCL default removes the block on long-tail subresources).
28/28 integration tests pass.
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <zhangyang.crazyone@gmail.com>
Co-authored-by: SGavrl <277843151+SGavrl@users.noreply.github.com>
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes#113.
document.readyState was hardcoded to "complete" so scripts checking
readyState === 'loading' always took the wrong branch. Fixed by using a
JS global (__documentReadyState__) that Rust sets to "loading" before
any scripts execute, "interactive" before DOMContentLoaded fires, and
"complete" after load.
document.addEventListener and document.dispatchEvent on Document were
no-ops, so DOMContentLoaded listeners registered by inline scripts were
never called. Added proper listener storage and dispatch on Document.
Adds document.getElementsByName() (was missing entirely).
Page.reload is now implemented (re-navigates to the current URL),
resolving "Unknown Page method: reload" errors from Playwright.
data: URL navigation now works, fixing Playwright's goto("data:text/html,...")
pattern that previously errored with "invalid format".
Page.navigate logic extracted into do_navigate() so reload can reuse it.
Fixes#137, #146 (SPA bootstrap lifecycle ordering).
