Colors: oklch()/oklab() and color-mix(); linear-gradient backgrounds.
Grid: named-line placement, repeat() expansion, drop whitespace-only
grid items; upgrade taffy 0.7 -> 0.12 so aspect-ratio images in
auto-height rows no longer collapse to slivers.
CSS: @layer, nesting, @property, :is()/:where(), clamp(), @media not all;
source/track/param/area and svg desc/metadata as display:none.
Images: webp + intrinsic sizing, srcset/picture, browser UA + Accept for
CDN fetches, grey placeholder on failure; width/height HTML attributes
rank below author CSS with aspect-ratio preserved for auto axes.
End-to-end rasterization behind the render feature (which now enables
obscura-render/paint):
- obscura-render: screenshot_png(dom, viewport) paints and encodes to PNG
via tiny-skia's encode_png (no extra dep).
- obscura-browser: Page::screenshot renders the live DOM at a viewport.
- CLI: obscura fetch --screenshot out.png writes the settled page to a PNG.
- CDP: Page.captureScreenshot returns the PNG base64-encoded, so
Puppeteer/Playwright page.screenshot() works. captureSnapshot still
errors (it is a DOM/layer snapshot, not an image).
Verified: both paths produce correct 1280x720 RGBA PNGs with elements
laid out and colored as the CSS specifies (block stacking, backgrounds,
white page). Default build unchanged; obstacle course 33/33.
Text, borders, and images are not yet painted; the box layout and
background colors are rendered.
getBoundingClientRect, offsetWidth/Height, and offsetTop/Left now read
real box geometry from the obscura-render layout cache when the render
feature is compiled in. elementFromPoint inherits correct hit-testing
because it already delegates to getBoundingClientRect per element.
Adds a render feature on obscura-js (propagated through obscura-browser
and obscura-cli), a lazily-computed layout cache and viewport in
ObscuraState (feature-gated, cleared on navigation), and an
op_layout_geometry op. bootstrap.js probes the op with typeof, so the
default scraping build is unchanged and falls through to the synthetic
rect.
With --features render, block siblings stack and flex children place
horizontally at correct positions, and elementFromPoint resolves the
right element. Default build returns the synthetic rect unchanged.
Obstacle course 33/33; obscura-js unit failures identical to main.
Each connection runs on its own OS thread with its own V8 isolates, so
connection count is what drives the server's thread footprint.
MAX_PENDING_WS_HANDOFFS bounds only the handoff queue: once a stream is handed
off, nothing limited how many could be live at once, and a client could grow
the server without limit.
Reserve a slot in the accept loop and release it from a guard that runs when
the connection thread unwinds, on every path including panics and the early
returns before the runtime is built -- a plain decrement at the end of the
closure would leak slots until the cap wedged the server shut. Connections
past the limit get an explicit 503 with X-Obscura-Reason: max-connections
rather than a bare reset, so a client can tell a refusal from a crash. The
refusal happens in the WS dispatch loop, so the HTTP control plane (#62) stays
reachable at the cap.
Default 128, matching the handoff bound and well above any real client fan-out
(Playwright/Puppeteer use one connection per browser). At the cap, with arenas
capped by the previous commit: 146 threads, 33.2 GiB of reserved address space
and 51 MiB resident -- nearly all of that reservation being V8's process-wide
sandbox, which is present at zero connections.
Threads are the honest cost here, not memory: at 100 connections each running
JS this server reserves 83.5 GiB against `main`'s 83.6 GiB, because `main`
pays the same V8 sandbox as soon as it runs any JS. What it does not pay is
118 threads against 17.
The test covers all three properties, and specifically that a closed
connection frees its slot: a leak there would only surface as a server that
silently stops accepting after N lifetime connections.
Refs #430.
obscura serve --workers N --proxy ... spawned each worker with the proxy
as a --proxy CLI flag, so the plaintext credential was visible to any
local user via ps / /proc/<pid>/cmdline (issue #366). scrape already
avoids this by passing the proxy in OBSCURA_PROXY, which is only readable
by the owner.
Pass the proxy to serve workers via OBSCURA_PROXY too, and have the serve
command read that env as a fallback when no --proxy flag is given. The
worker argv no longer contains the credential.
Fixes#366.
extract_readable_text recursed once per DOM nesting level, so a deeply
nested page overflowed the call stack and aborted the process (exit 134)
on --dump text. This is the CLI counterpart of the serialize/textContent
walks made iterative in obscura-dom; that fix did not cover this function.
Rewrite it as an iterative DFS over an explicit work stack, with a
Newline work item for a block element's trailing newline so output is
byte-for-byte unchanged, plus the same defense-in-depth node cap the DOM
walkers use. Relates to #362.
fetch handled one URL at a time, so checking many raw resources meant
wrapping it in an external loop. Add --file <path> to read newline-
delimited URLs (- for stdin; blank lines and # comments skipped) and
--concurrency <N> to fetch them in parallel bounded by a semaphore.
Batch mode is raw HTTP only (--dump original, the default in this mode)
and prints one JSON status line per URL (url, ok, status, content_type,
bytes, elapsed_ms, or error) in input order. Rendered/DOM batch work
still belongs to scrape. A single positional URL is unchanged.
Fixes#349.
The single-worker serve path honored --host, but run_multi_worker_serve
hardcoded the load balancer to 127.0.0.1. With --host 0.0.0.0 and
--workers >1 (the common Docker setup) the balancer bound loopback only,
so the mapped port was refused from outside the container. Bind the
balancer to the requested host. Workers stay on loopback and are reached
only by the balancer.
Fixes#336.
--stealth was a per-subcommand flag defined separately on fetch, serve,
and mcp, and missing from scrape entirely. You could not enable stealth
for parallel scraping, and `obscura --stealth fetch` (flag before the
subcommand) errored.
Promote --stealth to a single global flag like --proxy, so it is valid
before or after any subcommand and applies uniformly to fetch, serve,
scrape, and mcp. scrape now forwards it to its workers via OBSCURA_STEALTH
(mirroring OBSCURA_PROXY), and the bare `obscura` default CDP server
honors it too.
Verified: navigating directly to a blocklisted tracker renders 55503
chars without stealth and 39 (blocked) with it, identically on fetch and
scrape. Obstacle course 33/33.
Adds DumpFormat::Cookies that serializes the full browser cookie jar as
pretty-printed JSON after page load. Unlike document.cookie, this captures
HttpOnly cookies inaccessible from JS — useful for extracting session
tokens set by anti-bot challenges.
--dump assets only walked static DOM attributes (script[src], img[src], ...),
so resources a page pulled in with fetch() or XHR never showed up (issue #301).
Record each absolute URL requested through op_fetch_url in ObscuraState,
expose it via the runtime and Page, and append those URLs to --dump assets as
type "fetch", deduped against URLs already listed from the DOM.
* feat(stealth): randomize browser profile per session
Add a pool of 8 realistic browser profiles (Win10/Win11/macOS, Chrome
143-146). One is picked randomly per BrowserContext instead of always
advertising Linux, which is a strong bot signal.
Each profile keeps navigator.platform, userAgentData.platform, and
userAgentData.platformVersion internally consistent with the UA string.
- profiles.rs: static PROFILES pool + random_profile() picker
- context.rs: pick profile on construction, store platform fields
- page.rs: call rt.set_platform() alongside rt.set_user_agent()
- runtime.rs: add set_platform() to inject __obscura_platform globals
- bootstrap.js: read platform/userAgentData from globals (default Win32)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(stealth): match GPU renderer pool to OS profile
Windows profiles get ANGLE Direct3D11 renderers; macOS profiles get
ANGLE Metal renderers. Vendor strings updated to match (Google Inc. (Apple)
for Apple Silicon/Intel Mac, Google Inc. (NVIDIA/AMD/Intel) for Windows).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(stealth): navigator.webdriver returns false instead of undefined
Browsers set navigator.webdriver to false when automation is not active.
Returning undefined is a detectable bot signal — fingerprinting scripts
check typeof navigator.webdriver === 'boolean' as part of headless detection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Make stealth profile stable by default, rotation opt-in
Picking a random profile per BrowserContext meant one address would cycle
through different browser identities, which is itself a bot signal, and the
rotated profile carries no matching TLS or timezone fingerprint. Default to a
single stable profile; opt into rotation with OBSCURA_ROTATE_PROFILE, or pin
one with OBSCURA_PROFILE=<index>.
Drive timezone from the process TZ so Intl.DateTimeFormat and Date report one
consistent zone. Default Europe/Berlin; override with OBSCURA_TIMEZONE.
Update navigator.webdriver test to expect false, matching real Chrome.
---------
Co-authored-by: SGavrl <277843151+SGavrl@users.noreply.github.com>
serve rejected --quiet ("unexpected argument"), so there was no way to
silence the per-page WARN logs (obscura_browser::page script errors) that
flood the console when serving pages with broken scripts. Add the flag to
the Serve command and cover it in is_quiet_command, so it maps to the same
"off" log filter that fetch and scrape already use. Closes#264.
The default V8 flags only set --max-old-space-size (a ceiling, not actual
use). The young generation (default 16MB per semi-space) plus V8's
memory-heavy codegen inflate RSS during parse and JS allocation. Add
--max-semi-space-size=4 and --optimize-for-size to the defaults.
Cuts RSS ~10-17% on real pages (ycombinator.com 170MB -> 141MB,
react.dev 94MB -> 83MB, wikipedia 58MB -> 53MB) with no measurable speed
cost: a JIT-sensitive 8M-iteration loop is unchanged (V8 still optimizes
hot paths), and the obstacle course stays 33/33 at the same latency.
Users can still override via --v8-flags.
Add a --host CLI flag to 'obscura mcp --http' so the server can bind to
addresses other than 127.0.0.1 (e.g. 0.0.0.0 for Docker Compose sidecar
deployments). The default remains 127.0.0.1 for security.
Closes#253
* fix(dom): read insert_before prev_sibling after detach to avoid a sibling cycle
insert_before captured the reference node's prev_sibling BEFORE detaching the
node being inserted. When page JS inserts a node before its own immediate
previous sibling (a no-op reorder frameworks do constantly), detach moves that
pointer, so the stale prev_id spliced new.next_sibling = new. The resulting
sibling self-cycle made descendants() inner next_sibling walk loop forever in
native Rust, uninterruptible by tokio or the V8 watchdog. This hung ebay.com,
found by a 1500-page crash/hang sweep.
Read prev_sibling after detach so it reflects the post-detach tree. Cap the inner
sibling-collection loops in descendants() and log when a descendants cap fires,
as defense in depth so a sibling cycle from any path cannot hang the engine. Add
test_insert_before_previous_sibling_no_cycle.
ebay.com now renders (192K chars) instead of hanging. Verified: 33/33 obscura-dom
tests, obstacle course 33/33.
* fix(cli): settle async --eval work before reading the page (#248)
--eval ran after the selector check and returned its value immediately, so an
expression that only kicks off async work (e.g. fetch().then(t => DOM write))
returned the still-pending Promise as {} and any --selector/--dump were ignored.
When --eval is combined with --selector and/or --dump, run the eval to kick off
its async work, settle the event loop again so it completes, then read the page
(selector wait + dump) instead of returning the Promise. A bare --eval still
returns its value unchanged, so synchronous expressions are not affected, and the
eval stays bounded by evaluate_with_timeout.
dump becomes Option<DumpFormat> so an explicit --dump is distinguishable from the
default. Verified: the repro now returns the fetched content, a plain
JSON.stringify eval is unchanged, 37 + 16 cli tests pass.
* dom: complete detached documents; add ProgressEvent and parseHTMLUnsafe
createHTMLDocument returned a document whose body was null: parsing the
skeleton string as innerHTML of <html> collapsed through the fragment parser
and kept only <title>. Build head>title and body explicitly instead. The
DOMParser doc node also gains appendChild/removeChild/insertBefore and a
doctype getter, and createDocument carries its doctype argument.
This lets dom/common.js setupRangeTests() finish (it builds fixtures with
new Document(), createHTMLDocument(), createDocument()) instead of aborting at
harness setup, so the dom/ranges files run. Range-collapse 186/186,
Range-attributes 1/1, Range-cloneRange 55/62, StaticRange-constructor 14/17.
The large O(ranges x points) matrix files (Range-set, comparePoint,
compareBoundaryPoints) still time out: the boundary-point comparison walks the
tree through per-step DOM ops, which is too slow for those matrices. Speeding
that up (a Rust node-position op) is a separate change.
Also add globalThis.ProgressEvent (Event subclass with lengthComputable/
loaded/total) and the static Document.parseHTMLUnsafe(html).
* perf(dom): compute Range tree-order in Rust ops
The Range boundary-point algorithms compared nodes by walking ancestor chains
and sibling lists from JS, one DOM op per step. The dom/ranges matrices
(testRanges x testPoints, thousands of comparisons) made that time out.
Add node_index, compare_order, and node_root ops that do the walk in Rust over
the DOM tree (slab lookups, no per-step FFI), and use them in _rngNodeIndex,
_rngOrder, and _rngRoot. Range-comparePoint 3534/5580, Range-isPointInRange
5197/5733, Range-compareBoundaryPoints 6441/9313, all completing in under 10s
where they previously did not finish. Not on any engine hot path (querySelector,
mutation, navigation, fetch do not use these).
* fileapi: real byte-backed Blob/File, FileReader state machine, Request.blob
Blob was a one-line stub that string-joined its parts (wrong size for binary
or multibyte input) and exposed only text(). Replace it with a Uint8Array-backed
Blob: normalizes string/ArrayBuffer/typed-array/Blob parts, correct size,
slice(), text()/arrayBuffer()/bytes(), type normalization (lowercased, cleared
when non-ASCII), the endings option (native CRLF normalization), and
Symbol.toStringTag. File adds name/lastModified, the 2-argument requirement, and
its own toStringTag.
FileReader becomes a real state machine: EMPTY/LOADING/DONE, throws
InvalidStateError when a read starts while already LOADING, and fires
loadstart/progress/load/loadend (and abort) to both on* handlers and
addEventListener. readAsText/DataURL/ArrayBuffer/BinaryString produce correct
results. Request gains blob().
FileReader-multiple-reads 6/6, File-constructor-endings 10/11, Blob-slice
141/150.
* encoding: TextDecoder supports legacy charsets via encoding_rs
TextDecoder was a UTF-8-only JS shim that ignored its label, so
new TextDecoder('shift_jis'|'euc-jp'|'big5'|'gbk'|'windows-1252'|'iso-8859-1')
.decode() returned garbage. That matters for scraping: pages and scripts decode
legacy-encoded bytes (XHR/fetch responses, File/Blob text) all the time.
Back it with encoding_rs (already used by the net layer for response bodies):
obscura-net gains label_name() and decode_with_label(), exposed to JS as
op_encoding_for_label and op_text_decode. The constructor validates the label
(RangeError) and sets the canonical .encoding/.fatal/.ignoreBOM; decode()
supports fatal mode (TypeError on invalid input) and BOM handling.
Plain UTF-8 non-fatal decoding keeps a pure-JS fast path (no op, no JSON), so
Response.text()/Blob.text() on large bodies are unchanged; the op only runs for
legacy encodings or fatal mode.
textdecoder-labels 222/222, textdecoder-fatal 36/36, byte-order-marks 3/3,
utf16-surrogates 10/10, single-byte-decoder 168/504.
* selection: real Selection over Range (single instance per document)
getSelection() returned a fresh plain object each call, so it was not a Selection
instance, had no identity (window.getSelection() !== document.getSelection()),
and was missing removeRange/empty/collapseToStart/collapseToEnd/setPosition/
setBaseAndExtent/modify while collapse/extend/getRangeAt were broken stubs. That
was ~4862 failing 'selection[method] is not a function' subtests.
Replace it with a real Selection class built on the live Range: one cached
instance per document (window.getSelection() === document.getSelection(), null
for documents with no defaultView), an internal range + direction, derived
anchor/focus/isCollapsed/type/rangeCount, and the full method set
(getRangeAt with IndexSizeError, addRange, removeRange, removeAllRanges/empty,
collapse/setPosition, collapseToStart/End, extend, setBaseAndExtent,
selectAllChildren, containsNode, toString) with the spec DOMException names.
modify() stays a no-op (needs visual layout). No ops.rs changes; not on any hot
path.
setBaseAndExtent 120/120, removeAllRanges 104/104, collapseToStartEnd 51/51,
type 26/26, extend-exception 1/1.
* forms: HTMLInputElement valueAsNumber/valueAsDate/stepUp/stepDown
These were absent. Implement the number/date conversion API for the applicable
types (number, range, date, month, week, time, datetime-local): parsers and
formatters per type (Date.UTC for date/datetime-local, ISO-week Monday math for
week, months-since-epoch for month), valueAsNumber get/set, valueAsDate get/set,
and stepUp/stepDown(n) with the step scale/default per type, step-base snapping,
and min/max clamping. Non-applicable types and step="any" throw InvalidStateError;
non-finite valueAsNumber throws TypeError; range with empty value reads as the
midpoint (50 for 0..100). Plus min/max/step attribute reflection.
All on Element accessors gated by input type, only evaluated on explicit access;
not on any hot path.
input-stepup 51/53, input-valueasnumber 60/64, input-valueasdate 27/30.
* perf(encoding): skip label op for no-arg TextDecoder (Response/Blob text path)
new TextDecoder() with no argument (what Response.text()/Blob.text() and most
pages use) is UTF-8; resolve that without the op_encoding_for_label round trip.
Explicit labels still validate via the op.
* core: HTMLDialogElement, Node.normalize, Range.createContextualFragment
HTMLDialogElement was a bare alias of Element, so ~40 Core files crashed at the
first dialog.show()/showModal(). Add the dialog members on Element.prototype
(matching the popover/input convention): open reflection, returnValue (internal
slot, String coercion), oncancel/onclose defaulting to null, show()/showModal()
/close()/requestClose() with the open-state transitions, InvalidStateError
throwing (show-on-modal, showModal-on-open, showModal-disconnected), the async
non-bubbling close event, the synchronous cancelable cancel event (no re-entrant
cancel), beforetoggle/toggle, and closedBy. matches(':modal') reports a modally
shown dialog (like :popover-open). Modal top-layer/focus/render stays unhandled
(layout).
Node.normalize() now merges adjacent Text nodes and drops empty ones (was a
no-op). Range.createContextualFragment(html) parses a fragment.
createContextualFragment 34/35, dialog-open 3/3, dialog-return-value 1/1,
dialog-showModal 5/10, dialog-requestclose 22/75. Not on hot paths.
* custom-elements: CustomElementRegistry class, getName, ElementInternals, HTMLUnknownElement
customElements was a bare object literal, so 'CustomElementRegistry is not
defined' crashed ~22 Core files at top level (idlharness/feature-detect), and
HTMLUnknownElement/ElementInternals/attachInternals/getName were missing.
Promote the registry to a real CustomElementRegistry class (window.customElements
is now an instance, so 'define in CustomElementRegistry.prototype' works and the
class is exposed). Add spec define() validation/throw order (TypeError for a
non-constructor, SyntaxError for an invalid name, NotSupportedError for a
duplicate name/constructor or reentrant define), getName(), and a SyntaxError
reject in whenDefined for invalid names. The existing define -> upgrade ->
connectedCallback path (used by lit/Polymer on real pages) is preserved.
Add HTMLUnknownElement, ElementInternals (setFormValue/setValidity/checkValidity/
validity/validationMessage/form/states), and Element.attachInternals() with its
NotSupportedError cases.
Does NOT include the [[HTMLConstructor]] (new.target) construction rework or the
reaction wiring into appendChild/setAttribute (hot paths) - deferred to avoid
element-model and perf risk. getName 4/4.
* shadow-dom: ShadowRoot.adoptedStyleSheets (mirror Document)
ShadowRoot lacked adoptedStyleSheets; constructible-stylesheet adoption into a
shadow root threw. Add the getter/setter on ShadowRoot.prototype, matching
Document.adoptedStyleSheets.
* conformance: charset/URL encoding override, IDL reflection, Range and Storage fixes
Encoding: track the detected document character encoding and expose it as
document.characterSet/charset/inputEncoding instead of hardcoding UTF-8. Add the
WHATWG URL encoding override for <a>/<area> hrefs, so a legacy-charset document
percent-encodes the query in its own encoding (EUC-JP U+8108 becomes %CC%AE).
ASCII delimiters stay literal so structured queries are unaffected, and
unmappable code points become %26%23NNN%3B. UTF-8 documents take an early-return
so the href getter hot path is unchanged. fetch()/XHR text() now decode using
the response Content-Type charset instead of always UTF-8. Adds op_url_encode_query
and a document_encoding op; encoding state is plumbed through Page and the runtime.
Reflection: install the reflected IDL attribute layer once on Element.prototype
(no per-element defineProperty, so element creation and query paths are
untouched): global attributes (title, lang, dir, accessKey, slot, autofocus,
hidden, tabIndex) with correct type coercion and enum canonicalization, plus the
ARIAMixin (aria-* reflected as the ariaXxx nullable DOMString properties).
Range: compareBoundaryPoints ToUint16-converts the how argument (NaN/Infinity
no longer false-throw) and raises a real WrongDocumentError DOMException for
ranges that do not share a root.
Storage: reimplement localStorage/sessionStorage as a WHATWG Proxy so property
access (localStorage.foo = x, delete, in, Object.keys) stays in sync with
length, key() and iteration.
* conformance: HTML attribute folding, compareDocumentPosition, URLSearchParams, DOMParser, fetch UA
Next WPT Core batch, all verified live and benchmarked for no hot-path regression.
DOM:
- compareDocumentPosition rewritten to spec: real PRECEDING/FOLLOWING via the
compare_order op, CONTAINS/CONTAINED_BY, and DISCONNECTED|IMPLEMENTATION_SPECIFIC
for different roots. Adds the DOCUMENT_POSITION_* and missing nodeType constants
on Node and its prototype. WPT getPosition() depends on this, so it cascades into
the Range comparePoint/compareBoundaryPoints/isPointInRange expectations.
- HTML elements now ASCII-lowercase attribute names in getAttribute/setAttribute/
removeAttribute/hasAttribute, so el.setAttribute('accessKey') reflects via
getAttribute('accesskey'). The fold is gated behind a cheap uppercase scan and a
namespaceURI HTML guard (SVG/foreign attributes keep their case), and getAttribute
only folds on a miss, so the all-lowercase hot path is unchanged (benchmarked flat).
- Range comparePoint/isPointInRange ToUint32-coerce the offset (negative throws
IndexSizeError); createTreeWalker/createNodeIterator keep an explicit whatToShow of 0.
URL:
- URLSearchParams uses a proper form-urlencoded byte decoder (partial/invalid escapes
stay literal instead of decodeURIComponent throwing on the whole string), 2-arg
has(name,value)/delete(name,value), and a sequence-pair TypeError. Adds URL.parse
and URL.canParse statics.
Other:
- fetch()/XHR send a default Chrome User-Agent (the navigation path set one but
scripted requests went out with none, so UA-gated servers rejected them); honors an
explicit override.
- DOMParser documents expose URL/compatMode/characterSet/readyState/styleSheets/
defaultView and createTreeWalker/createNodeIterator; XMLDocument is defined globally.
* perf(dom): O(1) querySelector('#id') via the id index
querySelector('#id') went through the full selector engine, scanning descendants
in tree order until the first match (O(n), ~68us per call over a 2000-element
document in the micro-bench). A bare ASCII id selector now resolves through the
existing id index in O(1): ~68us -> ~1.25us per call, about 55x faster, with no
change to any other path (create/getAttribute/setAttribute/querySelectorAll/href
all flat). querySelector by id is one of the hottest scraping operations.
The fast path is conservative: only a bare `#name` selector with an ASCII
letter/underscore-led identifier takes it; escapes, combinators, commas,
whitespace, compounds, and non-ASCII fall through to the full engine unchanged.
For a scoped element.querySelector it returns the indexed element only when it is
a strict descendant of the scope, and falls back to the scan otherwise, so
duplicate-id scoped queries stay correct.
Also makes the id index keep the FIRST element in tree order per id (or_insert at
parse time) instead of the last, so getElementById and querySelector('#id') return
the first-in-tree-order element on duplicate ids, per spec (previously last).
* fix: bound synchronous V8 hangs with a termination watchdog and add op panic safety
Heavy pages could hang obscura fetch indefinitely: tokio::time::timeout only
cancels at await points, so synchronous V8 work (a page's script bootstrap, a
microtask storm) pinned the thread and ran unbounded.
Add a reusable V8 terminate_execution watchdog (arm_watchdog/disarm_watchdog) and
apply it to every place that runs synchronous V8 work: the post-load settle, both
navigation event-loop pumps, the whole script-execution phase, and --eval. A CLI
process-level hard deadline backstops the rest, since a hang inside a Rust op
cannot be cancelled by tokio or terminate_execution.
Also harden against panics: op_dom is wrapped in catch_unwind so a DOM-op panic
degrades to a null result instead of unwinding into V8's FFI frame and aborting
the process; the append_text unwrap is removed; panic = unwind is pinned in the
release profile.
Verified: obstacle course 33/33, no perf regression (per-op deltas within the
measurement noise floor), 6 of 8 previously-hanging real sites now render and the
other 2 exit deterministically at the hard deadline.
* fix(dom): reject cyclic reparenting so descendants() cannot hang the engine
appendChild/insertBefore only guarded the self-append case (parent == child),
never the case where the moved node is an ancestor of the new parent. When page
JS does node.appendChild(ancestorOfNode) the parent/child graph becomes cyclic;
the next querySelector runs descendants() (a DFS with no visited set) and loops
forever, growing a Vec unbounded and pinning the thread in native Rust where
neither tokio nor the V8 watchdog can interrupt it. This hung techcrunch and
kayak.
Reject the cycle at the mutation site per the DOM HierarchyRequestError rule: if
the moved node is an inclusive ancestor of its target parent, treat the move as a
no-op. The walk is gated on the moved node having children, so a fresh/leaf
append (the common case) stays O(1); only relocating a populated subtree pays
O(depth). descendants() also gets a result-length cap as defense in depth so a
cycle from any future path cannot wedge the engine.
techcrunch and kayak now render (37K and 1.26M chars). Verified: new
test_reparent_cycle_is_rejected, obstacle course 33/33, no perf regression.
* fix: restore regressions introduced by #227#227 deleted build.rs (reverting #224's version-from-tag feature),
removed DOM cycle guards in append_child/insert_before that prevent
infinite loops on self-append/self-insert, dropped the CharacterData
textContent fast-path, removed PI/Doctype Rust ops, and cut the
env_allows_private_network() check from validate_fetch_url.
Restored:
- crates/obscura-cli/build.rs (version from tag/env)
- OBSCURA_BUILD_VERSION in version string and banner
- allow_private_network env var propagation in main.rs
- append_child/insert_before self-cycle guards in tree.rs
- textContent CharacterData fast-path in tree.rs
- create_processing_instruction, create_doctype, pi_target,
doctype_name, doctype_public_id ops in ops.rs
- ProcessingInstruction branch in set_text_content op
- env_allows_private_network() check in validate_fetch_url
* fix(js): Europe/Berlin timezone, plugins iterator recursion
Two bugs in the stealth bootstrap code from #227:
- 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat
silently fell back to UTC. Changed to 'Europe/Berlin'.
- plugins[Symbol.iterator] called this[Symbol.iterator]() recursively,
blowing the stack on Array.from(navigator.plugins). Fixed to bind
Array.prototype[Symbol.iterator] directly to the array.
* url: back URL/URLSearchParams with the url crate, add <a>/<area> decomposition
The URL global was a regex shim: it mis-parsed userinfo (hostname came back as
the username, port as "pass@host"), had no username/password, and its setters
were no-ops. Replace it with two ops (op_url_parse, op_url_set) backed by the
WHATWG-compliant url crate, and add the HTMLHyperlinkElementUtils members
(protocol/host/hostname/port/pathname/search/hash/origin/username/password and
their setters) to <a> and <area>, which had none.
This was the single largest WPT failure. a.search returned undefined, so the
encoding encode-href tests threw "Cannot read properties of undefined (reading
'substr')" about 225k times. With the fix url-constructor passes 831/888,
url-setters 192/279, urlsearchparams-sort 17/17, url/a-element 415/887, and the
encode-href files run their assertions instead of crashing.
op_url_set runs the url crate inside catch_unwind: a few setters panic on the
pathological inputs the url-setters tests use, and a panicking setter must be a
no-op (the WHATWG "do nothing on invalid value" rule) rather than abort.
URLSearchParams also gained getAll/sort/size/keys/values/entries/iterator and a
back-reference so mutations write through to the bound URL.
* dom: real Range/StaticRange, createDocumentType, XML Document type
Range was a no-op stub (setStart/setEnd discarded their arguments, so
startContainer/endContainer read back undefined and every dom/ranges helper
that walked range.endContainer crashed with "reading 'nodeType' of undefined").
Replace it with a property-storing Range over the live tree: setStart/End and
the *Before/*After/selectNode/selectNodeContents/collapse positioning, plus the
boundary-point algorithms comparePoint, compareBoundaryPoints, isPointInRange,
intersectsNode, commonAncestorContainer, cloneRange and a real stringifier, all
built on a correct preorder tree-order comparison. Add StaticRange.
new Document() now reports an XML contentType (it has no backing node id), so
createCDATASection no longer throws NotSupportedError on it. Add
DOMImplementation.createDocumentType with the lenient name validation browsers
use and an owner-document backref so DocumentType.ownerDocument is correct.
StaticRange-constructor 14/17. This does not fully unblock dom/ranges yet:
dom/common.js also needs createHTMLDocument()/createDocument() to return
documents with a working body/createComment/etc., which is a separate fix.
* fix(browser,cli): settle the event loop after navigation
obscura fetch read the page at the load event and never pumped the event
loop afterward on the eval/dump path, so --wait was effectively ignored
there. Pages that schedule timers, do async work, or only report through a
completion callback that runs after load (testharness add_completion_callback,
SPAs) were observed frozen at the load event and never produced their result.
Add Page::settle, which drives run_event_loop until the loop goes idle or
--wait elapses (returns early when idle, so static pages stay fast), and run
it after navigation in the fetch path.
* fix: restore regressions introduced by #227#227 deleted build.rs (reverting #224's version-from-tag feature),
removed DOM cycle guards in append_child/insert_before that prevent
infinite loops on self-append/self-insert, dropped the CharacterData
textContent fast-path, removed PI/Doctype Rust ops, and cut the
env_allows_private_network() check from validate_fetch_url.
Restored:
- crates/obscura-cli/build.rs (version from tag/env)
- OBSCURA_BUILD_VERSION in version string and banner
- allow_private_network env var propagation in main.rs
- append_child/insert_before self-cycle guards in tree.rs
- textContent CharacterData fast-path in tree.rs
- create_processing_instruction, create_doctype, pi_target,
doctype_name, doctype_public_id ops in ops.rs
- ProcessingInstruction branch in set_text_content op
- env_allows_private_network() check in validate_fetch_url
* fix(js): Europe/Berlin timezone, plugins iterator recursion
Two bugs in the stealth bootstrap code from #227:
- 'Europe/Germany' is not a valid IANA timezone; Intl.DateTimeFormat
silently fell back to UTC. Changed to 'Europe/Berlin'.
- plugins[Symbol.iterator] called this[Symbol.iterator]() recursively,
blowing the stack on Array.from(navigator.plugins). Fixed to bind
Array.prototype[Symbol.iterator] directly to the array.
Set the CLI build version from OBSCURA_VERSION, GitHub tag refs, or Cargo.toml fallback. Use that same build version for both --version and the startup banner, and pass the Docker tag version into Docker builds.
Engine-only changes pulled out of the WPT conformance branch. The
tests, runner, and harness vendoring that drove these have moved to a
separate benchmark repo (obscura-bench).
Real bug fixes:
- Tree::insert_before created a self-cycle when the inserted node was
its own reference (the spec-conformant `child.after(x)` path that
the testharness's ChildNode helpers exercise). The fixup wrote
node.prev_sibling = node.next_sibling = own id; every subsequent
childNodes / querySelectorAll walk then looped forever and obscura
burned RAM until OOM. Per spec the op is a no-op; we now early-return.
- Tree::text_content extended to return own data for direct
CharacterData access (Text / Comment / ProcessingInstruction).
- Element.namespaceURI now reflects the namespace passed to
createElementNS instead of collapsing every non-svg namespace to
xhtml. Without this, code that creates elements in MathML, custom
XML namespaces, or the empty namespace gets misclassified as HTML.
- AbortController / AbortSignal: the previous shim was a plain object
whose addEventListener was a no-op, so listeners for 'abort'
silently never fired and any AbortSignal.timeout / AbortSignal.any
chains broke. New impl tracks aborted/reason, has a real
addEventListener / removeEventListener / dispatchEvent surface,
honors `once`, and implements the static abort / timeout / any
factories crawlers use for budgeted fetches.
- StorageEvent class added. localStorage / sessionStorage mutations
now dispatch 'storage' events on the parent window and every iframe
window so cross-frame onstorage listeners fire.
- ProcessingInstruction (nodeType 7) and DocumentType (nodeType 10)
constructors plus document.createProcessingInstruction and
document.implementation.createDocumentType. textContent returns
null for Document and DocumentType per spec.
- insertAdjacentText / insertAdjacentElement on Element. Without
these the testharness output renderer threw in a forEach without
try/catch, breaking the completion callback chain for every test
with subtests.
- Window + Node addEventListener filter non-callable listeners,
accept the EventListenerObject shape (object with handleEvent),
dedupe by (type, callback, capture), honor the `once` option, and
snapshot the listener list before dispatch so a `once: true`
removal mid-iteration doesn't corrupt the loop.
- Iframe HTML loader now extracts head <script> blocks and body
on-handler attributes (`<body onstorage=...>`), evals scripts in
the parent realm after the iframe window is built, re-attaches
handlers as window listeners. Same iframe element reusing its src
no longer double-registers in the iframe registry.
- ops.rs validate_fetch_url honors the OBSCURA_ALLOW_PRIVATE_NETWORK
env var, and the CLI --allow-private-network flag mirrors itself
into that env so the js-side fetch gate matches the http_client
gate. Without this, iframe loads of 127.0.0.1 resources were
blocked even with the flag set.
- ops.rs: create_processing_instruction, create_doctype, pi_target,
doctype_name, doctype_public_id ops added; set_text_content extended
to handle PI nodes.
Picks up @Timtech4u's PR #33, which has been waiting since April. The
design (CLI flag + threading through BrowserContext + http client) is
theirs and is correct; this is a clean rewrite on top of current main
because repeated rebase rounds had been reverting unrelated work that
landed in the meantime.
Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent
SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is
process-wide and clunky in pipelines. Multiple users (@Timtech4u in
the original PR, then @duartemvix and @necrogami in the comment
thread) reported they cannot use obscura for development against
http://localhost:N without it.
Adds --allow-private-network as a global CLI flag, works on fetch,
serve, and scrape. Threads through BrowserContext::with_storage_and_network
and the http client's validate_url. CdpContext gets a new_full
builder and obscura_cdp::start_with_full_serve_options entry point so
'obscura serve --allow-private-network' lets Puppeteer/Playwright
navigate to localhost too. Default remains blocked.
Backward compat: the env var still works (OR'd with the flag inside
validate_url). All existing constructors and entry points unchanged.
Verified end-to-end against a local http.server on 127.0.0.1: fetch
and CDP Page.navigate both blocked by default, both succeed with the
flag. Public URLs unaffected. Integration suite 28/28, full workspace
cargo test all green.
V8 embedded defaults to ~1.4 GB max-old-space on 64-bit. Pages with
heavy fingerprinting or analytics bundles (e.g. demo.fingerprint.com/
web-scraping) blow past it and the worker dies with
'Fatal JavaScript out of memory' and SIGTRAP, taking the CDP session
with it.
Always set --max-old-space-size=4096 (1024 on 32-bit targets) before
V8 startup, matching headless Chrome's default. User-supplied
--v8-flags are appended after the default so they win (V8 parses
left-to-right and the last value wins), which keeps the existing
escape hatch intact for memory-constrained environments.
* feat: cookie jar persistence + storage-dir + filtering fix (clean v4)
* fix: resolve build errors, add storage_dir param throughout server chain
* fix: storage_dir end-to-end, plus two test-literal field additions
Five issues found while testing PR 114 end-to-end.
Build fixes (two test literals out of sync with new struct fields):
- crates/obscura-net/src/cookies.rs: CookieInfo literal in
test_domain_matches_subdomain_without_leading_dot was missing the
same_site and expires fields added in #170.
- crates/obscura-cli/src/main.rs: Command::Fetch literal in
matcher_still_uses_fetch_variant was missing the new storage_dir
field. Without these cargo test --workspace would not compile.
run_fetch was branching between BrowserContext::with_storage (which
hardcodes proxy=None, stealth=false, user_agent=None) and
with_options (which has no storage_dir slot), so --storage-dir +
--proxy + --stealth silently dropped the proxy and stealth flags.
Same bug SGavrl flagged on dispatch.rs in the previous round, just on
the other code path. Switched to with_storage_full which carries all
four. Verified the fetch now errors out against an invalid proxy.
run_fetch --eval short-circuited the function before
context.save_cookies() ran, so cookies set during an eval session
were never persisted to {storage_dir}/cookies.json. Moved the save
above the return.
obscura serve loaded cookies on startup but never saved them
anywhere, so a session that set cookies via Network.setCookie or
navigation dropped them on shutdown. Subscribed to Ctrl-C once in
put a single save_cookies() after the loop so both the Ctrl-C path
and the channel-closed path persist cookies. Verified end-to-end.
---------
Co-authored-by: zhangyang-crazy-one <zhangyang.crazyone@gmail.com>
Co-authored-by: SGavrl <277843151+SGavrl@users.noreply.github.com>
feat: add `--dump assets` to enumerate page sub-resources
Adds a `--dump assets` flag to output all page-referenced sub-resources
(scripts, stylesheets, images, iframes, media, embeds, objects) as NDJSON,
allowing users to easily replay the asset graph.
- Output format: `{"url":"https://example.test/app.js","type":"script"}`
- URLs are resolved against the page origin (matching `--dump links`).
- The `type` tag is derived from the source element. `<link>` elements map
their `rel` attribute to specific types (stylesheet, icon, manifest, preload,
prefetch, modulepreload, dns-prefetch, preconnect, alternate), falling back
to "link" if missing or unknown.
- Adds 7 tests covering URL resolution, rel-mapping, supported selectors,
and empty-attribute skipping.
- Updates README dump-format table and examples.
Two improvements while investigating #147:
1. `obscura --version` / `-V` now print the CARGO_PKG_VERSION instead of
erroring out. The issue reporter explicitly called out that there was
no version flag, and the existing `obscura serve` banner is the only
way to discover the version.
2. New regression test in obscura-js verifies that a TypeError thrown
from one script does not poison the runtime: subsequent scripts and
DOM queries must continue to work. The reporter's symptom was that
offside.js crashing collapsed --dump text output to 1 byte; that
cascade should never happen and this test pins it down.
The classList error itself is offside.js's own bug (it indexes into the
querySelectorAll result without checking for an empty array) and only
fires when the .slideout-navigation element is absent from the DOM.
On current main that element parses correctly and offside.js runs
without error against a captured copy of the affected page.
The previous workaround in page.rs rewrote socks5:// to http:// before
handing the proxy to the stealth client. That works for Clash-style
mixed-mode proxies, but breaks plain SOCKS5 servers like `ssh -ND`,
producing a confusing `TunnelUnexpectedEof` deep in the network layer.
The stealth client is backed by wreq, which does not speak SOCKS5,
so silently substituting schemes can never be correct in general.
Refuse the combination at CLI startup with a clear message that tells
the user what to do (use --proxy http://... or drop --stealth) and
revert the rewrite in Page::new. SOCKS5 without --stealth still works
through the reqwest path.
* feat(cli): add --v8-flags plumbing to configure V8 at startup (#130)
Obscura had no way to pass flags to V8, so users hitting OOM on pages
larger than ~5MB had no escape hatch. Adds a generic passthrough rather
than a one-off heap knob, so future V8 tuning needs no new code.
- New `obscura_js::set_v8_flags` (Once-guarded, empty-string no-op).
- Global `--v8-flags <FLAGS>` arg with `allow_hyphen_values = true`,
applied once in `main` before any isolate is created.
- Usage: `obscura --v8-flags "--max-old-space-size=4096" fetch <url>`.
Closes#130.
* test(cli): expand --v8-flags coverage with normalize helper
Extract normalize_v8_flags() from main() so the trim/empty-check logic
is unit-testable without needing to invoke V8. Adds parse-coverage tests
for --v8-flags with serve/scrape subcommands and empty-string values,
plus pure-function tests for the normalizer.
GHSA-q55h-vfv9-qcr5. Any process able to reach the CDP port could
send Page.navigate {"url":"file:///etc/shadow"} and read arbitrary
local files (the process's UID's worth of read access). When obscura
runs as root inside Docker or as a systemd service that translates
straight into /etc/shadow, /root/.ssh/id_rsa, /proc/1/environ, etc.
Validate at the CDP boundary, not in obscura-net, so direct CLI
callers ("obscura fetch file://...") keep working. A new
BrowserContext.allow_file_access flag defaults to false; Page.navigate
returns an error when the requested URL is file:// and the flag is
off. CdpContext::new_with_security threads the flag from the server
entry point, and "obscura serve --allow-file-access" opts in for
local-HTML testing workflows.
Tested:
- Default obscura serve: CDP Page.navigate {file:///etc/passwd} ->
refused with descriptive error.
- obscura serve --allow-file-access: same call succeeds.
- obscura fetch file://... CLI path: unchanged, still reads the file.
- All earlier PoCs (GHSA-6pj9 JS->file SOP, GHSA-4789 sub-resource
SSRF, GHSA-4789 CSS template injection) still pass.
- cargo test --workspace clean.
Fixes#117. Adds an `Original` variant to `--dump` that streams the raw
HTTP response body verbatim to stdout (or `--output`), bypassing the
browser/JS/DOM stack entirely. Useful for binary payloads (jpg, png,
fonts) and any non-HTML resource where parsing through the DOM layer
would corrupt or discard data.
The new path uses ObscuraHttpClient directly, honours --proxy and
--user-agent, and writes bytes through tokio::io::stdout so binary
payloads (including embedded NUL bytes) round-trip cleanly.
Adds a multi-stage Dockerfile (rust:1-slim-bookworm builder → distroless/cc
runtime) plus a --host flag for obscura serve that defaults to 127.0.0.1
for safety but can be set to 0.0.0.0 so the container's port is reachable
via docker run -p mapping.
Dockerfile cherry-picked from #84. The SPA chunk loading commit from that
PR was left out — verified it didn't actually fix the Next.js webpack
errors on kiwi.com (#86).
Surfaces the existing LP.getMarkdown CDP machinery as a CLI flag so
text-pipeline users (LLM agents, scrapers feeding markdown to other
tools) don't need a separate HTML-to-markdown converter step.
obscura fetch --dump markdown 'https://example.com'
Layout:
- crates/obscura-js/src/markdown.rs holds HTML_TO_MARKDOWN_JS as a
pub const so every layer can share the same DOM-walker. obscura-js
is at the bottom of the dep DAG (obscura-browser -> obscura-js,
obscura-cdp -> obscura-js, obscura-cli -> obscura-js) so a single
source of truth works without a circular dependency.
- crates/obscura-browser/src/lib.rs re-exports it for ergonomics.
- crates/obscura-cdp/src/domains/lp.rs drops its inline duplicate
and calls the shared const.
- crates/obscura-cli/src/main.rs adds the `Markdown` DumpFormat
variant + dump_markdown(&mut Page) which page.evaluate's the
shared script and prints the result.
Net diff: a new 75-line markdown.rs, a 70+ line deletion in lp.rs
that's reborn as 3 lines, and ~10 lines added across the other four
files. No behavior change for `--dump html|text|links`, and
LP.getMarkdown over CDP keeps the exact same JSON shape.
Four new unit tests in obscura-js cover headings, inline emphasis +
links, ordered/unordered lists, and that <script> / <style> contents
do not leak into the output.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
