The template.content bridge (#463) routed every element through
content_source(), which called get_node().clone() to read the Copy
template_contents field. That deep-cloned each element's Node (its attrs
Vec plus a String per attribute) on every innerHTML/outerHTML/dump-html
serialization, once per non-void element.
template_contents is already destructured from the node in the serialize
loop, so read it there directly. content_source() (inner_html path) now
reads it through with_node() by reference. Serialization output is
byte-identical; ~10% faster on serialization-heavy pages.
on_request/on_response stored callbacks in append-only Vecs with no way
to remove one, so a crawler that installed a response-body capture hook
for one phase could never detach it: it fired for the shared client's
whole lifetime, re-registering across navigations piled up duplicates,
and (because the client is shared per context) it saw sibling pages'
requests too.
Tag each callback with a stable id. on_request/on_response now return
the id and off_request/off_response (remove_on_request/remove_on_response
on the client) detach a specific one. Storage is Vec<(u64, Callback)>;
all fire sites destructure the tuple. Closes#408.
Requests made by page JS (fetch/XHR/dynamic script and stylesheet loads)
run in the V8 op layer and never produced a NetworkEvent, so CDP emitted
Network.requestWillBeSent / responseReceived only for the static
navigation subresources. Puppeteer/Playwright page.on('request') and
on('response') therefore missed every XHR/JSON response, which breaks
response capture (and is the root cause of the Aviasales half of #394).
op_fetch_url now records a JsNetworkEvent keyed by the same fetch-{N} id
under which it already stores the response body. The Page drains these
into its network_events before the CDP layer emits, so they flow through
emit_navigation_events as requestWillBeSent / responseReceived /
loadingFinished alongside the document subresources, and
Network.getResponseBody resolves for them via the shared id. Applied to
both navigation paths (server process_navigation and in-process
do_navigate). The queue is capped at 4096 events per page.
Closes#406.
Runtime.executionContextsCleared on navigation told clients every prior
context was gone, but valid_context_ids was insert-only, so a
Runtime.callFunctionOn targeting a pre-navigation context id still ran
instead of being rejected with Chrome's 'Cannot find context with
specified id'. On a long-lived CDP connection the set also grew by one
id per navigation and was never pruned.
Clear valid_context_ids when executionContextsCleared is emitted, then
re-register the default world (id 2) and the isolated worlds as their
executionContextCreated events are emitted. Closes#407.
A <link rel=stylesheet href> appended via JS (createElement +
appendChild) was never fetched and fired neither load nor error, so any
Promise.all awaiting the link's onload (lazy CSS loaders, antd/bootstrap
fetchers) hung forever and the page stalled at its pre-load stage.
appendChild already drives this for <script src>; mirror it for links:
resolve the href against <base>/document URL, fetch it, and dispatch
load on success or error on failure. CSS is not applied (no layout
engine yet) but the load-event contract matches Chrome.
Read the stylesheet intent from both the rel attribute and the rel
property, since obscura does not yet reflect the rel IDL attribute back
to the content attribute. Closes#409.
Network.getResponseBody returns the whole body in one base64 blob and the
cache drops anything over 2 MiB, so large downloads (images, PDFs,
archives) either failed or forced the long-running server to hold the
full body in memory (issue #360).
Add Fetch.takeResponseBodyAsStream + IO.read + IO.close. The body is moved
out of the page cache into a stream handle and delivered in bounded chunks
via IO.read, then freed on IO.close, so it is held once and released after
the download. Binary bodies are byte-exact (stored base64). To stream a
large body, raise OBSCURA_NETWORK_BODY_BUFFER_BYTES so it is cached.
This is entirely opt-in: nothing runs unless a client calls the new
methods, so the default fetch/render/scrape path is unchanged (verified:
obstacle course 33/33, obscura-vs-Chrome head-to-head within noise of the
pre-change baseline).
Fixes#360.
load_module and load_inline_module drove run_event_loop to full idle
before reading the module's mod_evaluate result. A page timer keeps the
loop busy forever (Vite's HMR / React-Refresh client installs a
setInterval), so the preamble inline module burned the entire budget
waiting for an idle that never came, and the later module that mounts the
app was skipped, leaving #root empty while the production preview rendered
fine (issue #374).
Drive the event loop and the mod_evaluate result together and return as
soon as the module finishes evaluating, so a busy background timer no
longer starves the remaining modules. Post-eval async work is still
drained by the normal post-load settle.
Fixes#374.
The script-execution phase had a 10s soft deadline. On a heavy SPA the
remaining scripts were skipped before the app finished booting, so it
never fired its XHR/fetch calls and page.on('response') captured zero API
responses (issue #361). The knob to extend it, OBSCURA_SCRIPT_DEADLINE_MS,
existed but was undocumented, so it was undiscoverable.
Raise the default to 30s and document the env var. Only pages that keep
running past the deadline are affected; fast pages finish and return well
before it, so the larger budget costs them nothing, and the per-phase
watchdog (armed at deadline + 1s) still bounds a real synchronous hang.
Very heavy SPAs on slow networks can raise it further with
OBSCURA_SCRIPT_DEADLINE_MS, paired with a matching client nav timeout.
Fixes#361.
obscura serve --workers N --proxy ... spawned each worker with the proxy
as a --proxy CLI flag, so the plaintext credential was visible to any
local user via ps / /proc/<pid>/cmdline (issue #366). scrape already
avoids this by passing the proxy in OBSCURA_PROXY, which is only readable
by the owner.
Pass the proxy to serve workers via OBSCURA_PROXY too, and have the serve
command read that env as a fallback when no --proxy flag is given. The
worker argv no longer contains the credential.
Fixes#366.
SGavrl