feat(cdp): stream large response bodies via takeResponseBodyAsStream + IO (#381)

Network.getResponseBody returns the whole body in one base64 blob and the
cache drops anything over 2 MiB, so large downloads (images, PDFs,
archives) either failed or forced the long-running server to hold the
full body in memory (issue #360).

Add Fetch.takeResponseBodyAsStream + IO.read + IO.close. The body is moved
out of the page cache into a stream handle and delivered in bounded chunks
via IO.read, then freed on IO.close, so it is held once and released after
the download. Binary bodies are byte-exact (stored base64). To stream a
large body, raise OBSCURA_NETWORK_BODY_BUFFER_BYTES so it is cached.

This is entirely opt-in: nothing runs unless a client calls the new
methods, so the default fetch/render/scrape path is unchanged (verified:
obstacle course 33/33, obscura-vs-Chrome head-to-head within noise of the
pre-change baseline).

Fixes #360.
This commit is contained in:
SGavrl authored and GitHub committed 2026-07-10 20:38:28 +00:00
1 parent d3a8b9a482
commit 1ca6047351
8 files changed
+123 -1

No files matched your search

+25
View File
@@ -1482,6 +1482,31 @@ impl Page {
})
}
/// Take a stored response body as raw bytes for CDP streaming
/// (Fetch.takeResponseBodyAsStream). Removes it from the in-memory cache and
/// transfers ownership to the caller, so a large body is held once and freed
/// when the stream is closed rather than lingering in this long-running
/// process (issue #360). Binary bodies are stored base64 (byte-exact); text
/// bodies return their UTF-8 bytes. Returns None if the body was never
/// cached (e.g. it exceeded OBSCURA_NETWORK_BODY_BUFFER_BYTES and was
/// dropped) or the id is unknown.
pub fn take_response_body_raw(&mut self, request_id: &str) -> Option<Vec<u8>> {
let stored = if let Some(body) = self.response_bodies.remove(request_id) {
self.response_body_order.retain(|id| id != request_id);
body
} else {
self.js.as_ref()?.get_network_response_body(request_id).map(|b| StoredResponseBody {
body: b.body,
base64_encoded: b.base64_encoded,
})?
};
if stored.base64_encoded {
BASE64.decode(stored.body.as_bytes()).ok()
} else {
Some(stored.body.into_bytes())
}
}
/// Make the body stored under `from_id` also retrievable under `to_id`.
/// The main navigation resource is stored under its internal request id, but
/// the CDP layer reports it to clients with the navigation's loaderId as the