feat(cdp): stream large response bodies via takeResponseBodyAsStream + IO (#381)
Network.getResponseBody returns the whole body in one base64 blob and the cache drops anything over 2 MiB, so large downloads (images, PDFs, archives) either failed or forced the long-running server to hold the full body in memory (issue #360). Add Fetch.takeResponseBodyAsStream + IO.read + IO.close. The body is moved out of the page cache into a stream handle and delivered in bounded chunks via IO.read, then freed on IO.close, so it is held once and released after the download. Binary bodies are byte-exact (stored base64). To stream a large body, raise OBSCURA_NETWORK_BODY_BUFFER_BYTES so it is cached. This is entirely opt-in: nothing runs unless a client calls the new methods, so the default fetch/render/scrape path is unchanged (verified: obstacle course 33/33, obscura-vs-Chrome head-to-head within noise of the pre-change baseline). Fixes #360.
This commit is contained in:
1 parent
d3a8b9a482
commit
1ca6047351
8 files changed
+123
-1
No files matched your search
@@ -1482,6 +1482,31 @@ impl Page {
|
||||
})
|
||||
}
|
||||
|
||||
/// Take a stored response body as raw bytes for CDP streaming
|
||||
/// (Fetch.takeResponseBodyAsStream). Removes it from the in-memory cache and
|
||||
/// transfers ownership to the caller, so a large body is held once and freed
|
||||
/// when the stream is closed rather than lingering in this long-running
|
||||
/// process (issue #360). Binary bodies are stored base64 (byte-exact); text
|
||||
/// bodies return their UTF-8 bytes. Returns None if the body was never
|
||||
/// cached (e.g. it exceeded OBSCURA_NETWORK_BODY_BUFFER_BYTES and was
|
||||
/// dropped) or the id is unknown.
|
||||
pub fn take_response_body_raw(&mut self, request_id: &str) -> Option<Vec<u8>> {
|
||||
let stored = if let Some(body) = self.response_bodies.remove(request_id) {
|
||||
self.response_body_order.retain(|id| id != request_id);
|
||||
body
|
||||
} else {
|
||||
self.js.as_ref()?.get_network_response_body(request_id).map(|b| StoredResponseBody {
|
||||
body: b.body,
|
||||
base64_encoded: b.base64_encoded,
|
||||
})?
|
||||
};
|
||||
if stored.base64_encoded {
|
||||
BASE64.decode(stored.body.as_bytes()).ok()
|
||||
} else {
|
||||
Some(stored.body.into_bytes())
|
||||
}
|
||||
}
|
||||
|
||||
/// Make the body stored under `from_id` also retrievable under `to_id`.
|
||||
/// The main navigation resource is stored under its internal request id, but
|
||||
/// the CDP layer reports it to clients with the navigation's loaderId as the
|
||||
|
||||
Reference in new issue
Block a user