fix(cdp): cap glibc malloc arenas so memory does not track peak concurrency
glibc hands each new thread its own 64 MiB arena (up to 8x cores). With one thread per connection that arena is the dominant per-connection memory term, and freed chunks stay reserved in it rather than returning to the OS: resident memory tracks the *peak* number of concurrent connections and never comes back down, which reads as a leak although nothing is leaked and no thread is stuck. Measured with reliability/conn-scale.py in obscura-benchmark. 100 connections each running JS reserve 90.0 GiB of address space uncapped versus 83.5 GiB capped; per idle connection, 65 MiB of reserved address space drops to 2.0 MiB. `main` at that same 100-connection JS workload reserves 83.6 GiB, so with the cap this server is level with it -- most of the total is V8's process-wide sandbox, which `main` pays too the moment it runs any JS. The resident-set effect is the one that bites in production. Measured in the container image against Google Maps, four concurrent connections per round: 350 / 619 / 826 MiB over three rounds uncapped and still climbing linearly, versus 166 / 235 / 269 MiB capped, on a decelerating curve. Two arenas cost no measurable throughput here -- 8 concurrent connections x 12 navigations, 1.53s uncapped against 1.50s capped over three runs. V8 allocates the JS heap through its own allocator and the Rust side is dominated by network I/O rather than malloc traffic. Applied only from `serve`, which owns the process, and skipped when the caller has already set MALLOC_ARENA_MAX. Refs #430.
This commit is contained in:
1 parent
1f5bffc4de
commit
dc79fde1a0
3 files changed
+49
No files matched your search
@@ -20,6 +20,9 @@ futures-util = "0.3"
|
||||
base64 = "0.22"
|
||||
obscura-net = { workspace = true }
|
||||
|
||||
[target.'cfg(target_env = "gnu")'.dependencies]
|
||||
libc = "0.2"
|
||||
|
||||
[dev-dependencies]
|
||||
tokio-tungstenite = { workspace = true }
|
||||
futures-util = "0.3"
|
||||
@@ -241,6 +241,8 @@ pub async fn start_with_full_serve_options(
|
||||
// dropping one runtime here does the one-time setup single-threaded.
|
||||
drop(obscura_js::runtime::ObscuraJsRuntime::new());
|
||||
|
||||
cap_malloc_arenas();
|
||||
|
||||
// Accept loop: hand each WebSocket connection to its own OS thread so its
|
||||
// pages' isolates live on a dedicated thread.
|
||||
loop {
|
||||
@@ -272,6 +274,49 @@ pub async fn start_with_full_serve_options(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Cap the number of per-thread malloc arenas glibc will create.
|
||||
///
|
||||
/// glibc hands each new thread its own 64 MiB arena (up to 8x cores). With one
|
||||
/// thread per connection that is the dominant per-connection memory term:
|
||||
/// measured with `reliability/conn-scale.py`, 100 connections each running JS
|
||||
/// reserve 90.0 GiB of address space uncapped and 83.5 GiB capped, and 100 idle
|
||||
/// connections go from 65 MiB of reserved address space per connection to
|
||||
/// 2.0 MiB.
|
||||
///
|
||||
/// For scale: at the same 100-connection JS workload `main` (one shared
|
||||
/// isolate) reserves 83.6 GiB, so with the cap this server is level with it on
|
||||
/// address space. Most of that total is V8's process-wide sandbox, which `main`
|
||||
/// pays too as soon as it runs any JS at all.
|
||||
///
|
||||
/// The resident-set effect matters more than the reservation: freed chunks stay
|
||||
/// in their arena rather than returning to the OS, so RSS tracks the *peak*
|
||||
/// number of concurrent connections and never comes back down, which reads as a
|
||||
/// leak. Measured in the container image against Google Maps, four concurrent
|
||||
/// connections per round: 350 / 619 / 826 MiB over three rounds uncapped and
|
||||
/// still climbing linearly, versus 166 / 235 / 269 MiB capped, on a
|
||||
/// decelerating curve.
|
||||
///
|
||||
/// Two arenas cost no measurable throughput here (8 concurrent connections x 12
|
||||
/// navigations: 1.53s uncapped, 1.50s capped): V8 allocates the JS heap through
|
||||
/// its own allocator, and the Rust side is dominated by network I/O rather than
|
||||
/// malloc traffic. Only `serve` calls this, and it owns the process. Respects a
|
||||
/// caller-set `MALLOC_ARENA_MAX`.
|
||||
fn cap_malloc_arenas() {
|
||||
#[cfg(target_env = "gnu")]
|
||||
{
|
||||
if std::env::var_os("MALLOC_ARENA_MAX").is_some() {
|
||||
return;
|
||||
}
|
||||
// M_ARENA_MAX is not exported by the libc crate.
|
||||
const M_ARENA_MAX: libc::c_int = -8;
|
||||
// SAFETY: mallopt is thread-safe; called once here before any
|
||||
// connection thread exists.
|
||||
if unsafe { libc::mallopt(M_ARENA_MAX, 2) } != 1 {
|
||||
warn!("mallopt(M_ARENA_MAX) failed; memory will scale with peak concurrency");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Run one WebSocket connection on its own OS thread: a `current_thread` tokio
|
||||
/// runtime + `LocalSet` hosting this connection's `cdp_processor` (with its own
|
||||
/// `CdpContext` and pages) and its frame reader. Confining a connection's pages
|
||||
|
||||
Reference in new issue
Block a user