feat(cdp): bound every CDP command with a shared V8 watchdog (#250)

A hung page (a runaway Runtime.evaluate, a synchronous DOM op) holds the
process-wide V8 lock and wedges every other session forever. The one-shot CLI has
a process-level hard deadline for this; the long-running server cannot force-exit,
so terminate just the offending isolate instead.

dispatch() arms a watchdog on the session page's isolate after taking the V8 lock
and disarms it after the handler; on overrun it terminates the isolate and clears
the termination flag so the session keeps working. OBSCURA_CDP_COMMAND_TIMEOUT_MS
tunes the bound (default 60s, 0 disables); navigation already self-bounds well
under it.

The watchdog is a single long-lived shared thread (cdp_watchdog), not a thread
per command: the V8 lock serializes commands so at most one is ever armed, and
arm/disarm are a mutex plus notify with no measurable per-command cost. Also
fixes a lost-wakeup in the per-call watchdog (spawn_watchdog) where disarming a
fast call before the watchdog thread started blocked for the full budget.

Verified: a hung Runtime.evaluate is terminated at the budget and the session
recovers; CDP integration 37/37, obstacle course 33/33; per-command latency
unchanged (watchdog on equals off within noise).
This commit is contained in:
SGavrl authored and GitHub committed 2026-06-05 21:28:21 +00:00
1 parent 32ddf93885
commit 1a0644f7bc
5 files changed
+255 -34

No files matched your search

+37
View File
@@ -288,6 +288,27 @@ pub async fn dispatch(req: &CdpRequest, ctx: &mut CdpContext) -> CdpResponse {
Some(obscura_js::v8_lock::global().lock().await)
};
// Per-command V8 watchdog. The lock above keeps each handler contiguous on
// the thread, but it does not bound how long a handler runs: a hung page (a
// runaway Runtime.evaluate, a synchronous DOM op) would hold the
// process-wide V8 lock and wedge every other session forever. The one-shot
// CLI uses a process-level hard deadline for this; the long-running server
// cannot force-exit, so we terminate just the offending isolate instead.
// OBSCURA_CDP_COMMAND_TIMEOUT_MS tunes the bound (0 disables); the default
// leaves headroom for the slowest legitimate navigation, which already
// self-bounds via OBSCURA_NAV_TIMEOUT_MS plus the watchdog-bounded settle.
let cmd_budget_ms: u64 = std::env::var("OBSCURA_CDP_COMMAND_TIMEOUT_MS")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(60_000);
let cmd_watchdog = if cmd_budget_ms == 0 || is_v8_free_method(&req.method) {
None
} else {
ctx.get_session_page(&req.session_id)
.and_then(|p| p.isolate_handle())
.map(|h| obscura_js::cdp_watchdog::arm(h, std::time::Duration::from_millis(cmd_budget_ms)))
};
let (domain, method) = match req.method.split_once('.') {
Some((d, m)) => (d, m),
None => {
@@ -324,6 +345,22 @@ pub async fn dispatch(req: &CdpRequest, ctx: &mut CdpContext) -> CdpResponse {
_ => Err(format!("Unknown domain: {}", domain)),
};
// Stop the per-command watchdog. If it fired (the handler held V8 past the
// budget), V8 is left in a terminating state, so clear that flag before the
// next command runs on this page.
if let Some(wd) = cmd_watchdog {
if obscura_js::cdp_watchdog::disarm(wd) {
tracing::warn!(
"CDP command {} held V8 past {}ms; terminated the isolate to free the dispatcher",
req.method,
cmd_budget_ms
);
if let Some(page) = ctx.get_session_page_mut(&req.session_id) {
page.cancel_v8_termination();
}
}
}
drain_binding_calls(ctx);
match result {