feat(cdp): bound every CDP command with a shared V8 watchdog (#250)

A hung page (a runaway Runtime.evaluate, a synchronous DOM op) holds the
process-wide V8 lock and wedges every other session forever. The one-shot CLI has
a process-level hard deadline for this; the long-running server cannot force-exit,
so terminate just the offending isolate instead.

dispatch() arms a watchdog on the session page's isolate after taking the V8 lock
and disarms it after the handler; on overrun it terminates the isolate and clears
the termination flag so the session keeps working. OBSCURA_CDP_COMMAND_TIMEOUT_MS
tunes the bound (default 60s, 0 disables); navigation already self-bounds well
under it.

The watchdog is a single long-lived shared thread (cdp_watchdog), not a thread
per command: the V8 lock serializes commands so at most one is ever armed, and
arm/disarm are a mutex plus notify with no measurable per-command cost. Also
fixes a lost-wakeup in the per-call watchdog (spawn_watchdog) where disarming a
fast call before the watchdog thread started blocked for the full budget.

Verified: a hung Runtime.evaluate is terminated at the budget and the session
recovers; CDP integration 37/37, obstacle course 33/33; per-command latency
unchanged (watchdog on equals off within noise).
This commit is contained in:
SGavrl authored and GitHub committed 2026-06-05 21:28:21 +00:00
1 parent 32ddf93885
commit 1a0644f7bc
5 files changed
+255 -34

No files matched your search

+117
View File
@@ -0,0 +1,117 @@
//! Shared per-command V8 watchdog for the CDP server.
//!
//! The CDP dispatcher holds a process-wide V8 lock around every V8-touching
//! command, so at most one command's isolate is ever executing at a time. That
//! lets a single long-lived watchdog thread bound the current command with a
//! deadline, instead of spawning+joining a thread per command (which adds
//! ~240us per command on the hot dispatch path). `arm` and `disarm` are just a
//! mutex + condvar notify, in the low microseconds.
//!
//! Correctness rests on the V8-lock serialization: between `arm` and `disarm`
//! the caller holds the V8 lock, so no other command can arm, and the single
//! slot is exclusively this command's.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Condvar, Mutex, OnceLock};
use std::time::{Duration, Instant};
use crate::runtime::IsolateHandle;
struct Slot {
deadline: Instant,
handle: IsolateHandle,
gen: u64,
fired: Arc<AtomicBool>,
}
struct Shared {
// (current armed slot, monotonic generation counter)
state: Mutex<(Option<Slot>, u64)>,
cv: Condvar,
}
static SHARED: OnceLock<Arc<Shared>> = OnceLock::new();
fn shared() -> &'static Arc<Shared> {
SHARED.get_or_init(|| {
let s = Arc::new(Shared {
state: Mutex::new((None, 0)),
cv: Condvar::new(),
});
let worker = s.clone();
std::thread::Builder::new()
.name("cdp-watchdog".into())
.spawn(move || watchdog_loop(worker))
.expect("spawn cdp watchdog");
s
})
}
fn watchdog_loop(s: Arc<Shared>) {
let mut guard = s.state.lock().unwrap();
loop {
let next_deadline = match &guard.0 {
None => None,
Some(slot) => {
let now = Instant::now();
if slot.deadline <= now {
// Overran: terminate this isolate and clear the slot. The
// dispatcher's disarm will observe `fired` and clear the V8
// termination flag before the next command.
slot.fired.store(true, Ordering::SeqCst);
slot.handle.terminate_execution();
guard.0 = None;
None
} else {
Some(slot.deadline - now)
}
}
};
guard = match next_deadline {
// No armed command: block until arm() notifies. The worker holds the
// lock until it waits, so arm() (which needs the lock) cannot notify
// into the void; no lost wakeup.
None => s.cv.wait(guard).unwrap(),
Some(dur) => s.cv.wait_timeout(guard, dur).unwrap().0,
};
}
}
/// Handle to an armed command; pass to [`disarm`].
pub struct Armed {
gen: u64,
fired: Arc<AtomicBool>,
}
/// Arm the shared watchdog for the current command. If the isolate is still
/// executing `budget` later, it is terminated. O(1), no thread spawn.
pub fn arm(handle: IsolateHandle, budget: Duration) -> Armed {
let s = shared();
let mut guard = s.state.lock().unwrap();
guard.1 += 1;
let gen = guard.1;
let fired = Arc::new(AtomicBool::new(false));
guard.0 = Some(Slot {
deadline: Instant::now() + budget,
handle,
gen,
fired: fired.clone(),
});
s.cv.notify_one();
Armed { gen, fired }
}
/// Disarm the command's watchdog. Returns true if it had already fired
/// (terminated the isolate), in which case the caller must clear the V8
/// termination flag before the next command runs.
pub fn disarm(armed: Armed) -> bool {
let s = shared();
let mut guard = s.state.lock().unwrap();
// Only clear the slot if it is still ours (a newer command would have a
// higher gen, though the V8 lock prevents that overlap in practice).
if guard.0.as_ref().map(|sl| sl.gen) == Some(armed.gen) {
guard.0 = None;
s.cv.notify_one();
}
armed.fired.load(Ordering::SeqCst)
}