SGavrl
4e500a2aad
- --stealth is now a global flag: document it on every subcommand including scrape, and that it works before or after the subcommand. - Correct the stealth description: it presents a consistent Chrome fingerprint (TLS, headers, JS surfaces), not a randomized one, and scripted fetch()/XHR share that fingerprint. - Add --dump cookies to the CLI reference and the extract-data guide. - Note --dump assets now also lists fetch()/XHR request URLs.
111 lines
5.5 KiB
Markdown
111 lines
5.5 KiB
Markdown
Obscura is a workspace of eight crates.
|
|
|
|
```
|
|
obscura-cli CLI entry point. fetch, serve, scrape, mcp.
|
|
obscura-cdp Chrome DevTools Protocol server. WebSocket, dispatch, domain handlers.
|
|
obscura-browser Page type, navigation, lifecycle events.
|
|
obscura-js V8 runtime via deno_core. bootstrap.js + Rust ops.
|
|
obscura-dom DOM tree implementation.
|
|
obscura-net HTTP client, stealth client, cookie jar, robots cache, tracker blocklist.
|
|
obscura-mcp Model Context Protocol server.
|
|
obscura Embeddable Rust library API (Browser, Page, Element, CookieStore).
|
|
```
|
|
|
|
## Request flow
|
|
|
|
A `Page.navigate` from a CDP client:
|
|
|
|
```
|
|
CDP client (Puppeteer)
|
|
│ WebSocket frame
|
|
▼
|
|
obscura-cdp/server.rs accept, route by sessionId
|
|
│
|
|
▼
|
|
obscura-cdp/dispatch.rs method router, acquires v8_lock
|
|
│
|
|
▼
|
|
obscura-cdp/domains/page.rs Page.navigate handler
|
|
│
|
|
▼
|
|
obscura-browser/page.rs navigate_with_wait
|
|
│
|
|
├──► obscura-net/client.rs HTTP fetch
|
|
│
|
|
├──► obscura-dom/tree.rs parse HTML into the tree
|
|
│
|
|
└──► obscura-js/runtime.rs run inline scripts
|
|
│
|
|
└──► bootstrap.js + ops.rs DOM bindings
|
|
```
|
|
|
|
The dispatcher emits CDP events (`Network.requestWillBeSent`, `Page.frameNavigated`, `Page.lifecycleEvent`) back to the client through the same WebSocket.
|
|
|
|
## Single V8 isolate
|
|
|
|
All pages in a process share one V8 isolate. The isolate is single-threaded by design.
|
|
|
|
`obscura_js::v8_lock::global()` is a `tokio::sync::Mutex` that serializes V8 work. A handler that wants to run JS must acquire the lock first:
|
|
|
|
```rust
|
|
let _guard = obscura_js::v8_lock::global().lock().await;
|
|
page.evaluate(expr).await
|
|
```
|
|
|
|
The dispatcher routes long-running operations (navigation, eval) through `process_with_interception` in `server.rs`, which spawns the work onto the tokio `LocalSet` and releases the dispatcher to keep handling other CDP messages.
|
|
|
|
This is why `Target.createTarget` from many concurrent clients works: each `newPage` returns immediately while the actual navigation runs in a spawned task.
|
|
|
|
## Robustness
|
|
|
|
One page cannot hang or crash the process. `obscura-js/runtime.rs` provides a V8 termination watchdog (`arm_watchdog`, `run_event_loop_bounded`) that terminates the isolate from a separate thread when synchronous work overruns a budget, because `tokio::time::timeout` cannot preempt synchronous V8. It bounds the post-load settle, the navigation event-loop pumps, and `--eval`. `obscura-js/cdp_watchdog.rs` is a single shared watchdog the dispatcher arms around every CDP command, so a runaway page cannot hold the V8 lock and wedge other sessions (tunable via `OBSCURA_CDP_COMMAND_TIMEOUT_MS`). `op_dom` is wrapped in `catch_unwind` so a DOM-op panic degrades to a null result instead of aborting the process through V8's FFI frame, and `obscura-dom/tree.rs` rejects cyclic reparenting that would make tree walks loop forever. Scripted `fetch()`/XHR and module loads are timeout-bounded (`OBSCURA_FETCH_TIMEOUT_MS`), and the one-shot `fetch` CLI has a process-level hard deadline as a final backstop.
|
|
|
|
## JS bridge
|
|
|
|
`obscura-js/js/bootstrap.js` provides the browser globals: `document`, `window`, `navigator`, `location`, observers, fetch, indexedDB, etc.
|
|
|
|
`obscura-js/src/ops.rs` registers Rust ops that the bootstrap calls into:
|
|
|
|
```js
|
|
Deno.core.ops.op_dom('insert_before', parentNid, refNid, newNid);
|
|
```
|
|
|
|
Adding a Web API usually means:
|
|
|
|
1. JS shim in `bootstrap.js` that exposes the API surface.
|
|
2. Rust op in `ops.rs` that performs the side effect (DOM mutation, fetch, crypto).
|
|
3. Register the op in `build_extension()`.
|
|
|
|
Worked example: [Adding a CDP method or Web API](Adding-a-CDP-method-or-Web-API.md).
|
|
|
|
## CDP session model
|
|
|
|
Each CDP client connection gets attached to one or more targets.
|
|
Session IDs are `"{targetId}-session"`. The dispatcher routes by `sessionId` in the incoming frame to the right `Page`.
|
|
|
|
Targets are created by `Target.createTarget`. Closing the WebSocket detaches all sessions but leaves the pages running.
|
|
|
|
## Lifecycle
|
|
|
|
Lifecycle events are emitted by `obscura-browser/lifecycle.rs` as the page transitions:
|
|
|
|
```
|
|
init → commit → domcontentloaded → load → networkidle2 → networkidle0
|
|
```
|
|
|
|
`waitUntil` on `Page.navigate` blocks until the requested level is reached. The Puppeteer / Playwright `goto` resolves on the matching `Page.lifecycleEvent` client-side.
|
|
|
|
## Storage
|
|
|
|
`--storage-dir` persists cookies (`cookies.json`) and localStorage (`localStorage/<origin>.json`). Reads on process start, writes on every navigation and on graceful shutdown.
|
|
|
|
## Stealth
|
|
|
|
`--stealth` swaps the default `reqwest` client for `obscura-net/wreq_client.rs`, which presents a real browser's TLS ClientHello, ALPN, and cipher order (a consistent Chrome fingerprint, not a randomized one) so the TLS layer matches the User-Agent and JS surfaces. It also applies the bundled tracker blocklist before any request leaves the process. Scripted `fetch()`/XHR go through the same stealth client, so subresource requests carry the same fingerprint as the navigation. `--stealth` is a global CLI flag that applies to `fetch`, `serve`, `scrape`, and `mcp`.
|
|
|
|
## Workspace conventions
|
|
|
|
- One crate per layer. Cross-crate calls go through the layer above, not sideways.
|
|
- All async is `tokio` with a `LocalSet` because V8 is `!Send`.
|
|
- All DOM ops go through `op_dom` to keep the JS/Rust boundary narrow.
|