From 329514302532ac510b6f6a476be5fcc48bd5dc4c Mon Sep 17 00:00:00 2001 From: SGavrl <277843151+SGavrl@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:33:51 +0000 Subject: [PATCH] docs(render): publish automation guide Document screenshots, scrolling, screencasting, PDF, MCP media output, release builds, and evidence-based render verification across the public guides and agent skill. --- AGENTS.md | 105 +++++---- CONTRIBUTING.md | 33 +-- README.md | 85 ++++++-- RENDER.md | 133 +++++------- docs/Adding-a-CDP-method-or-Web-API.md | 4 +- docs/Architecture-overview.md | 17 +- docs/Build-from-source.md | 21 +- docs/CLI-reference.md | 15 +- docs/Configure-stealth-and-proxies.md | 4 +- docs/Connect-Puppeteer-or-Playwright.md | 26 ++- docs/Environment-variables.md | 2 +- docs/Installation.md | 13 +- docs/README.md | 19 +- ...ering-screenshots-screencasting-and-PDF.md | 199 ++++++++++++++++++ docs/SUMMARY.md | 1 + docs/Testing-and-debugging.md | 35 ++- docs/Use-the-MCP-server.md | 21 +- docs/Use-with-Playwright.md | 32 ++- docs/Use-with-Puppeteer.md | 27 ++- skills/obscura/SKILL.md | 156 ++++++-------- 20 files changed, 645 insertions(+), 303 deletions(-) create mode 100644 docs/Rendering-screenshots-screencasting-and-PDF.md diff --git a/AGENTS.md b/AGENTS.md index 91fc1f0..c272a1e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,15 +4,18 @@ Guidance for AI coding agents and contributors working in the Obscura repo. This is the non-obvious stuff you can't infer from the code; read it before building, testing, or changing anything. -Obscura is a headless browser engine in Rust. It runs real JavaScript through -V8 (`deno_core`), keeps a real DOM tree, speaks the Chrome DevTools Protocol, -and is a drop-in replacement for headless Chrome with Puppeteer and Playwright. -It targets web scraping and AI-agent automation. +Obscura is a headless browser engine in Rust. It runs JavaScript through V8 +(`deno_core`), owns its DOM, layout, and paint pipeline, and speaks the Chrome +DevTools Protocol for Puppeteer and Playwright automation. It targets web +scraping and AI-agent workflows without bundling Chromium. ## Build ```bash -cargo build --release # binary at ./target/release/obscura +CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release --features render + +# Rendering plus the optional stealth transport +CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release --features render,stealth ``` - The first build compiles V8 from source: ~5 minutes and a few GB of disk. @@ -20,9 +23,9 @@ cargo build --release # binary at ./target/release/obscura - **Iterating on one crate? Scope it:** `cargo build -p obscura-cli`. A bare `cargo build` can re-link the whole workspace; the V8 compile is the cost, so avoid touching it when you don't need to. -- **Stealth:** `cargo build --release --features stealth` pulls in BoringSSL +- **Stealth:** `--features render,stealth` pulls in BoringSSL (`btls-sys`), which builds through CMake, so `cmake` must be installed. The - default build uses rustls and needs neither CMake nor OpenSSL. + render-only build uses rustls and needs neither CMake nor OpenSSL. - If the vendored OpenSSL build hits an AVX-512 assembler error on your host, build with `OPENSSL_NO_VENDOR=1`. @@ -31,15 +34,16 @@ cargo build --release # binary at ./target/release/obscura Run tests with **`cargo nextest`, not `cargo test`**: ```bash -cargo nextest run --workspace # or: -p while iterating +cargo nextest run --release --features render -p +cargo nextest run --release --features render --no-fail-fast ``` `cargo test` runs the whole test binary in one process, but the engine holds a single V8 isolate per process, so the runtime tests fail under it. `nextest` runs each test in its own process, which is the only supported way. -The authoritative behavioral gate is the **obstacle course** in the companion -repo `obscura-benchmark` (33 capability + speed stages, must stay 33/33): +The obstacle course in the companion `obscura-benchmark` repository is an +additional behavioral gate: ```bash OBSCURA_BIN=./target/release/obscura python3 obstacle-course/run.py --runs 1 --warmup 0 @@ -53,10 +57,12 @@ pass %, not whole-file pass. For any code change: -1. `cargo build --release` (or `-p `) compiles clean. -2. `cargo nextest run` for the crates you touched. -3. The obstacle course still reports **33/33**. -4. For stealth changes, re-test with `--stealth` (a non-stealth binary won't +1. Run focused release-mode nextest coverage for the crates and repro involved. +2. Run `cargo nextest run --release --features render --no-fail-fast`. +3. Run the exact release build shown above. +4. For render changes, run deterministic fixtures and broad top/bottom real-site + captures using the methodology below. +5. For stealth changes, re-test with `--stealth` (a non-stealth binary won't exercise the `wreq` path). Do not bulk-run `cargo fmt`: the tree is not rustfmt-clean, so a blanket format @@ -65,25 +71,59 @@ edit instead. ## Architecture -- **obscura-cli** — CLI: `fetch` (`--dump assets|html|text|links|markdown|original|cookies`, `--eval `), `serve` (CDP server), `scrape`, `mcp`. `--proxy`, `--stealth`, and `--allow-private-network` are global flags: valid before or after the subcommand and applied to `fetch`, `serve`, `scrape`, and `mcp` (a `scrape` run forwards `--stealth` to each worker via `OBSCURA_STEALTH`). +- **obscura-cli** — CLI: `fetch` (`--dump assets|html|text|links|markdown|original|cookies`, `--eval `, `--screenshot `), `serve` (CDP server), `scrape`, `mcp`. `--proxy`, `--stealth`, and `--allow-private-network` are global flags: valid before or after the subcommand and applied to `fetch`, `serve`, `scrape`, and `mcp` (a `scrape` run forwards `--stealth` to each worker via `OBSCURA_STEALTH`). - **obscura-cdp** — Chrome DevTools Protocol server (WebSocket). Sessions are `"{targetId}-session"`. - **obscura-js** — V8/`deno_core` runtime. `js/bootstrap.js` is the DOM/browser shim; `src/ops.rs` bridges JS to Rust DOM ops; `src/runtime.rs` owns the isolate and the per-page `ObscuraState`. - **obscura-dom** — DOM tree (`src/tree.rs`). - **obscura-net** — HTTP client (`client.rs`), stealth client (`wreq_client.rs`), cookie jar, robots cache, tracker blocklist. - **obscura-browser** — the `Page` type, navigation, JS evaluation. +- **obscura-render** — selector cascade, computed style, retained layout, + scrolling, text shaping, images/SVG/canvas, and CPU-backed paint. The + `render` feature powers geometry, screenshots, CDP screencasting, and PDF. +- **obscura-mcp** — stateful MCP automation tools. Render builds expose + `browser_screenshot` and `browser_pdf`; streaming screencasts remain CDP-only. - **obscura** — embeddable Rust library API (git dependency; builds V8 locally, not on crates.io). Public request-interception API on `Page`: `add_preload_script`, `enable_interception` (channel of `InterceptedRequest`, resolved with `InterceptResolution::{Continue, Fulfill, Fail}`), and passive `on_request` / `on_response`. `op_fetch_url` invokes these for JS `fetch()`/XHR, so when touching it keep a `Continue` URL rewrite behind `validate_fetch_url` (the SSRF gate, same as redirects). ## Conventions -- **Performance is a hard constraint** (Obscura is ~12x faster and uses ~6x less - memory than headless Chrome on framework pages). Keep native Rust fast paths; - add a JS fallback only for real spec edge cases, and benchmark old-vs-new - interleaved, min-of-N (noise floor is about +-10%). +- **Performance is a hard constraint.** Keep native Rust fast paths and avoid + repeated full-tree work. Benchmark old and new revisions interleaved with the + same release build, page, network, viewport, settle policy, and capture path. + Report distributions and resource use, not a single favorable run. - **Keep ops panic-safe.** `op_dom` is wrapped in `catch_unwind` so a DOM-op panic returns null instead of aborting the process inside V8's FFI frame. New ops must not unwind into V8. - **Commits/PRs/comments:** short and factual, no em dashes, no AI filler. +## Rendering verification + +Use deterministic fixtures before real sites. Put generated output in a +disposable directory outside the repository: + +```bash +RUN_ROOT="$(mktemp -d)" +OBSCURA_BIN=./target/release/obscura render-repros/run.sh "$RUN_ROOT/fixtures" +OBSCURA_BIN=./target/release/obscura render-repros/representative-suite/run.sh "$RUN_ROOT/top" +OBSCURA_BIN=./target/release/obscura render-repros/representative-suite/run.sh "$RUN_ROOT/bottom" bottom +``` + +The harness accepts `BASELINE_BIN` or `CHROMIUM_BIN` for paired output. A +latency-only run may use `SUITE_MODE=latency SETTLE_MS=0`, but zero settle is +not valid fidelity evidence. + +Compare the same viewport, device scale, identity, network inputs, settle +policy, scroll position, animation time, and capture boundary. First confirm +both navigations succeeded and both images are nonblank. Then inspect missing +resources, geometry, text flow, structural edges, clipping, fixed/sticky +behavior, and a reduced fixture. Pixel-distance metrics are useful regression +tripwires, not standalone correctness verdicts. Never add hostname-specific +layout, style, or resource behavior. + +`render-repros/**` is the tracked public evidence harness. Git-ignored internal +handover notes are private working material: do not edit them, link them from +public documentation, stage them, or commit them. Do not commit generated +screenshots or reports. + ## Gotchas - **DOM mutation arg order:** `insertBefore` / `replaceChild` in `bootstrap.js` @@ -122,30 +162,3 @@ adjustments) are privacy-first anti-fingerprinting: they present a normal, consistent browser fingerprint (user agent, timezone, navigator properties, and similar surfaces) so ordinary automation traffic is not singled out. They contain no bot or automation-abuse payload. - -## Recommended proxy provider: NodeMaven - -For production scraping you usually want residential or ISP IPs rather than a -datacenter address, since many sites block datacenter ranges. We use -**NodeMaven** and it works well with Obscura: residential/ISP proxies, sticky -sessions, and per-request country/region targeting. A good default if you don't -already have a provider. - -Route Obscura through it with the global `--proxy` flag (use your own -credentials): - -```bash -# HTTP -obscura --proxy http://USER:PASS@gate.nodemaven.com:8080 fetch https://example.com --dump text - -# SOCKS5 -obscura --proxy socks5://USER:PASS@gate.nodemaven.com:1080 scrape url1 url2 --concurrency 25 -``` - -The targeting options (country, region, session id) live in the proxy username, -so a sticky session keeps the same exit IP across requests. - -Sign up: https://go.nodemaven.com/obscuraagent - -Obscura discount codes: `OBSCURA35` (35% off mobile and residential), -`OBSCURA40` (40% off ISP / static). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bb73cfb..5f6ea28 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -30,14 +30,14 @@ A few notes to keep the project maintainable: ## Building ```bash -cargo build --release # binary at ./target/release/obscura +cargo build --release --features render # binary at ./target/release/obscura ``` - The first build compiles V8 from source: roughly 5 minutes and a few GB of disk. Incremental builds are seconds. - Iterating on one crate? Scope it: `cargo build -p obscura-cli`. -- **Stealth** (`--features stealth`) pulls in BoringSSL through CMake, so `cmake` - must be installed. The default build uses rustls and needs neither CMake nor +- **Stealth** (`--features render,stealth`) pulls in BoringSSL through CMake, so `cmake` + must be installed. The render-only build uses rustls and needs neither CMake nor OpenSSL. - If the vendored OpenSSL build hits an AVX-512 assembler error on your host, build with `OPENSSL_NO_VENDOR=1`. @@ -47,7 +47,8 @@ cargo build --release # binary at ./target/release/obscura Run tests with **`cargo nextest`, not `cargo test`**: ```bash -cargo nextest run --workspace # or -p while iterating +cargo nextest run --release --features render -p +cargo nextest run --release --features render --no-fail-fast ``` `cargo test` runs the whole test binary in one process, but the engine holds a @@ -68,15 +69,21 @@ It serves local fixtures, so it is deterministic and offline. For any code change: -1. `cargo build --release` (or `-p `) compiles clean. -2. `cargo nextest run` passes for the crates you touched. -3. The obstacle course still reports **33/33**. -4. **Performance is a hard constraint.** Obscura is roughly 12x faster and uses - about 6x less memory than headless Chrome on framework pages. Keep the native - Rust fast paths and add a JS fallback only for real spec edge cases. If your - change could affect performance, benchmark old vs new interleaved, min-of-N - (the noise floor is about plus or minus 10%). -5. For stealth changes, re-test with `--stealth`. A non-stealth binary does not +1. `cargo nextest run --release --features render` passes for the crates you touched. +2. The full render-feature nextest command above passes. +3. `CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release --features render` compiles clean. +4. The obstacle course still reports **33/33**. +5. **Performance is a hard constraint.** Keep native Rust fast paths and avoid + repeated full-tree work. If your change could affect performance, benchmark + old and new revisions interleaved with identical release builds, fixtures, + networks, viewport, settle policy, and capture path. Report a distribution + and memory use rather than a single favorable run. +6. For rendering changes, run the deterministic `render-repros` fixtures and + the representative real-site suite at both the top and bottom of pages. + Confirm resource and navigation success before interpreting image diffs; + pixel error is a tripwire, not a verdict. Never add hostname-specific layout + or style behavior. +7. For stealth changes, re-test with `--stealth`. A non-stealth binary does not exercise the `wreq` path. Keep ops panic-safe: a panic in an op must degrade to a null result, never diff --git a/README.md b/README.md index bcce3f6..e9994b0 100644 --- a/README.md +++ b/README.md @@ -18,21 +18,24 @@ --- -Obscura is a headless browser engine written in Rust, built for web scraping and AI agent automation. It runs real JavaScript via V8, supports the Chrome DevTools Protocol, and acts as a drop-in replacement for headless Chrome with Puppeteer and Playwright. +Obscura is a headless browser engine written in Rust for web scraping and AI-agent automation. It runs JavaScript through V8, owns its DOM, layout, and paint pipeline, and exposes Chrome DevTools Protocol (CDP) compatibility for common Puppeteer and Playwright workflows. ### Why Obscura over headless Chrome? Designed for automation at scale, not desktop browsing. -| Metric | Obscura | Headless Chrome | -|--------------|--------------|------------------| -| Memory | **30 MB** | 200+ MB | -| Binary size | **70 MB** | 300+ MB | -| Anti-detect | **Built-in** | None | -| Page load | **85 ms** | ~500 ms | -| Startup | **Instant** | ~2s | -| Puppeteer | **Yes** | Yes | -| Playwright | **Yes** | Yes | +| Capability | Obscura | Headless Chrome | +|------------|---------|-----------------| +| JavaScript and DOM | V8 + native Rust DOM | V8 + Blink | +| Puppeteer / Playwright | CDP compatibility | Native | +| Screenshots | PNG, JPEG, lossless WebP | Yes | +| Screencasting | Activity-driven CDP frames | Yes | +| PDF | Raster-backed print output | Vector/text-aware output | +| Deployment | Single-purpose browser binary | Full browser distribution | + +Performance varies substantially by page, hardware, network, build features, +and capture settings. Use the published benchmark suite, or measure your own +workload with identical inputs, before making capacity decisions. ## 🎉 10,000 stars and what's next @@ -176,7 +179,7 @@ No Chrome, no Node.js, no dependencies. Release archives include both `obscura` and `obscura-worker`; keep them in the same directory for the parallel `scrape` command. -Default archive names contain the lean binary. Archives ending in +Standard archive names contain the render-enabled rustls binary. Archives ending in `-stealth.tar.gz` or `-stealth.zip` additionally include the wreq/BoringSSL transport used for TLS impersonation. @@ -189,17 +192,17 @@ usable on common LTS servers with glibc 2.35+. docker run -d --name obscura -p 127.0.0.1:9222:9222 h4ckf0r0day/obscura ``` -Image on [Docker Hub](https://hub.docker.com/r/h4ckf0r0day/obscura). Multi-stage build on `distroless/cc`, no shell, no package manager, ~57 MB compressed. +Image on [Docker Hub](https://hub.docker.com/r/h4ckf0r0day/obscura). Multi-stage build on `distroless/cc`, with no shell or package manager in the runtime image. ### Build from source ```bash git clone https://github.com/h4ckf0r0day/obscura.git cd obscura -cargo build --release +cargo build --release --features render -# With stealth mode (anti-detection + tracker blocking) -cargo build --release --features stealth +# Rendering plus the optional stealth transport +cargo build --release --features render,stealth ``` Requires Rust 1.75+ ([rustup.rs](https://rustup.rs)). First build takes ~5 min (V8 compiles from source, cached after). @@ -210,7 +213,7 @@ CMake, Clang, and the libclang/LLVM development libraries. On Ubuntu/Debian: sudo apt-get install build-essential cmake clang libclang-dev llvm-dev ``` -The default build does not require these additional stealth dependencies. +The render-only build does not require these additional stealth dependencies. ## Quick Start @@ -244,8 +247,33 @@ obscura fetch https://example.com --wait-until networkidle0 # Bound navigation time for slow or broken pages obscura fetch https://example.com --timeout 10 + +# Capture the settled page as PNG +obscura fetch https://example.com --screenshot page.png ``` +## Rendering and capture + +Official release archives and the Docker image include the rendering engine. +It provides modern CSS layout and paint, viewport and full-page screenshots, +scroll-aware fixed and sticky geometry, activity-driven CDP screencasting, and +raster PDF export without starting Chromium. + +```javascript +await page.setViewport({ width: 1440, height: 1000 }); +await page.goto('https://example.com', { waitUntil: 'load' }); +await page.screenshot({ path: 'page.png', fullPage: true }); +await page.pdf({ path: 'page.pdf', format: 'A4', printBackground: true }); +``` + +The engine covers the common block, inline, flex, grid, table, float, +positioning, overflow, transform, text, image, SVG, canvas, background, border, +and animation paths used by modern sites. It remains an evolving independent +engine: long-tail CSS, some Web APIs, media playback, compositor effects, and +platform font rasterization may differ from Chromium. See the +[rendering guide](docs/Rendering-screenshots-screencasting-and-PDF.md) for +scrolling, screencasting, raw CDP, MCP, PDF limits, and comparison methodology. + ### Start the CDP server ```bash @@ -331,7 +359,7 @@ await page.evaluate(() => { ## Benchmarks -Page load: +Historical page-load samples from the benchmark repository: | Page | Obscura | Chrome | |------|---------|--------| @@ -339,11 +367,15 @@ Page load: | JS + XHR + fetch | **84 ms** | ~800 ms | | Dynamic scripts | **78 ms** | ~700 ms | -The full benchmark suite (WPT conformance, obstacle course, real-world corpus, and vs-Chrome speed) lives in a separate repo: https://github.com/h4ckf0r0day/obscura-benchmark +These figures are not universal performance guarantees. The full benchmark +suite (WPT conformance, obstacle course, real-world corpus, and comparative +speed tests) lives in a separate repo: +https://github.com/h4ckf0r0day/obscura-benchmark ## Stealth Mode -Enable with `--features stealth`. +Build with `--features render,stealth`, then enable the behavior at runtime with +the global `--stealth` flag. ### Anti-fingerprinting - Per-session fingerprint randomization (GPU, screen, canvas, audio, battery) @@ -366,7 +398,7 @@ Obscura implements the Chrome DevTools Protocol for Puppeteer/Playwright compati | Domain | Methods | |--------|---------| | **Target** | createTarget, closeTarget, attachToTarget, createBrowserContext, disposeBrowserContext | -| **Page** | navigate, getFrameTree, addScriptToEvaluateOnNewDocument, lifecycleEvents | +| **Page** | navigate, getFrameTree, lifecycleEvents, captureScreenshot, start/stopScreencast, printToPDF | | **Runtime** | evaluate, callFunctionOn, getProperties, addBinding | | **DOM** | getDocument, querySelector, querySelectorAll, getOuterHTML, resolveNode | | **Network** | enable, setCookies, getCookies, setExtraHTTPHeaders, setUserAgentOverride | @@ -417,7 +449,9 @@ Fetch and render a single page. | `--eval` | — | JavaScript expression to evaluate | | `--wait-until` | `load` | Wait: `load`, `domcontentloaded`, `networkidle0` | | `--timeout` | `30` | Maximum navigation time in seconds | +| `--wait` | adaptive, up to `5` | Post-load settling; an explicit value is a fixed delay in seconds | | `--selector` | — | Wait for CSS selector | +| `--screenshot` | — | Write a PNG screenshot (single URL; render-enabled build) | | `--stealth` | off | Anti-detection mode | | `--output` | — | Write dump or eval output to a file | | `--quiet` | off | Suppress banner | @@ -480,9 +514,11 @@ Optional flags (both transports): | Tool | Description | |------|-------------| | `browser_navigate` | Navigate to a URL (`url`, optional `waitUntil`: `load` / `domcontentloaded` / `networkidle0`) | -| `browser_snapshot` | Return the current page URL, title, and body text | -| `browser_click` | Click an element by CSS selector | -| `browser_fill` | Set an input value (triggers `input` + `change` events) | +| `browser_snapshot` | Return the current page URL, title, readable body text, and element references | +| `browser_screenshot` | Return the current page as an MCP PNG image (render-enabled build) | +| `browser_pdf` | Return the current page as an embedded PDF resource (render-enabled build) | +| `browser_click` | Click by current snapshot reference or CSS selector | +| `browser_fill` | Set an input value by reference or selector (triggers `input` + `change`) | | `browser_type` | Append text to an input | | `browser_press_key` | Dispatch a keyboard event (`key`, optional `selector`) | | `browser_select_option` | Select an `