* docs(skills): add agent skill for Obscura
Adds skills/obscura/SKILL.md following the agent-skill convention used
by addyosmani/agent-skills and vercel-labs/agent-browser. AI coding
agents that scan a repo for SKILL.md files will discover Obscura and
know when to reach for it instead of Chrome.
Covers: build (with and without stealth), CLI fetch, CDP server with
Puppeteer/Playwright examples, localhost / private-network opt-in,
scaling profile, and known limits.
* docs(skills): address review feedback
- Remove --allow-private-network and OBSCURA_ALLOW_PRIVATE_NETWORK examples (not on main)
- Drop "screenshot" trigger word; Obscura doesn't support screenshots (#37, #123)
- Reword Known Limits: screenshots are not implemented, not low-fidelity
Surfaces the existing LP.getMarkdown CDP machinery as a CLI flag so
text-pipeline users (LLM agents, scrapers feeding markdown to other
tools) don't need a separate HTML-to-markdown converter step.
obscura fetch --dump markdown 'https://example.com'
Layout:
- crates/obscura-js/src/markdown.rs holds HTML_TO_MARKDOWN_JS as a
pub const so every layer can share the same DOM-walker. obscura-js
is at the bottom of the dep DAG (obscura-browser -> obscura-js,
obscura-cdp -> obscura-js, obscura-cli -> obscura-js) so a single
source of truth works without a circular dependency.
- crates/obscura-browser/src/lib.rs re-exports it for ergonomics.
- crates/obscura-cdp/src/domains/lp.rs drops its inline duplicate
and calls the shared const.
- crates/obscura-cli/src/main.rs adds the `Markdown` DumpFormat
variant + dump_markdown(&mut Page) which page.evaluate's the
shared script and prints the result.
Net diff: a new 75-line markdown.rs, a 70+ line deletion in lp.rs
that's reborn as 3 lines, and ~10 lines added across the other four
files. No behavior change for `--dump html|text|links`, and
LP.getMarkdown over CDP keeps the exact same JSON shape.
Four new unit tests in obscura-js cover headings, inline emphasis +
links, ordered/unordered lists, and that <script> / <style> contents
do not leak into the output.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
headless_chrome (and older Puppeteer) wrap every CDP call inside
Target.sendMessageToTarget. Obscura was returning "Unknown Target
method" for the wrapper, which broke navigation through the high-level
client API even though the WebSocket itself connected.
Add a wrapper-level handler that:
1. Parses the inner JSON-encoded CDP request from params.message
2. Recursively dispatches it (boxed future to sidestep async-fn
recursion limits) using the wrapper's sessionId so routing works
3. Re-emits the inner response as a Target.receivedMessageFromTarget
event — the legacy contract these clients listen on instead of
correlating responses by id
4. Returns {} for the wrapper itself
This addresses the headless_chrome leg of #26. The chromiumoxide
"data did not match any variant of untagged enum Message" decode
failure is a separate event/response shape issue — needs a focused
investigation against a chromiumoxide repro that's not in the issue,
so it's left out of this PR.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
macOS 26 moved libc++ headers inside the SDK, breaking Apple clang 17's
default header search. The boring-sys cmake step fails with
'fatal error: memory file not found' when building --features stealth.
Adding CXXFLAGS and SDKROOT to .cargo/config.toml [env] propagates them
to all build scripts (Cargo's documented mechanism), so cmake picks up the
SDK's C++ include path without any manual env-var ceremony.
force=false preserves CI overrides and Xcode users who already have
SDKROOT set. Non-macOS platforms ignore SDKROOT; clang silently skips
-isystem paths that do not exist on disk.
Fixes#136
When --stealth is enabled, StealthHttpClient was created without
the proxy URL from --proxy, causing all stealth-mode requests to
bypass the proxy.
Also convert SOCKS5 proxy URL to HTTP when passing to wreq-based
StealthHttpClient, because wreq does not support SOCKS5 protocol
(Clash mixed port supports both protocols on the same port).
Fixes proxy connectivity for 'obscura serve --stealth --proxy socks5://...'
Co-authored-by: handsome.cheng <handsome.cheng@temu.com>
* fix(#19): serialize V8 work across pages with process-wide async lock
Concurrent CDP clients (≥5) trip V8's
`heap->isolate() == Isolate::TryGetCurrent()` invariant and abort the
process via V8_Fatal — no Rust panic, just abort(3). Each Page owns its
own JsRuntime/Isolate and they all run on a single OS thread via
LocalSet+spawn_local; when two pages' V8-touching futures interleave
across an .await, V8 trips the check.
This adds a process-wide tokio::sync::Mutex (obscura_js::v8_lock) that
is acquired:
- At the top of obscura_cdp::dispatch::dispatch (covers every CDP
handler that may call into a JsRuntime).
- In server.rs's process_with_interception spawn_local that runs
page.navigate_with_wait + execute_preload_script (this path
bypasses dispatch).
The guard keeps each V8-touching block contiguous on the thread: V8
fully exits one Isolate before another page is allowed in. Trade-off:
non-V8 handlers (Browser.*, Storage.*, Emulation.*) also serialize,
but those are cheap and the safety win dominates.
The properly concurrent fix is to pin each JsRuntime to its own OS
thread and message-pass; tracked as the larger #19 follow-up.
Adds an opt-in smoke test (`--ignored`) that boots the CDP server and
runs 5 parallel clients through createTarget+navigate. Note: the
deterministic abort needs subresource-fetch yields (page.rs:285
join_all) which data: URLs skip — full repro requires JS-heavy real
traffic (issue author used scrapegraphai.com).
* fix(#19): suspend other Isolates and defer foreign CDP during interception nav
Address maintainer feedback on PR #36 (comment 4328238087): the V8 fatal
still reproduces with `Fetch.enable patterns:["*"]` + 5 concurrent
`Page.navigate`s. The `obscura_js::v8_lock` mutex was insufficient
because V8's actual invariant is "only one Isolate may be entered per
OS thread", and the mutex can't enforce that across `.await` points
inside V8 ops.
Two distinct paths violated the invariant:
1. Foreign CDP messages dispatched while a nav task was running.
`process_with_interception`'s `select!` routed every non-Fetch
message through `process_cdp_message → dispatch → page::handle::*`.
`get_session_page_mut` then `suspend_js`'es other pages (drops
their `JsRuntime` → `Isolate::Exit`) while the spawned nav task is
parked at `op_fetch_url`'s `resolve_rx.await` with its Isolate
still entered. Drop trips
`heap->isolate() == Isolate::TryGetCurrent()`.
2. Subsequent navs through `process_with_interception` did not
suspend prior pages. After nav-1 completes (page-1's JsRuntime
stays alive in `ctx.pages`), nav-2's `init_js` constructs
Isolate-2 while Isolate-1 is still current. The next V8 scope
unwind aborts via `Context::Exit`.
Fix (3 changes in `crates/obscura-cdp/src/server.rs`):
- `cdp_processor`: add a `VecDeque<ServerMessage> deferred` drained
before each new `rx.recv()`, so deferred messages are processed
with no nav task in flight.
- `process_with_interception` `select!`: only handle `NewConnection`
and `Fetch.{continue,fulfill,fail}Request` inline (safe — no V8
enter on this side); push everything else into `deferred`.
- `process_with_interception` entry: explicitly `suspend_js()` every
other page in `ctx.pages` before spawning the nav task, mirroring
the invariant the regular dispatch path already enforces via
`get_session_page_mut`.
Verification:
- New test `crates/obscura-cdp/tests/concurrent_navigations_with_fetch.rs`
is the maintainer's exact repro (5 clients, `Fetch.enable patterns:["*"]`,
concurrent `Page.navigate` to https://example.com, auto-reply
`Fetch.continueRequest`). Aborts on PR #36 head, passes after fix
(5/5 stable runs, ~1.2s each).
- Existing data-URL smoke test (`concurrent_navigations`) still passes.
- Full workspace test suite (90 tests) still passes.
Run with:
cargo test -p obscura-cdp --test concurrent_navigations_with_fetch \
-- --nocapture --ignored
Honest caveat. This is still the lock/serialize approach — it converts
the abort into latency. The properly concurrent fix (Option 2 in
issue #19: pin each `JsRuntime` to its own OS thread and message-pass)
remains the right long-term direction.
* fix(cdp): bound deferred message queue to prevent OOM during stalled navigation
PR #36 comment 4341743194: process_with_interception deferred foreign
CDP messages into an unbounded VecDeque while a navigation task was in
flight. A stalled navigation (e.g. waiting for Fetch interception
resolution) could accumulate arbitrary queued messages and OOM the
process.
Add MAX_DEFERRED_MESSAGES = 256. When the cap is reached, return an
explicit CDP error (-32000) to the client instead of silently dropping
or growing without bound.
All 90 workspace tests pass.
Puppeteer's first CDP call after connect is Target.getBrowserContexts.
When the single cdp_processor is busy (e.g. Page.navigate with fetch
interception), queued Target commands stall and Puppeteer hits
protocolTimeout on getBrowserContexts before any user code runs.
Answer getBrowserContexts synchronously in fast_path_response with the
same payload as domains::target::handle (default context id "default"),
matching Browser context initialization in CdpContext.
Co-authored-by: Obscura Contributor <contributor@users.noreply.github.com>
- obscura-net/cookies.rs: #[allow(dead_code)] on CookieEntry.same_site
(parsed today, kept for future SameSite enforcement).
- obscura-js/runtime.rs: drop vestigial BOOTSTRAP_JS const; bootstrap.js
is already embedded into the V8 snapshot via build.rs.
- obscura-js/ops.rs: prefix the four unused fields in
InterceptResolution::Continue with _.
- obscura-cdp/dispatch.rs: drop unused InterceptResolution import.
- obscura-cdp/domains/fetch.rs: drop unused Arc / Mutex / CdpEvent imports.
- obscura-cdp/domains/input.rs: prefix _button / _click_count; CDP
parses them but the JS click stub is hardcoded for now.
- obscura-cdp/server.rs: prefix _processor_handle and _ctx; drop dead
initial Ok(())/None for navigate_result/page_back - they are
unconditionally overwritten in the only loop break arm.
cargo build --workspace: 0 warnings.
cargo test --workspace: 113 passed.
Playwright's `page.pdf()` issues a `Page.printToPDF` CDP request which
Obscura was not handling, so the dispatcher fell through to the generic
catch-all and clients saw:
Page.pdf: Protocol error (Page.printToPDF): Unknown Page method: printToPDF
This reads like an oversight rather than a deliberate limitation.
Obscura intentionally has no layout engine — option (2) from the issue —
so make the limitation explicit by handling the method and returning a
clear error that names the method, says why it isn't supported, and
points users at a working alternative (extract HTML via
Runtime.evaluate, then render to PDF in their client).
The behavior change is small but meaningful for DX: the JSON-RPC error
code stays -32601 (the dispatcher's mapping for `Err(_)`), but the
human-readable message becomes actionable instead of looking like a
forgotten method.
Adds a unit test that pins the regression: printToPDF must NOT fall
through to the "Unknown Page method" catch-all and must mention both
"not supported" and a workaround.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
`obscura fetch --dump text` walked the entire <body> and emitted the
text of navigation menus, sidebars, and footers alongside the article
body. The skip list inside extract_readable_text already excluded
<script>/<style>; extend it with the four standard semantic-boilerplate
tags so the dump is closer to the article content users actually want
to scrape.
Behaviour change for users who relied on the noise being included is
acknowledged but unlikely — these tags are the canonical wrappers
boilerplate goes in. Anyone needing the full body should use --dump
html and post-process.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
* fix(cli): make --quiet suppress JS console errors on stderr (#56)
The --quiet flag was only gating banner prints in run_fetch but not
the global tracing_subscriber, so ERROR-level JS console output kept
leaking to stderr.
Detect quiet from the parsed Fetch subcommand before initializing
the subscriber and switch the filter to "off" in that case. Verbose
still wins over quiet to keep diagnostic mode usable.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-Authored-By: Ora Studio <noreply@oratelecom.net>
* test(cli): add integration tests for --quiet matcher wiring
Address review feedback on PR #74: previous tests only covered the
pure helper select_log_filter, not the matches!() wiring in main()
that maps Command::Fetch { quiet: true, .. } → quiet flag.
Extract the matcher into is_quiet_command(&Option<Command>) and add
six new tests that parse real CLI args via Args::try_parse_from and
assert the full chain:
- parsed_fetch_with_quiet_flag_is_detected: --quiet on fetch → true
- parsed_fetch_without_quiet_is_not_detected: bare fetch → false
- parsed_serve_command_is_not_quiet: serve subcommand → false
- no_subcommand_is_not_quiet: None → false
- parsed_fetch_quiet_resolves_to_off_filter: end-to-end clap → "off"
- matcher_still_uses_fetch_variant: regression guard if Command::Fetch
variant is renamed/restructured
cargo test -p obscura-cli: 10 passed; 0 failed.
Co-Authored-By: Ora Studio <noreply@oratelecom.net>
---------
Co-authored-by: Ora Studio <noreply@oratelecom.net>
obscura serve --user-agent <ua> only logged the value via tracing::info!
and never passed it to the CDP server, so every navigation kept using
the default Chrome UA string and httpbin.org/user-agent reported the
default rather than the override.
Plumb the option end-to-end:
- BrowserContext::with_full_options(id, proxy, stealth, user_agent)
initializes both BrowserContext::user_agent and the underlying
ObscuraHttpClient::user_agent RwLock at construction time so the
first navigation already sees the override.
- CdpContext::new_with_full_options forwards the UA to the default
BrowserContext.
- obscura_cdp::start_with_full_options accepts the UA and passes it
through cdp_processor.
- main.rs Serve arm calls start_with_full_options with the parsed UA.
- run_multi_worker_serve forwards --user-agent <ua> when spawning
worker subprocesses so multi-worker mode honors the flag too.
Existing two-arg start_with_options and three-arg with_options stay
as thin shims (call the full variant with None) so older callers and
the public crate API remain source-compatible.
Three new unit tests in obscura-browser cover the propagation, the
Chrome default fallback, and the legacy with_options path.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
Puppeteer's FrameManager.initialize calls Audits.enable on connect.
Without it puppeteer.connect() fails with "Unknown domain: Audits"
before any user code runs. Playwright doesn't enable Audits, which is
why the workaround in #57 sidesteps the issue.
Add Audits to the existing accepted-but-no-op domain list, alongside
Log/Performance/Security/etc. that already swallow Puppeteer's connect
chatter. No new module needed.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
document.elementFromPoint(x, y) was undefined, which caused Google
Publisher Tag, viewability libs and any code doing hit-testing to
throw on bootstrap and trigger retry loops in the surrounding
framework. This is one of the throwing APIs feeding into the wedge
described in #62.
Without a layout engine we can't return the actual topmost element,
but a non-throwing stub is strictly better than "undefined" for the
headless-scraping use case:
- coords inside the viewport return document.body (or
document.documentElement as a fallback) so callers that just want
a hit target stop throwing
- coords outside the viewport (negative, > innerWidth/innerHeight,
NaN/Infinity) return null per spec
- elementsFromPoint mirrors the same shape but returns a one- or
zero-length array
- ShadowRoot.prototype gets the same treatment for completeness
Five new unit tests in obscura-js cover function presence, in-viewport
hit, out-of-viewport coords, non-numeric input, and the array shape.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
document.createEvent('CustomEvent') was returning a generic Event,
so the legacy DOM Level 2 pattern
var e = document.createEvent('CustomEvent');
e.initCustomEvent('foo', true, false, payload);
threw "TypeError: e.initCustomEvent is not a function" because the
returned object was an Event, not a CustomEvent — and CustomEvent
didn't define initCustomEvent anyway. The Starbucks China bundle
(and a long tail of other CDN bundles using the same shim) depended
on this and aborted bootstrap.
Two changes in crates/obscura-js/js/bootstrap.js:
1. Add CustomEvent.prototype.initCustomEvent(type, bubbles, cancelable,
detail) matching the spec.
2. Rewrite Document.prototype.createEvent so it dispatches to the
right class based on the type string ('CustomEvent', 'MouseEvent',
'MouseEvents', 'KeyboardEvent', 'FocusEvent', 'InputEvent',
'UIEvent', 'WheelEvent', 'PointerEvent', 'ErrorEvent',
'PopStateEvent', 'AnimationEvent', 'TransitionEvent', and
case-insensitive 'Events' aliases). The HTMLDocument variant on
the second Document class delegates to the canonical impl.
Spec note: createEvent returns the event with an empty type until
init*Event() is called — matched exactly.
Four new unit tests cover initCustomEvent presence and field setters,
class dispatch for CustomEvent / MouseEvent / MouseEvents / KeyboardEvent,
and the unknown-type fallback to Event.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
jQuery DataTables references the standard CharacterData interface which
was undefined in obscura's DOM environment, causing ReferenceError and
empty page renders.
- Add CharacterData class extending Node with data/length/substringData/
appendData/insertData/deleteData/replaceData
- Make Text and Comment extend CharacterData instead of Node
- Update _wrap() to create proper Text/Comment instances by nodeType
- Add create_comment_node op for Document.createComment()
- Fix set_text_content to handle Comment node mutations
Fixes#72
Co-authored-by: KuaaMU <xcm853629353@gmail.com>
The current ubuntu-latest Linux release can require a newer glibc than common LTS servers provide. Pinning the x86_64 Linux release job to Ubuntu 22.04 keeps the prebuilt binary compatible with glibc 2.35+ environments while leaving macOS and Windows release jobs unchanged.
Also document the Linux LTS compatibility target in the install section so users understand why the release runner is pinned.
Playwright (and recent Puppeteer) call Target.attachToBrowserTarget on
connect to obtain a session for the implicit "browser" target. Without
the handler the dispatcher returns "Unknown Target method:
attachToBrowserTarget" and the connect handshake aborts before any
user code runs.
Add a handler that mirrors Target.attachToTarget but for the browser
target: register a "browser-session" → "browser" mapping in
ctx.sessions, emit Target.attachedToTarget with type=browser, and
return the new session id.
Note: this addresses item (1) from #64. The Rocket Loader / Cloudflare
JS execution gap and Page.getLayoutMetrics are tracked separately
(getLayoutMetrics is fixed in PR #79).
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
Playwright calls Page.getLayoutMetrics before every page.screenshot() to
size the capture. Without it the call fails with "Unknown Page method:
getLayoutMetrics" and screenshot() throws.
Obscura has no visual layout engine, so we return a fixed 1280x720
viewport (Chrome's default) and try to derive the content height from
document.documentElement.scrollHeight via the JS runtime, falling back
to the viewport height when no live page is bound.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <noreply@oratelecom.net>
Addresses follow-up issues from PR #44 bot review:
- Add parentId field to AXNode (walk DOM ancestors to resolve AX parent)
- Omit optional fields when empty (name, value, properties, childIds)
instead of emitting null/empty arrays, per CDP spec
- Add ax_value_boolean() and ax_value_integer() helpers for consistent
AXValue construction across all property types
- Use typed helpers throughout compute_properties() instead of inline
JSON construction
CDP spec reference: Accessibility.AXNode requires parentId as AXNodeId,
and optional fields should be absent when not applicable.
Co-authored-by: zhangyang-crazy-one <208990894+zhangyang-crazy-one@users.noreply.github.com>
Fetch.enable was setting page.intercept_enabled to false and clearing
intercept_block_patterns, which caused should_block_url() to always
return false. As a result, Fetch.requestPaused events were never
triggered and Playwright's page.route() handlers never executed.
- Set intercept_enabled to true (was false)
- Propagate URL patterns to page.intercept_block_patterns (was cleared)
Fixes#49
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per qodo-ai-reviewer feedback on PR #44: Accessibility.getFullAXTree
was emitting AXNode.role as a plain string and name/value as plain
strings or null, instead of CDP-compliant AXValue objects.
Changes:
- Add ax_value_role() and ax_value_string() helpers
- Change compute_name()/compute_value() to return Option<String>
- Wrap role, name, value as AXValue in build_ax_node()
The properties array already used the correct AXValue format.
Fixes#1: replace unwrap() panics in run_multi_worker_serve with match on the Result.
Fixes#2: route /json round-robin via worker_port instead of hardcoded port+1.
Fixes#3: return 502 Bad Gateway when a worker is unreachable instead of silently dropping the connection.
Fixes#5: thread --stealth through start_with_options -> CdpContext -> BrowserContext::with_options so the flag actually has an effect in serve mode (was let _ before). Log message now reflects the actual capability based on the stealth cargo feature being compiled in.
Fixes#15: try/catch the parentNode assignment in the attachShadow polyfill so it doesn't throw a TypeError against the read-only Node.parentNode getter (Cloudflare Turnstile triggers this path).
h4ckf0r0day