Adds a multi-stage Dockerfile (rust:1-slim-bookworm builder → distroless/cc
runtime) plus a --host flag for obscura serve that defaults to 127.0.0.1
for safety but can be set to 0.0.0.0 so the container's port is reachable
via docker run -p mapping.
Dockerfile cherry-picked from #84. The SPA chunk loading commit from that
PR was left out — verified it didn't actually fix the Next.js webpack
errors on kiwi.com (#86).
Surfaces the existing LP.getMarkdown CDP machinery as a CLI flag so
text-pipeline users (LLM agents, scrapers feeding markdown to other
tools) don't need a separate HTML-to-markdown converter step.
obscura fetch --dump markdown 'https://example.com'
Layout:
- crates/obscura-js/src/markdown.rs holds HTML_TO_MARKDOWN_JS as a
pub const so every layer can share the same DOM-walker. obscura-js
is at the bottom of the dep DAG (obscura-browser -> obscura-js,
obscura-cdp -> obscura-js, obscura-cli -> obscura-js) so a single
source of truth works without a circular dependency.
- crates/obscura-browser/src/lib.rs re-exports it for ergonomics.
- crates/obscura-cdp/src/domains/lp.rs drops its inline duplicate
and calls the shared const.
- crates/obscura-cli/src/main.rs adds the `Markdown` DumpFormat
variant + dump_markdown(&mut Page) which page.evaluate's the
shared script and prints the result.
Net diff: a new 75-line markdown.rs, a 70+ line deletion in lp.rs
that's reborn as 3 lines, and ~10 lines added across the other four
files. No behavior change for `--dump html|text|links`, and
LP.getMarkdown over CDP keeps the exact same JSON shape.
Four new unit tests in obscura-js cover headings, inline emphasis +
links, ordered/unordered lists, and that <script> / <style> contents
do not leak into the output.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
`obscura fetch --dump text` walked the entire <body> and emitted the
text of navigation menus, sidebars, and footers alongside the article
body. The skip list inside extract_readable_text already excluded
<script>/<style>; extend it with the four standard semantic-boilerplate
tags so the dump is closer to the article content users actually want
to scrape.
Behaviour change for users who relied on the noise being included is
acknowledged but unlikely — these tags are the canonical wrappers
boilerplate goes in. Anyone needing the full body should use --dump
html and post-process.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
* fix(cli): make --quiet suppress JS console errors on stderr (#56)
The --quiet flag was only gating banner prints in run_fetch but not
the global tracing_subscriber, so ERROR-level JS console output kept
leaking to stderr.
Detect quiet from the parsed Fetch subcommand before initializing
the subscriber and switch the filter to "off" in that case. Verbose
still wins over quiet to keep diagnostic mode usable.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-Authored-By: Ora Studio <[email protected]>
* test(cli): add integration tests for --quiet matcher wiring
Address review feedback on PR #74: previous tests only covered the
pure helper select_log_filter, not the matches!() wiring in main()
that maps Command::Fetch { quiet: true, .. } → quiet flag.
Extract the matcher into is_quiet_command(&Option<Command>) and add
six new tests that parse real CLI args via Args::try_parse_from and
assert the full chain:
- parsed_fetch_with_quiet_flag_is_detected: --quiet on fetch → true
- parsed_fetch_without_quiet_is_not_detected: bare fetch → false
- parsed_serve_command_is_not_quiet: serve subcommand → false
- no_subcommand_is_not_quiet: None → false
- parsed_fetch_quiet_resolves_to_off_filter: end-to-end clap → "off"
- matcher_still_uses_fetch_variant: regression guard if Command::Fetch
variant is renamed/restructured
cargo test -p obscura-cli: 10 passed; 0 failed.
Co-Authored-By: Ora Studio <[email protected]>
---------
Co-authored-by: Ora Studio <[email protected]>
obscura serve --user-agent <ua> only logged the value via tracing::info!
and never passed it to the CDP server, so every navigation kept using
the default Chrome UA string and httpbin.org/user-agent reported the
default rather than the override.
Plumb the option end-to-end:
- BrowserContext::with_full_options(id, proxy, stealth, user_agent)
initializes both BrowserContext::user_agent and the underlying
ObscuraHttpClient::user_agent RwLock at construction time so the
first navigation already sees the override.
- CdpContext::new_with_full_options forwards the UA to the default
BrowserContext.
- obscura_cdp::start_with_full_options accepts the UA and passes it
through cdp_processor.
- main.rs Serve arm calls start_with_full_options with the parsed UA.
- run_multi_worker_serve forwards --user-agent <ua> when spawning
worker subprocesses so multi-worker mode honors the flag too.
Existing two-arg start_with_options and three-arg with_options stay
as thin shims (call the full variant with None) so older callers and
the public crate API remain source-compatible.
Three new unit tests in obscura-browser cover the propagation, the
Chrome default fallback, and the legacy with_options path.
Generated by Ora Studio
Vibe coded by ousamabenyounes
Co-authored-by: Ora Studio <[email protected]>
Fixes#1: replace unwrap() panics in run_multi_worker_serve with match on the Result.
Fixes#2: route /json round-robin via worker_port instead of hardcoded port+1.
Fixes#3: return 502 Bad Gateway when a worker is unreachable instead of silently dropping the connection.
Fixes#5: thread --stealth through start_with_options -> CdpContext -> BrowserContext::with_options so the flag actually has an effect in serve mode (was let _ before). Log message now reflects the actual capability based on the stealth cargo feature being compiled in.
Fixes#15: try/catch the parentNode assignment in the attachShadow polyfill so it doesn't throw a TypeError against the read-only Node.parentNode getter (Cloudflare Turnstile triggers this path).
Closes#7.
A stuck worker subprocess in `obscura scrape` could hold its semaphore
permit forever and hang the entire job. The two `BufReader::read_line`
calls and the final `child.wait()` were unbounded; a worker that never
wrote a response line, or a `--eval` expression that never terminated,
would block indefinitely.
This wraps the per-worker spawn->navigate->evaluate->shutdown sequence
in `tokio::time::timeout(worker_timeout, ...)` and adds tighter
`tokio::time::timeout(read_timeout, ...)` wrappers around each
`read_line` call. Shutdown waits and the kill-on-timeout path are
bounded with a 5s `shutdown_timeout`, so a misbehaving worker can't
block cleanup either.
Adds a `--timeout <SECS>` flag on the `scrape` subcommand (default 60,
clap-validated `>= 1`). On timeout, the child is killed and the result
JSON entry becomes `{"url": ..., "error": "timeout", "time_ms": ...}`,
keeping the output stream well-formed.
Per-worker max time is now bounded by `worker_timeout` (outer) plus the
shutdown_timeout cleanup window, regardless of worker behavior.
SGavrl