Picks up @Timtech4u's PR #33, which has been waiting since April. The
design (CLI flag + threading through BrowserContext + http client) is
theirs and is correct; this is a clean rewrite on top of current main
because repeated rebase rounds had been reverting unrelated work that
landed in the meantime.
Issue #4 blocked all loopback / RFC1918 / link-local fetches to prevent
SSRF. The only escape was OBSCURA_ALLOW_PRIVATE_NETWORK=1, which is
process-wide and clunky in pipelines. Multiple users (@Timtech4u in
the original PR, then @duartemvix and @necrogami in the comment
thread) reported they cannot use obscura for development against
http://localhost:N without it.
Adds --allow-private-network as a global CLI flag, works on fetch,
serve, and scrape. Threads through BrowserContext::with_storage_and_network
and the http client's validate_url. CdpContext gets a new_full
builder and obscura_cdp::start_with_full_serve_options entry point so
'obscura serve --allow-private-network' lets Puppeteer/Playwright
navigate to localhost too. Default remains blocked.
Backward compat: the env var still works (OR'd with the flag inside
validate_url). All existing constructors and entry points unchanged.
Verified end-to-end against a local http.server on 127.0.0.1: fetch
and CDP Page.navigate both blocked by default, both succeed with the
flag. Public URLs unaffected. Integration suite 28/28, full workspace
cargo test all green.
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes#113.
SGavrl