fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes #113.
This commit is contained in:
1 parent
f4f49ebab4
commit
52fcf2a35a
6 files changed
+235
-8
No files matched your search
@@ -1,5 +1,6 @@
|
||||
pub mod client;
|
||||
pub mod cookies;
|
||||
pub mod encoding;
|
||||
pub mod interceptor;
|
||||
pub mod robots;
|
||||
pub mod blocklist;
|
||||
@@ -8,6 +9,7 @@ pub mod wreq_client;
|
||||
|
||||
pub use client::{ObscuraHttpClient, ObscuraNetError, RequestInfo, ResourceType, Response};
|
||||
pub use cookies::{CookieInfo, CookieJar};
|
||||
pub use encoding::{decode_non_html, decode_response};
|
||||
pub use robots::RobotsCache;
|
||||
pub use blocklist::is_blocked as is_tracker_blocked;
|
||||
#[cfg(feature = "stealth")]
|
||||
|
||||
Reference in new issue
Block a user