fix(net,browser): detect response charset, decode non-UTF-8 pages (#182)

obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.

Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:

  1. HTTP Content-Type charset=... parameter (authoritative when present)
  2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
     first 1KB of the body (HTML only)
  3. UTF-8 fallback

Use it in:
  - obscura-net Response::text (signature changed: String, not Result)
  - obscura-browser page.rs HTML body decode (the primary scrape path)
  - obscura-browser page.rs JS body decode
  - obscura-browser page.rs CSS body decode

For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.

Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.

Closes #113.
This commit is contained in:
SGavrl authored and GitHub committed 2026-05-24 20:14:53 +00:00
1 parent f4f49ebab4
commit 52fcf2a35a
6 files changed
+235 -8

No files matched your search

+11 -2
View File
@@ -21,8 +21,17 @@ pub struct Response {
}
impl Response {
pub fn text(&self) -> Result<String, std::string::FromUtf8Error> {
String::from_utf8(self.body.clone())
/// Decode the body as text, honoring the response charset.
///
/// Uses the HTTP `Content-Type` header's `charset=` parameter, then for
/// HTML responses falls back to sniffing `<meta charset>` in the first
/// 1KB, then UTF-8. Mirrors browser behaviour per the HTML5 spec.
pub fn text(&self) -> String {
if self.is_html() {
crate::encoding::decode_response(&self.body, self.content_type())
} else {
crate::encoding::decode_non_html(&self.body, self.content_type())
}
}
pub fn header(&self, name: &str) -> Option<&str> {