SGavrl
52fcf2a35a
obscura was calling String::from_utf8_lossy on every response body, which
silently corrupts every non-UTF-8 page (GBK, Big5, Shift-JIS, Windows-125x,
EUC-KR, ISO-8859-x). Whole language families came through as replacement
characters.
Add an encoding module (obscura-net::encoding) that picks the decoder
the way real browsers do per HTML5 § 8.2.2.4:
1. HTTP Content-Type charset=... parameter (authoritative when present)
2. <meta charset> or <meta http-equiv=Content-Type> sniffed from the
first 1KB of the body (HTML only)
3. UTF-8 fallback
Use it in:
- obscura-net Response::text (signature changed: String, not Result)
- obscura-browser page.rs HTML body decode (the primary scrape path)
- obscura-browser page.rs JS body decode
- obscura-browser page.rs CSS body decode
For JS / CSS we deliberately skip the meta sniff: those bodies are not
HTML and any '<meta charset=...>' substring inside them is not a charset
declaration.
Adds 7 unit tests for the encoding module covering header precedence,
quoted charset values, meta charset sniff, http-equiv form, the 1KB
sniff window, and a round-trip decode of GBK bytes. Adds a Python
feature test exercising file:// + meta charset and a live HTTP server
serving Shift_JIS with the wrong meta tag.
Closes #113.
31 lines
1.0 KiB
TOML
31 lines
1.0 KiB
TOML
[package]
|
|
name = "obscura-net"
|
|
version.workspace = true
|
|
edition.workspace = true
|
|
|
|
[features]
|
|
default = []
|
|
stealth = ["wreq", "wreq-util"]
|
|
|
|
[dependencies]
|
|
reqwest = { workspace = true }
|
|
tokio = { workspace = true }
|
|
url = { workspace = true }
|
|
serde = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
tracing = { workspace = true }
|
|
thiserror = { workspace = true }
|
|
async-trait = "0.1"
|
|
encoding_rs = "0.8"
|
|
wreq-util = { version = "3.0.0-rc.10", optional = true }
|
|
|
|
# `prefix-symbols` renames BoringSSL exports to avoid clashes with system OpenSSL,
|
|
# but the renaming is only correct on Linux and Android (per wreq's README).
|
|
# Enabling it on other platforms produces unresolved `build_script_main_*`
|
|
# symbols at link time (see issue #39).
|
|
[target.'cfg(any(target_os = "linux", target_os = "android"))'.dependencies]
|
|
wreq = { version = "6.0.0-rc.28", optional = true, features = ["prefix-symbols"] }
|
|
|
|
[target.'cfg(not(any(target_os = "linux", target_os = "android")))'.dependencies]
|
|
wreq = { version = "6.0.0-rc.28", optional = true }
|