The wreq dependency was declared without the gzip, brotli, deflate and zstd
features, so the stealth client returned bodies still compressed while the
reqwest path decoded them. The Chrome145 emulation advertises those encodings,
so origins do compress, and the raw gzip bytes reached the HTML parser as
document text.
Add a regression test that serves one gzip response and asserts the stealth
client decodes it.
Fixes#563.
The stealth (wreq) client gated its custom-trust-store build on
`var_os(...).is_some()`, which is true for a set-but-empty variable. Supplying
a store to wreq's `tls_cert_store` REPLACES the bundled webpki roots, so an
empty `SSL_CERT_FILE` (a common shell accident, e.g. `export SSL_CERT_FILE=$UNSET`)
built a near-empty default-paths store and made every HTTPS request fail with
CERTIFICATE_VERIFY_FAILED -- while the reqwest client, which already filters
empty values in configured_root_paths(), kept working. Same input, opposite
behavior.
Extract a shared `custom_cert_store_requested` predicate in client.rs that
applies the same `!is_empty()` filter, and call it from the stealth client's
guard. The predicate takes the values as arguments so it is unit-tested under
default features (the stealth feature pulls in boringssl and is not built in
the normal test job).
Fixes#551
The #525 fix landed on the reqwest path (`client.rs`: `configured_root_paths()`
-> `add_root_certificate`) but did not cover `wreq_client.rs`. The stealth
client builds its `wreq::Client` without a certificate store, so those
variables are silently ignored whenever `--stealth` is on.
The consequence is inverted: the transport with the best TLS fingerprint is the
only one that cannot reach a host behind a private or incomplete CA chain --
precisely the environments #525 described.
Same binary on current main, same private-CA server, only the flag differs:
SSL_CERT_FILE=ca.pem obscura fetch https://127.0.0.1:8443/ -> ok
SSL_CERT_FILE=ca.pem obscura --stealth fetch https://127.0.0.1:8443/ -> fails
`wreq` already exposes the hook -- `ClientBuilder::tls_cert_store(CertStore)`,
and `CertStore::builder().set_default_paths()` reads both variables. It is
simply never called.
Two constraints, both of which fail silently if got wrong:
* `tls_cert_store`, not `tls_options`. `emulation()` is documented as
"This will overwrite the existing configuration" and internally calls
`self.tls_options(emulation.tls_options)`; setting TLS options here would
discard the Chrome fingerprint, which is the whole point of this client.
`tls_cert_store` is a separate config field and composes with emulation.
* Opt-in only. Supplying a store REPLACES the webpki roots (see
`set_cert_store` in `tls/conn/ext.rs`), it does not add to them. Applying it
unconditionally would break ordinary sites whenever the bundle is
incomplete. With neither variable set, behaviour is byte-for-byte unchanged.
Verified: with the variable, a private-CA host resolves under --stealth;
without it, the same host still fails; with a deliberately EMPTY bundle,
example.com fails -- which is what proves the store is actually applied rather
than the bundle merely happening to be right.
Refs #538, #525
op_fetch_url backed JS fetch() and XHR with a plain reqwest/rustls client, so
in stealth mode scripted requests carried a non-browser TLS fingerprint and
none of the Chrome client hints the main navigation sends. Bot managers that
score subresource requests (the AWS WAF challenge verify call, Akamai sensors)
read that as a script and reject it. Add StealthHttpClient::send_single and a
stealth_fetch_all path that op_fetch_url uses when a stealth client is
installed, preserving the redirect, SSRF, and CORS handling so JS-level
requests share the navigation's Chrome TLS fingerprint and sec-ch-ua client
hints. The reqwest path is unchanged for non-stealth. Adds a stealth feature to
obscura-js gated on obscura-net/stealth.
Stealth mode forced a Linux navigator UA and platform but reported Windows
in userAgentData.getHighEntropyValues and served Windows Direct3D11 WebGL
renderers, a combination no real Linux Chrome ever produces and that
cross-layer fingerprint checks flag. Present one consistent Windows Chrome
145 identity instead: the wreq emulation uses Platform::Windows, and the
navigator surfaces read the UA and platform from shared constants so the
HTTP User-Agent header, sec-ch-ua-platform, navigator.userAgent, and
userAgentData all agree. userAgentData brands are now derived from the
Chrome major version via Chromium's GREASE algorithm rather than a single
hardcoded token, and a Linux WebGL renderer pool is added so a Linux UA can
never report a Direct3D11 renderer.
wreq upgraded to 6.0.0-rc.29 and wreq-util to 3.0.0-rc.12 to apply the
latest Chrome 145 TLS fingerprint (16 cipher suites, proper extension
order, ALPS, GREASE).
__obscura_stealth global variable introduced and passed from Rust to JS,
allowing stealth-specific JS behaviors to run only when the stealth flag
is enabled.
Tightened hardwareConcurrency and deviceMemory pools in stealth mode
to desktop-realistic values (e.g. min 4GB RAM, 4 cores), eliminating
mobile-tier values that leak the spoof.
navigator.userAgentData and getHighEntropyValues() are now overridden
in stealth mode to perfectly match the sec-ch-ua header sent by
wreq_util::Profile::Chrome145.
Fixed usedJSHeapSize sometimes exceeding totalJSHeapSize.
set_default_paths only resolves OpenSSL's compile-time CA dirs, which
exist on Linux but are empty on Windows. The stealth client's cert store
ended up empty there and every TLS handshake failed with
CERTIFICATE_VERIFY_FAILED inside BoringSSL handshake.cc.
CertStore::default() uses wreq's bundled webpki-root-certs (Mozilla's
trust list), which behaves the same on every platform and removes the
dependency on a system CA store.
SGavrl