Reading a gzipped response hangs forever upon reaching the end of the response

[cyberpunkbln]

Hello,

i use the actual git version (that solves a problem with the cookiestore for me) and rust compiles without an error but the code hang on every function call like:

let test = httpreq.into_string().unwrap();
or
let json: serde_json::Value = httpreq.into_json().unwrap();

He hangs and i must break the compiled program.

[jsha]

Please add env_logger to your Cargo.toml, add env_logger::int() to your fn main, and run your program with RUST_LOG=debug cargo run, then paste the full debug output here. Also, please post a full copy of your program and any parameters you are invoking it with (like URLs or hostnames).

[joshstoik1]

I'm running into the same problem with the newly-released 2.6.0 anytime a gzipped-response has an unknown content-length.

# rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /usr/local/rustup

stable-x86_64-unknown-linux-gnu (default)
rustc 1.66.0 (69f9c33d7 2022-12-12)

Here is what env_logger spits out before the program hangs:

[2023-01-02T22:01:07Z DEBUG ureq::stream] connecting to github.com:443 at 140.82.112.3:443
[2023-01-02T22:01:07Z DEBUG rustls::client::hs] No cached session for DnsName(DnsName(DnsName("github.com")))
[2023-01-02T22:01:07Z DEBUG rustls::client::hs] Not resuming any session
[2023-01-02T22:01:07Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_128_GCM_SHA256
[2023-01-02T22:01:07Z DEBUG rustls::client::tls13] Not resuming
[2023-01-02T22:01:07Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck]
[2023-01-02T22:01:07Z DEBUG rustls::client::hs] ALPN protocol is None
[2023-01-02T22:01:07Z DEBUG ureq::stream] created stream: Stream(RustlsStream)
[2023-01-02T22:01:07Z DEBUG ureq::unit] sending request GET https://github.com/Fyrd/caniuse/raw/main/fulldata-json/data-2.0.json
[2023-01-02T22:01:07Z DEBUG ureq::unit] writing prelude: GET /Fyrd/caniuse/raw/main/fulldata-json/data-2.0.json HTTP/1.1
    Host: github.com
    Accept: */*
    user-agent: Mozilla/5.0
    accept-encoding: gzip
[2023-01-02T22:01:08Z DEBUG rustls::client::tls13] Ticket saved
[2023-01-02T22:01:08Z DEBUG rustls::client::tls13] Ticket saved
[2023-01-02T22:01:08Z DEBUG ureq::response] zero-length body returning stream directly to pool
[2023-01-02T22:01:08Z DEBUG ureq::pool] adding stream to pool: https|github.com|443 -> Stream(RustlsStream)
[2023-01-02T22:01:08Z DEBUG ureq::unit] response 302 to GET https://github.com/Fyrd/caniuse/raw/main/fulldata-json/data-2.0.json
[2023-01-02T22:01:08Z DEBUG ureq::unit] redirect 302 https://github.com/Fyrd/caniuse/raw/main/fulldata-json/data-2.0.json -> https://raw.githubusercontent.com/Fyrd/caniuse/main/fulldata-json/data-2.0.json
[2023-01-02T22:01:08Z DEBUG ureq::stream] connecting to raw.githubusercontent.com:443 at 185.199.108.133:443
[2023-01-02T22:01:08Z DEBUG rustls::client::hs] No cached session for DnsName(DnsName(DnsName("raw.githubusercontent.com")))
[2023-01-02T22:01:08Z DEBUG rustls::client::hs] Not resuming any session
[2023-01-02T22:01:08Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2023-01-02T22:01:08Z DEBUG rustls::client::tls13] Not resuming
[2023-01-02T22:01:08Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck]
[2023-01-02T22:01:08Z DEBUG rustls::client::hs] ALPN protocol is None
[2023-01-02T22:01:08Z DEBUG rustls::client::tls13] Ticket saved
[2023-01-02T22:01:08Z DEBUG ureq::stream] created stream: Stream(RustlsStream)
[2023-01-02T22:01:08Z DEBUG ureq::unit] sending request GET https://raw.githubusercontent.com/Fyrd/caniuse/main/fulldata-json/data-2.0.json
[2023-01-02T22:01:08Z DEBUG ureq::unit] writing prelude: GET /Fyrd/caniuse/main/fulldata-json/data-2.0.json HTTP/1.1
    Host: raw.githubusercontent.com
    Accept: */*
    user-agent: Mozilla/5.0
    accept-encoding: gzip
[2023-01-02T22:01:08Z DEBUG ureq::response] Body of unknown size - read until socket close
[2023-01-02T22:01:08Z DEBUG ureq::unit] response 200 to GET https://raw.githubusercontent.com/Fyrd/caniuse/main/fulldata-json/data-2.0.json

This can be reproduced with a cargo init boilerplate like:

Cargo.toml:

[package]
name = "ureqtest"
version = "0.1.0"
edition = "2021"

[dependencies]
env_logger = "*"

[dependencies.ureq]
version = "=2.6.0"
default-features = false
features = [ "tls", "gzip" ]

main.rs:

fn main() {
	env_logger::init();

	let res = ureq::get("https://github.com/Fyrd/caniuse/raw/main/fulldata-json/data-2.0.json")
		.set("user-agent", "Mozilla/5.0")
		.call()
		.expect("Call fail.");

	let mut out: Vec<u8> = Vec::new();
	res.into_reader().read_to_end(&mut out).expect("Read fail.");

	println!("{}", out.len()); // Never makes it here.
}

Trying to limit the reader with .take(10_000_000) doesn't fix the issue. The take amount has to be less than or equal to the actual decoded payload size or else it has no effect.

[jsha]

Excellent. Thanks for the detailed report. I'll reproduce this when I get home.

[jsha]

Fix is published in ureq 2.6.1. Thanks again for the detailed repro case! It helped me find the problem very quickly.

[joshstoik1]

Thanks for the quick fix @jsha! That seems to have done the trick.