-
Notifications
You must be signed in to change notification settings - Fork 221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intermittent Content-decoder error on compressed response from www.google.com #284
Comments
Have similar problem while working with https://www.googleapis.com/ Is it possible disable compression support in http-em-client? |
@outoftime you're advertising "compressed" encoding, which em-http does not support and hence fails to decode when it receives it. Update your header to @ayanko yes, you can pass in em-http-request/spec/client_spec.rb Line 434 in 7b8d47c
|
Thx! So this 69 faraday adapter line should be corrected at least: options[:head]['accept-encoding'] = 'gzip' |
Closing, please feel free to reopen if there is more to do here.. It'd be nice to figure out why we get the error, but we need a reproducible test case. |
This should be reopened. I tried the code snippet in the first comment with a single encoding at a time ( The error is not always the same: Also, em-http-request does support the |
I have encountered same problem. |
Echoing the concerns above. In my case, no additional headers are needed to see this behavior. Just a standard request to |
Hi! 👋 I also have this issue, and used Here's my reproduction script (based on @outoftime's): require 'eventmachine'
require_relative 'lib/em-http-request'
100.times do
EventMachine.run do
request = EventMachine::HttpRequest.new('https://www.googleapis.com/')
.get(head: { 'Accept-Encoding' => 'gzip' })
request.callback { EventMachine.stop }
request.errback do
exit! if request.error == 'Content-decoder error'
EventMachine.stop
end
end
end Here's my
Here's the output:
Hope that helps! I'll do some more investigation myself - but thought sharing my findings may help somebody else to diagnose the problem further. |
I've investigated this a little further and found that it seems to be related to the way that the stream of bytes is delivered to the gzip decoder. To find this I captured the chunks of data from the gzipped response body of a GET request to https://www.googleapis.com/ ( Here's a script that demonstrates the issue: require 'eventmachine'
require_relative 'lib/em-http/decoders'
GOOD = ["\x1F", "\x8B", "\b", "\x00", "\x00", "\x00", "\x00", "\x00", "\x00", "\x00", "\xF3", "\xCB", "/", "Q", "p\xCB/\xCDK\x01\x00M\x8Ct\xB1\t\x00\x00\x00"]
BAD = ["\x1F", "\x8B", "\b", "\x00", "\x00", "\x00", "\x00", "\x00\x00\x00\xF3\xCB/Qp\xCB/\xCDK\x01\x00M\x8Ct\xB1\t\x00\x00\x00"]
def decode(stream)
decoder = EventMachine::HttpDecoders::GZip.new {}
stream.each { |bytes| decoder << bytes }
end
puts 'Good stream - will succeed'
decode GOOD
puts 'Bad stream - will error'
decode BAD |
This fixes an issue where a gzip stream chunked into smaller pieces would result in a buffer overrun when decoding the headers. Previously the position where the headers end in `compressed` was determined by the delta of `@pos` before and after reading the headers. This didn't take into account the fact that `@data` may already contain data, but not enough to have advanced `@pos` on previous iterations.
Prevent data corruption when decoding gzip stream, fixes #284
We're observing an intermittent error (~40% of requests) when making a request to
https://www.google.com
with compressed responses accepted. Here's a reduction:Note that the
User-Agent
header is needed to reproduce the problem; without it, the error is not in evidence.When making requests to Google with curl, using the same headers, I haven't observed the problem, so I think the problem is specific to
em-http-request
or one of its dependencies.The text was updated successfully, but these errors were encountered: