HTTPProvider timeout is not working for chunked responses

[Uxio0]

What happened?

This is a problem with urllib3 and not with web3.py. If server answers with a chunked response, timeout will not be respected. More info:

• https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Timeout
• Request with chunked encoding doesn't respect timeout psf/requests#5668

Code that produced the error

from web3 import Web3
w3 = Web3(Web3.HTTPProvider("http://127.0.0.1:8545", request_kwargs={'timeout': 60}))


A big query to `tracing.trace_filter` using an Erigon node. Erigon answers with a `Transfer-Encoding: chunked` header and starts slowly streaming the result, not letting the timeout trigger, query takes more than 10 minutes.

Full error output

No error. Code hangs when `Web3.py` does the request until Erigon (or other RPC) ends streaming the response.

Fill this section in if you know how this could or should be fixed

Using requests.post(url, json=data, stream=True) would allow Web3.py to control for how long the request is stuck, and process the chunked data in a more optimal way. Also, if the data is not chunked it shouldn't be an issue at all.

web3 Version

6.19.0

Python Version

3.12.4

Operating System

linux

Output from pip freeze

aiohttp==3.9.5
aiosignal==1.3.1
amqp==5.2.0
asgiref==3.7.2
astroid==3.1.0
asttokens==2.4.1
attrs==23.2.0
billiard==4.2.0
bitarray==2.9.2
black==24.4.2
boto3==1.34.117
botocore==1.34.126
CacheControl==0.14.0
cached-property==1.5.2
cachetools==5.3.3
celery==5.4.0
certifi==2024.2.2
cffi==1.16.0
cfgv==3.4.0
charset-normalizer==3.3.2
ckzg==1.0.1
click==8.1.7
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
coverage==7.5.3
cron-descriptor==1.4.3
cryptography==42.0.5
cytoolz==0.12.3
debugpy==1.8.1
decorator==5.1.1
dill==0.3.8
distlib==0.3.8
Django==5.0.6
django-appconf==1.0.6
django-cache-memoize==0.2.0
django-celery-beat==2.6.0
django-cors-headers==4.3.1
django-db-geventpool==4.0.5
django-debug-toolbar==4.3.0
django-debug-toolbar-force==0.2
django-environ==0.11.2
django-extensions==3.2.3
django-filter==24.2
django-imagekit==5.0.0
django-model-utils==4.5.1
django-nine==0.2.7
django-redis==5.4.0
django-s3-storage==0.15.0
django-stubs==5.0.0
django-stubs-ext==5.0.0
django-test-migrations==1.3.0
django-timezone-field==6.1.0
djangorestframework==3.15.1
djangorestframework-camel-case==1.4.2
docker==7.0.0
docutils==0.21.2
drf-yasg==1.21.7
eth-account==0.11.2
eth-bloom==3.0.1
eth-hash==0.7.0
eth-keyfile==0.8.1
eth-keys==0.5.1
eth-rlp==1.0.1
eth-typing==4.1.0
eth-utils==4.1.0
eth_abi==5.1.0
executing==2.0.1
factory-boy==3.3.0
Faker==25.8.0
filelock==3.14.0
firebase-admin==6.5.0
flake8==7.0.0
flower==2.0.1
frozenlist==1.4.1
gevent==24.2.1
google-api-core==2.19.0
google-api-python-client==2.127.0
google-auth==2.29.0
google-auth-httplib2==0.2.0
google-cloud-core==2.4.1
google-cloud-firestore==2.16.0
google-cloud-storage==2.16.0
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.63.0
greenlet==3.0.3
grpcio==1.63.0
grpcio-status==1.62.2
gunicorn==22.0.0
hexbytes==0.3.1
hiredis==2.3.2
httplib2==0.22.0
humanize==4.9.0
identify==2.5.36
idna==3.7
inflection==0.5.1
iniconfig==2.0.0
ipdb==0.13.13
ipython==8.24.0
isort==5.13.2
jedi==0.19.1
jmespath==1.0.1
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
kombu==5.3.7
lru-dict==1.2.0
matplotlib-inline==0.1.7
mccabe==0.7.0
msgpack==1.0.8
multidict==6.0.5
mypy==1.10.0
mypy-extensions==1.0.0
nodeenv==1.8.0
packaging==24.0
parsimonious==0.10.0
parso==0.8.4
pathspec==0.12.1
pexpect==4.9.0
pika==1.3.2
pilkit==3.0
pillow==10.3.0
platformdirs==4.2.1
pluggy==1.5.0
pre-commit==3.7.0
prometheus_client==0.20.0
prompt-toolkit==3.0.43
proto-plus==1.23.0
protobuf==4.25.3
psutil==5.9.8
psycogreen==1.0.2
psycopg2==2.9.9
ptyprocess==0.7.0
pur==7.3.1
pure-eval==0.2.2
py==1.11.0
py-ecc==7.0.1
py-evm==0.10.1b1
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycodestyle==2.11.1
pycparser==2.22
pycryptodome==3.20.0
pyflakes==3.2.0
Pygments==2.17.2
PyJWT==2.8.0
pylint==3.1.0
pylint-django==2.5.5
pylint-plugin-utils==0.8.2
pyparsing==3.1.2
pytest==8.2.1
pytest-celery==1.0.0
pytest-django==4.8.0
pytest-docker-tools==3.1.3
pytest-env==1.1.3
pytest-rerunfailures==14.0
pytest-sugar==1.0.0
python-crontab==3.0.0
python-dateutil==2.9.0.post0
pytz==2024.1
pyunormalize==15.1.0
PyYAML==6.0.1
redis==5.0.6
referencing==0.35.1
regex==2024.4.28
requests==2.32.3
retry==0.9.2
rlp==4.0.1
rpds-py==0.18.0
rsa==4.9
s3transfer==0.10.1
safe-eth-py==6.0.0b31
safe-pysha3==1.0.4
setuptools==69.5.1
six==1.16.0
sortedcontainers==2.4.0
sqlparse==0.5.0
stack-data==0.6.3
swagger-spec-validator==3.0.3
termcolor==2.4.0
tomlkit==0.12.4
toolz==0.12.1
tornado==6.4
traitlets==5.14.3
trie==3.0.1
types-pytz==2024.1.0.20240417
types-PyYAML==6.0.12.20240311
typing_extensions==4.11.0
tzdata==2024.1
uritemplate==4.1.1
urllib3==2.2.1
vine==5.1.0
virtualenv==20.26.1
wcwidth==0.2.13
web3==6.19.0
websockets==12.0
wheel==0.43.0
yarl==1.9.4
zope.event==5.0
zope.interface==6.3

[kclowes]

You should be able to pass in stream=True as a request_kwarg to the provider. If that doesn't work and we need to dig in further, please reopen :)

[Uxio0]

@kclowes Even if I pass stream=True, as the response is not iterated and just parsed as a not streamed one in Web3.py that makes no difference. Web3.py is blocked until all the chunks in the response are received and parsed

[Uxio0]

(I don't have permissions to reopen the issue)

[kclowes]

Ah, interesting. I'll take a deeper look. Thanks!

[Uxio0]

I feel maybe #1368 can be implemented also when fixing this, as the timeout can be used in this case. Let me see if I can get some time to draft a PR, at least a PoC of my idea

[Uxio0]

@kclowes this is a very quick PoC that I think illustrates my idea: #3428

[kclowes]

Closed via #3428

[Uxio0]

Thank you very much for taking the time for this, this will help us a lot