-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix infinite loop releasing the connection when the writer is not finished #7798
Closed
Closed
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
86b741d
Fix infinite loop releasing the connection when the writer is not fin…
bdraco f19c64d
timeline
bdraco 259d5d5
lint
bdraco 8e4b1c1
3.8 is still supported
bdraco c1865d2
reduce code
bdraco ed0a472
improve docstring
bdraco 615c6ca
improve docstring
bdraco 6e1a017
Merge branch 'master' into loop_writer_not_done
bdraco 885afdb
ensure writer cleaned up before _release_connection
bdraco File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Fix infinite loop releasing the connection when the writer is not finished |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made it a separate function to ensure any future refactoring never generates a loop
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there definitely an issue here? The writer task itself does
self._writer = None
, which is why I originally left this code like this. It might be worth making a change to make it clearer that the behaviour is correct, I'm just aware that we already have multiple blocks of code in this class doing almost identical things, so I was trying to avoid adding another one.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The loop is
_release_connection
is called andself._writer
is not None so it calls theself._writer.add_done_callback
, the writer finishes and the callback fires, so_release_connection
is called again, and sinceself._writer
was never unset onClientResponse
it does theself._writer.add_done_callback
again and since its already done, it fires viacall_soon
, and the loop repeats.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also it happened without intervention and I only found the issue because the container was using 100% cpu.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, but why does the callback happen when the writer has not been reset? The task itself resets the attribute before completing...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait, just realised, that assignment in the init needs to trigger the callback logic. Just pushed a commit to ensure that happens. Can you try again with that one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll give it a shot as soon as I get back home < 1h
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cherry-picked f45d9e4 cleanly, results shortly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Came up cleanly, loop did not happen right away as before.
profile is clean, py-spy is clean
Will report back tomorrow as the original symptoms only happened after about 12 hours (not sure one which request)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything ran fine overnight.
I rebooted a few switches and routers to generate some network chaos and everything recovered just fine
Closing this PR in favor of #7815