Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't retry respond_peers message #3508

Merged
merged 1 commit into from
May 3, 2021
Merged

Don't retry respond_peers message #3508

merged 1 commit into from
May 3, 2021

Conversation

mariano54
Copy link
Contributor

No description provided.

@mariano54 mariano54 requested a review from Yostra May 3, 2021 07:33
arvidn
arvidn approved these changes May 3, 2021
View reviewed changes
Copy link
Contributor

@arvidn arvidn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. All I'm missing is the rationale to not retry respond_peers message

@mariano54
Copy link
Contributor Author

mariano54 commented May 3, 2021 •
edited
Loading

We are trying to broadcast more peers / min than the actual rate limit (which is 10 per minute). Rate limits are multiplied by 0.4 when we rate limit our own outgoing messages, so that would be max 4 per minute.

Before 1.1.2, there was no retrying, so if we sent to much they were just discarded. Now we are adding these messages to the queue over and over and the queue is not getting cleared fast enough, so the connection closes.

The real fix would be to increase the rate limit, but that's hard to do because we need to support old clients with the low limit.

@mariano54 mariano54 mentioned this pull request May 3, 2021
Closed
@mariano54 mariano54 merged commit 5ce1bfc into main May 3, 2021
@mariano54 mariano54 deleted the ms.rate_limit_bug branch May 3, 2021 09:58
@tedshroyer
Copy link

@hoffmang9 / @mariano54 should I close #3474 and open a bug instead for finding a long term solution?

@arvidn
Copy link
Contributor

arvidn commented May 3, 2021

I believe this issue in the rate limiter contributed to this problem, by pushing out and blocking other message types. #3518

It's probably a good idea to create a ticket for coming up with a long term solution to this. I imagine part of that is improving the rate limiter some as well (and hopefully simplify it).

wjblanke added a commit that referenced this pull request May 3, 2021
@wjblanke @mariano54
@hoffmang9 @arvidn
merge main into testnet7 (#3534)
3f1b9ff 
* 2 harvesting features (#3331)

* 3 harvesting features:
- Debug level shows the time for every quality lookup
- Warning level if takes longer than 5 seconds
- Allow configuration of plot loading interval (default 2 minutes)

* Comment out super logging

* Improve wallet consistency (#3305)

* Improve wallet consistency

* Improve CLI significantly, and fix self-tx balances

* Fix await

* Fix deadlock and test

* Remove spam.sh

* Changelog for 1.1.3 (#3345)

* Changelog for 1.1.3

* minor updates

* updates part 3

* Those engineers who don't update changelogs... :)

* Apologies to @Chida82 who added log rotate count! (#3369)

* Rust parse serialized (#3444)

* use rust implementation for finding length of a serialized clvm program

* bump clvm_rs version

* Don't retry respond_peers message (#3508)

Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com>
Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com>
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
wjblanke added a commit that referenced this pull request May 3, 2021
@wjblanke @mariano54
@hoffmang9 @arvidn
merge main to testnet7 (#3552)
a28c333 
* 2 harvesting features (#3331)

* 3 harvesting features:
- Debug level shows the time for every quality lookup
- Warning level if takes longer than 5 seconds
- Allow configuration of plot loading interval (default 2 minutes)

* Comment out super logging

* Improve wallet consistency (#3305)

* Improve wallet consistency

* Improve CLI significantly, and fix self-tx balances

* Fix await

* Fix deadlock and test

* Remove spam.sh

* Changelog for 1.1.3 (#3345)

* Changelog for 1.1.3

* minor updates

* updates part 3

* Those engineers who don't update changelogs... :)

* Apologies to @Chida82 who added log rotate count! (#3369)

* Rust parse serialized (#3444)

* use rust implementation for finding length of a serialized clvm program

* bump clvm_rs version

* Don't retry respond_peers message (#3508)

* don't increment counters for outgoing messages blocked by the rate limit. (#3518)

This was causing a problem where outbound messages, blocked by the rate limiter,
would still increment the counters as-if they had been sent. This, in turn,
could cause other message types to get blocked becuase the rate limiter thought
we had sent a lot of the other (blocked) message type.

Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com>
Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com>
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
wjblanke added a commit that referenced this pull request May 3, 2021
@wjblanke @mariano54
@hoffmang9 @arvidn @aqk
merge main into testnet7 (#3573)
dac6048 
* 2 harvesting features (#3331)

* 3 harvesting features:
- Debug level shows the time for every quality lookup
- Warning level if takes longer than 5 seconds
- Allow configuration of plot loading interval (default 2 minutes)

* Comment out super logging

* Improve wallet consistency (#3305)

* Improve wallet consistency

* Improve CLI significantly, and fix self-tx balances

* Fix await

* Fix deadlock and test

* Remove spam.sh

* Changelog for 1.1.3 (#3345)

* Changelog for 1.1.3

* minor updates

* updates part 3

* Those engineers who don't update changelogs... :)

* Apologies to @Chida82 who added log rotate count! (#3369)

* Rust parse serialized (#3444)

* use rust implementation for finding length of a serialized clvm program

* bump clvm_rs version

* Don't retry respond_peers message (#3508)

* don't increment counters for outgoing messages blocked by the rate limit. (#3518)

This was causing a problem where outbound messages, blocked by the rate limiter,
would still increment the counters as-if they had been sent. This, in turn,
could cause other message types to get blocked becuase the rate limiter thought
we had sent a lot of the other (blocked) message type.

* hide secret wallet key by default with 'chia keys show' (#3565)

Co-authored-by: Adam Kelly <aqk@aqk.im>

Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com>
Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com>
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: Adam Kelly <aqk@aqk.im>
@tedshroyer tedshroyer mentioned this pull request May 4, 2021
Closed
wjblanke added a commit that referenced this pull request May 4, 2021
@wjblanke @mariano54
@hoffmang9 @arvidn @aqk @almogdepaz
merge main to testnet7 (#3624)
efcf655 
* 2 harvesting features (#3331)

* 3 harvesting features:
- Debug level shows the time for every quality lookup
- Warning level if takes longer than 5 seconds
- Allow configuration of plot loading interval (default 2 minutes)

* Comment out super logging

* Improve wallet consistency (#3305)

* Improve wallet consistency

* Improve CLI significantly, and fix self-tx balances

* Fix await

* Fix deadlock and test

* Remove spam.sh

* Changelog for 1.1.3 (#3345)

* Changelog for 1.1.3

* minor updates

* updates part 3

* Those engineers who don't update changelogs... :)

* Apologies to @Chida82 who added log rotate count! (#3369)

* Rust parse serialized (#3444)

* use rust implementation for finding length of a serialized clvm program

* bump clvm_rs version

* Don't retry respond_peers message (#3508)

* don't increment counters for outgoing messages blocked by the rate limit. (#3518)

This was causing a problem where outbound messages, blocked by the rate limiter,
would still increment the counters as-if they had been sent. This, in turn,
could cause other message types to get blocked becuase the rate limiter thought
we had sent a lot of the other (blocked) message type.

* hide secret wallet key by default with 'chia keys show' (#3565)

Co-authored-by: Adam Kelly <aqk@aqk.im>

* brake if end of ses list (#3583)

* brake if end of ses list

* log

* throw if wp peak is not heavier

* comment

* handle sync from scratch

* comment

Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com>
Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com>
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: Adam Kelly <aqk@aqk.im>
Co-authored-by: Almog De Paz <almogdepaz@gmail.com>
wjblanke added a commit that referenced this pull request May 4, 2021
@wjblanke @mariano54
@hoffmang9 @arvidn @aqk @almogdepaz
merge main into testnet7 (#3710)
e195f08 
* 2 harvesting features (#3331)

* 3 harvesting features:
- Debug level shows the time for every quality lookup
- Warning level if takes longer than 5 seconds
- Allow configuration of plot loading interval (default 2 minutes)

* Comment out super logging

* Improve wallet consistency (#3305)

* Improve wallet consistency

* Improve CLI significantly, and fix self-tx balances

* Fix await

* Fix deadlock and test

* Remove spam.sh

* Changelog for 1.1.3 (#3345)

* Changelog for 1.1.3

* minor updates

* updates part 3

* Those engineers who don't update changelogs... :)

* Apologies to @Chida82 who added log rotate count! (#3369)

* Rust parse serialized (#3444)

* use rust implementation for finding length of a serialized clvm program

* bump clvm_rs version

* Don't retry respond_peers message (#3508)

* don't increment counters for outgoing messages blocked by the rate limit. (#3518)

This was causing a problem where outbound messages, blocked by the rate limiter,
would still increment the counters as-if they had been sent. This, in turn,
could cause other message types to get blocked becuase the rate limiter thought
we had sent a lot of the other (blocked) message type.

* hide secret wallet key by default with 'chia keys show' (#3565)

Co-authored-by: Adam Kelly <aqk@aqk.im>

* brake if end of ses list (#3583)

* brake if end of ses list

* log

* throw if wp peak is not heavier

* comment

* handle sync from scratch

* comment

* fix indentation issue in message send retry logic. factor out retry coroutine (#3629)

* Fix test (#3350)

* Mempool sorting and accept reverted pending transactions (#3683)

* Sort by fee/cost, and fix pending tx issue in reorgs

* Fix test name

* Bring the seen list size back to normal.

Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com>
Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com>
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: Adam Kelly <aqk@aqk.im>
Co-authored-by: Almog De Paz <almogdepaz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arvidn arvidn arvidn approved these changes

@Yostra Yostra Awaiting requested review from Yostra

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mariano54 @tedshroyer @arvidn