Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify libp2p ResourceMgr computed defaults (no System.StreamsInbound limits and allow more System.ConnsInbound per MB) #9587

Closed
wants to merge 6 commits into from
Closed

Simplify libp2p ResourceMgr computed defaults (no System.StreamsInbound limits and allow more System.ConnsInbound per MB) #9587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1ad48df
Showing how want to adjust the default resource manager limits
BigLep Jan 25, 2023
af03df3
Add limit on System.ConnsInbound
BigLep Jan 26, 2023
d9156b6
Update libp2p-resource-management.md
BigLep Jan 26, 2023
2b4bc9a
Create v0.18.1
BigLep Jan 26, 2023
d3a48e9
Updating based on peer feedback
BigLep Jan 26, 2023
92f6823
Update v0.18.1
BigLep Jan 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 31 additions & 43 deletions core/node/libp2p/rcmgr_defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ var noLimitIncrease = rcmgr.BaseLimitIncrease{
// This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled

// createDefaultLimitConfig creates LimitConfig to pass to libp2p's resource manager.
// The defaults follow the documentation in docs/config.md.
// The defaults follow the documentation in docs/libp2p-resource-management.md.
// Any changes in the logic here should be reflected there.
func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) {
maxMemoryDefaultString := humanize.Bytes(uint64(memory.TotalMemory()) / 4)
Expand All @@ -53,8 +53,10 @@ func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error)
if err != nil {
return rcmgr.LimitConfig{}, err
}
maxMemory = int64(maxMemory)
maxMemoryMB = maxMemory / (1024 * 1024)

numFD := cfg.ResourceMgr.MaxFileDescriptors.WithDefault(int64(fd.GetNumFDs()) / 2)
maxFD := int(cfg.ResourceMgr.MaxFileDescriptors.WithDefault(int64(fd.GetNumFDs()) / 2))

// We want to see this message on startup, that's why we are using fmt instead of log.
fmt.Printf(`
Expand All @@ -67,63 +69,49 @@ Run 'ipfs swarm limit all' to see the resulting limits.

`, maxMemoryString, numFD)

// At least as of 2023-01-25, it's possible to open a connection that
// doesn't for any memory usage with the libp2p Resource Manager/Accountant
// (see https://github.com/libp2p/go-libp2p/issues/2010#issuecomment-1404280736).
// As a result, we can't curretly rely on Memory limits to full protect us.
// Until https://github.com/libp2p/go-libp2p/issues/2010 is addressed,
// we take a proxy now of restricting to 1 inbound connection per MB.
// Note: this is more generous than go-libp2p's default autoscaled limits which do
// 64 connections per 1GB
// (see https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L357 ).
systemConnsInbound = 1 * maxMemoryMB

scalingLimitConfig := rcmgr.ScalingLimitConfig{
SystemBaseLimit: rcmgr.BaseLimit{
Memory: int64(maxMemory),
FD: int(numFD),
Memory: maxMemory,
FD: maxFD,

ConnsInbound: systemConnsInbound,

// By default, we just limit connections on the inbound side.
Conns: bigEnough,
ConnsInbound: rcmgr.DefaultLimits.SystemBaseLimit.ConnsInbound, // same as libp2p default
ConnsOutbound: bigEnough,

// We limit streams since they not only take up memory and CPU.
// The Memory limit protects us on the memory side,
// but a StreamsInbound limit helps protect against unbound CPU consumption from stream processing.
Streams: bigEnough,
StreamsInbound: rcmgr.DefaultLimits.SystemBaseLimit.StreamsInbound,
StreamsInbound: bigEnough,
StreamsOutbound: bigEnough,
},
// Most limits don't see an increase because they're already infinite/bigEnough or at their max value.
// The values that should scale based on the amount of memory allocated to libp2p need to increase accordingly.
SystemLimitIncrease: rcmgr.BaseLimitIncrease{
Memory: 0,
FDFraction: 0,

Conns: 0,
ConnsInbound: rcmgr.DefaultLimits.SystemLimitIncrease.ConnsInbound,
ConnsOutbound: 0,

Streams: 0,
StreamsInbound: rcmgr.DefaultLimits.SystemLimitIncrease.StreamsInbound,
StreamsOutbound: 0,
},
SystemLimitIncrease: noLimitIncrease,

// Transient connections won't cause any memory to accounted for by the resource manager.
// Only established connections do.
// As a result, we can't rely on System.Memory to protect us from a bunch of transient connection being opened.
// We limit the same values as the System scope, but only allow the Transient scope to take 25% of what is allowed for the System scope.
TransientBaseLimit: rcmgr.BaseLimit{
BigLep marked this conversation as resolved.
Show resolved Hide resolved
Memory: rcmgr.DefaultLimits.TransientBaseLimit.Memory,
FD: rcmgr.DefaultLimits.TransientBaseLimit.FD,
Memory: maxMemory / 4,
FD: maxFD / 4,

ConnsInbound: systemConnsInbound / 4,

Conns: bigEnough,
ConnsInbound: rcmgr.DefaultLimits.TransientBaseLimit.ConnsInbound,
ConnsOutbound: bigEnough,

Streams: bigEnough,
StreamsInbound: rcmgr.DefaultLimits.TransientBaseLimit.StreamsInbound,
StreamsInbound: bigEnough,
StreamsOutbound: bigEnough,
},

TransientLimitIncrease: rcmgr.BaseLimitIncrease{
Memory: rcmgr.DefaultLimits.TransientLimitIncrease.Memory,
FDFraction: rcmgr.DefaultLimits.TransientLimitIncrease.FDFraction,

Conns: 0,
ConnsInbound: rcmgr.DefaultLimits.TransientLimitIncrease.ConnsInbound,
ConnsOutbound: 0,

Streams: 0,
StreamsInbound: rcmgr.DefaultLimits.TransientLimitIncrease.StreamsInbound,
StreamsOutbound: 0,
},
TransientLimitIncrease: noLimitIncrease,

// Lets get out of the way of the allow list functionality.
// If someone specified "Swarm.ResourceMgr.Allowlist" we should let it go through.
Expand Down
36 changes: 36 additions & 0 deletions docs/changelogs/v0.18.1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Kubo changelog v0.18

## v0.18.1
Copy link
Member

@lidel lidel Jan 26, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we dont want a separate file, let's move this to docs/changelogs/v0.18, like we did with 0.13.1.

@ajnavarro I've already added my changes to 0.18.1 section in #9543 which is in master now, feel free to reuse


### Overview

Below is an outline of all that is in this release, so you get a sense of all that's included.

<!-- TOC depthfrom:3 -->

- [Overview](#overview)
- [🔦 Highlights](#-highlights)
- [Improving libp2p resource management integration](#improving-libp2p-resource-management-integration)
- [📝 Changelog](#-changelog)
- [👨‍👩‍👧‍👦 Contributors](#-contributors)

<!-- /TOC -->

### 🔦 Highlights

#### Improving libp2p resource management integration
This builds on the default protection nodes get against DoS (resource exhaustion) and eclipse attacks
with the [go-libp2p Network Resource Manager/Accountant](https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md)
that was fine-tuned in [Kubo 0.18](https://github.com/ipfs/kubo/blob/biglep/resource-manager-example-of-what-want/docs/changelogs/v0.18.md#improving-libp2p-resource-management-integration).

Adding default hard-limits from the Resource Manager/Accountant after the fact is tricky,
and some additional improvements have been made to improve the [computed defaults](https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md#computed-default-limits).
As much as possible, the aim is for a user to only think about how much memory they want to bound libp2p to,
and not need to think about translating that to hard numbers for connections, streams, etc.
More updates are likely in future Kubo releases, but with this release:
1. ``System.StreamsInbound`` is no longer bounded directly
2. ``System.ConnsInbound``, ``Transient.Memory``, ``Transiet.ConnsInbound`` have higher default computed values.

### 📝 Changelog

### 👨‍👩‍👧‍👦 Contributors
4 changes: 1 addition & 3 deletions docs/libp2p-resource-management.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,7 @@ The reason these scopes are chosen is because:

Within these scopes, limits are just set on
[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory),
[file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections),
and [*inbound* streams](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#streams).
[file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), and [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections).
Limits are set based on the `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` inputs above.

There are also some special cases where minimum values are enforced.
Expand All @@ -89,7 +88,6 @@ These become the [active limits](#how-does-one-see-the-active-limits).

While `Swarm.ResourceMgr.Limits` can be edited directly, it is also possible to use `ipfs swarm limit` command to inspect and tweak specific limits at runtime.


To see all resources that are close to hitting their respective limit:

```console
Expand Down