Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clened up 0.18 changelog for release #9497

Merged
merged 5 commits into from
Dec 13, 2022
Merged

Clened up 0.18 changelog for release #9497

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9ddfbcf
Add resource manager to changelog
BigLep Dec 13, 2022
477c97c
Added autogenerated TOC
BigLep Dec 13, 2022
e1f5cf5
Reodered sections and updated titles
BigLep Dec 13, 2022
861c3f8
Cleaned up content
BigLep Dec 13, 2022
5dcd9ad
Update docs/changelogs/v0.18.md
galargh Dec 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 93 additions & 76 deletions docs/changelogs/v0.18.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,26 +6,66 @@

Below is an outline of all that is in this release, so you get a sense of all that's included.

- [Kubo changelog v0.18](#kubo-changelog-v018)
- [v0.18.0](#v0180)
- [Overview](#overview)
- [🔦 Highlights](#-highlights)
- [(DAG-)JSON and (DAG-)CBOR Response Formats on Gateways](#dag-json-and-dag-cbor-response-formats-on-gateways)
- [🐎 Fast directory listings with DAG sizes](#-fast-directory-listings-with-dag-sizes)
- [Content Routing](#content-routing)
- [WebTransport enabled by default](#webtransport-enabled-by-default)
- [QUICv1 & WebTransport Config Migration](#quicv1--webtransport-config-migration)
- [Provider Record Republish and Expiration](#provider-record-republish-and-expiration)
- [Lowered `ConnMgr`](#lowered-connmgr)
- [Changelog](#changelog)
- [Contributors](#contributors)
<!-- TOC depthfrom:3 -->

- [Overview](#overview)
- [🔦 Highlights](#-highlights)
- [Content routing](#content-routing)
- [Default InterPlanetary Network Indexer](#default-interplanetary-network-indexer)
- [Increase provider record republish interval and expiration](#increase-provider-record-republish-interval-and-expiration)
- [Gateways](#gateways)
- [DAG-JSON and DAG-CBOR response formats](#dag-json-and-dag-cbor-response-formats)
- [🐎 Fast directory listings with DAG sizes](#-fast-directory-listings-with-dag-sizes)
- [QUIC and WebTransport](#quic-and-webtransport)
- [WebTransport enabled by default](#webtransport-enabled-by-default)
- [QUIC and WebTransport share a single port](#quic-and-webtransport-share-a-single-port)
- [Differentiating QUIC versions](#differentiating-quic-versions)
- [QUICv1 and WebTransport config migration](#quicv1-and-webtransport-config-migration)
- [Improving libp2p resource management integration](#improving-libp2p-resource-management-integration)
- [📝 Changelog](#-changelog)
- [👨‍👩‍👧‍👦 Contributors](#%E2%80%8D%E2%80%8D%E2%80%8D-contributors)
galargh marked this conversation as resolved.
Show resolved Hide resolved

<!-- /TOC -->

### 🔦 Highlights

#### (DAG-)JSON and (DAG-)CBOR Response Formats on Gateways
#### Content routing

##### Default InterPlanetary Network Indexer

Content routing is the process of discovering which peers provide a piece of content. Kubo has traditionally only supported [libp2p's implementation of Kademlia DHT](https://github.com/libp2p/specs/tree/master/kad-dht) for content routing.

Kubo can now bridge networks by including support for the [delegated routing HTTP API](https://github.com/ipfs/specs/pull/337). Users can compose content routers using the `Routing.Routers` config to pick content routers with different tradeoffs than a Kademlia DHT (e.g., high-performance and high-capacity centralized endpoints, dedicated Kademlia DHT nodes, routers with unique provider records, privacy-focused content routers).

One example is [InterPlanetary Network Indexers](https://github.com/ipni/specs/blob/main/IPNI.md#readme), which are HTTP endpoints that cache records from both the IPFS network and other sources such as web3.storage and Filecoin. This improves not only content availability by enabling Kubo to transparently fetch content directly from Filecoin storage providers, but also improves IPFS content routing latency by an order of magnitude and decreases resource consumption.

> *Note:* it's possible to retrieve content stored by Filecoin Storage Providers (SPs) from Kubo if the SPs service Bitswap requests. As of this release, some SPs are advertising Bitswap. You can follow the roadmap progress for IPNIs and Bitswap in SPs [here](https://www.starmaps.app/roadmap/github.com/protocol/bedrock/issues/1).

In this release, the default content router is changed from `dht` to `auto`. The `auto` router includes the IPFS DHT in addition to the [cid.contact](https://cid.contact) IPNI instance. In future releases, we plan to expand the functionality of `auto` to encompass automatic discovery of content routers, which will improve performance and content availability (for example, see [IPIP-342](https://github.com/ipfs/specs/pull/342)).

Previous behavior can be restored by setting `Routing.Type` to `dht`.

Alternative routing rules, including alternative IPNI endpoints, can be configured in `Routing.Routers` after setting `Routing.Type` to `custom`.

Learn more in the [`Routing` docs](https://github.com/ipfs/kubo/blob/master/docs/config.md#routing).

##### Increase provider record republish interval and expiration

Default `Reprovider.Interval` changed from 12h to 22h to match new defaults for the Provider Record Expiration (48h) in [go-libp2p-kad-dht v0.20.0](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.20.0).

The rationale for increasing this can be found in
[RFM 17: Provider Record Livenes Report](https://github.com/protocol/network-measurements/blob/master/results/rfm17-provider-record-liveness.md),
[kubo#9326](https://github.com/ipfs/kubo/pull/9326),
and the upstream DHT specifications at [libp2p/specs#451](https://github.com/libp2p/specs/pull/451).

Learn more in the [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#reprovider).

#### Gateways

##### (DAG-)JSON and (DAG-)CBOR response formats

Implemented [IPIP-328](https://github.com/ipfs/specs/pull/328) which adds support
to DAG-JSON and DAG-CBOR, as well as their non-DAG variants, to the gateway. Now,
for DAG-JSON and DAG-CBOR, as well as their non-DAG variants, to the gateway. Now,
CIDs that encode JSON, CBOR, DAG-JSON and DAG-CBOR objects can be retrieved, and
traversed thanks to the [special meaning of CBOR Tag 42](https://github.com/ipld/cid-cbor/).

Expand Down Expand Up @@ -74,83 +114,60 @@ $ curl "http://127.0.0.1:8080/ipfs/$DIR_CID?format=dag-json" | jq
}
```

#### 🐎 Fast directory listings with DAG sizes
##### 🐎 Fast directory listings with DAG sizes

Fast listings are now enabled for _all_ UnixFS directories: big and small.
There is no linear slowdown caused by reading size metadata from child nodes,
and the size of DAG representing child items is always present.

As an example, the CID
`bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm` represents UnixFS
directory with over 10k (10100) of files. Listing big directories was fast
`bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm` represents a UnixFS
directory with over 10k files. Listing big directories was fast
since Kubo 0.13, but in this release it will also include the size column.

#### Content Routing

Content routing is the process of discovering which peers provide a piece of content. Kubo has traditionally only supported [libp2p's implementation of Kademlia DHT](https://github.com/libp2p/specs/tree/master/kad-dht) for content routing.

Kubo can now bridge networks by including support for the [delegated routing HTTP API](https://github.com/ipfs/specs/pull/337). Users can compose content routers using the `Routing.Routers` config, to pick content routers with different tradeoffs than a Kademlia DHT (for example, high-performance and high-capacity centralized endpoints, dedicated Kademlia DHT nodes, routers with unique provider records, privacy-focused content routers, etc.).

One example is [InterPlanetary Network Indexers](https://github.com/ipni/specs/blob/main/IPNI.md#readme), which are HTTP endpoints that cache records from both the IPFS network and other sources such as web3.storage and Filecoin. This improves not only content availability by enabling Kubo to transparently fetch content directly from Filecoin storage providers, but also improves IPFS content routing latency by an order of magnitude and decreases resource consumption.
*Note:* it's possible to retrieve content stored by Filecoin Storage Providers (SPs) from Kubo if the SPs service Bitswap requests. As of this release, some SPs are advertising Bitswap. You can follow the roadmap progress for IPNIs and Bitswap in SPs [here](https://www.starmaps.app/roadmap/github.com/protocol/bedrock/issues/1).

In this release, the default content router is changed from `dht` to `auto`. The `auto` router includes the IPFS DHT in addition to the [cid.contact](https://cid.contact) IPNI instance. In future releases, we plan to expand the functionality of `auto` to encompass automatic discovery of content routers, which will improve performance and content availability (for example, see [IPIP-342](https://github.com/ipfs/specs/pull/342)).

Previous behavior can be restored by setting `Routing.Type` to `dht`.

Alternative routing rules, including alternative IPNI endpoints, can be configured in `Routing.Routers` after setting `Routing.Type` to `custom`.
#### QUIC and WebTransport

Learn more in [`Routing` docs](https://github.com/ipfs/kubo/blob/master/docs/config.md#routing).

#### WebTransport enabled by default

[WebTransport](https://github.com/libp2p/go-libp2p/issues/1717) is a new libp2p transport that [has been introduced in v0.16](v0.16.md#-webtransport-new-experimental-transport) that is based on top of QUIC and HTTP3.
##### WebTransport enabled by default
[WebTransport](https://docs.libp2p.io/concepts/transports/webtransport/) is a new libp2p transport that [was introduced in v0.16](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.16.md#-webtransport-new-experimental-transport) that is based on top of QUIC and HTTP3.

This allows browsers to contact Kubo nodes, so now instead of just serving requests for other system level applicative nodes, you can also serve requests directly to a browser.
For the full story see [connectivity.libp2p.io](https://connectivity.libp2p.io/).

##### WebTransport + QUIC on one single port

The new feature that allows us to ship WebTransport by default is that [go-libp2p now supports running WebTransport and QUIC transports on the same QUIC listener](https://github.com/libp2p/go-libp2p/issues/1759).
For the full story see [connectivity.libp2p.io](https://connectivity.libp2p.io/).

To use this feature, you just have to register two listen address on the same `/ipX/.../udp/XXX` prefix.
##### QUIC and WebTransport share a single port
WebTransport is enabled by default in part because [go-libp2p now supports running WebTransport and QUIC transports on the same QUIC listener](https://github.com/libp2p/go-libp2p/issues/1759). No additional port needs to be opened.

#### QUICv1 & WebTransport Config Migration
To use this feature, register two listen addresses on the same `/ipX/.../udp/XXX` prefix.

Go-libp2p now differentiate the first version of QUIC we implemented (and were using until then), `Draft-29`, and the ratified protocol in RFC9000, `QUICv1`.
This manifest as two different multiaddr components `/quic` (old Draft-29) and `/quic-v1`.
##### Differentiating QUIC versions
go-libp2p now differentiates the first version of QUIC that was originally implemented, `Draft-29`, from the ratified protocol in [RFC9000](https://www.rfc-editor.org/rfc/rfc9000.html), `QUICv1`.
This was done for performance (time to first byte) reasons as [outlined here](https://github.com/multiformats/multiaddr/issues/145).

Even tho Draft-29 and QUICv1 are similar they are not fully inter-compatible, go-libp2p do supports listening with both versions on one single listener.
This manifests as two different multiaddr components `/quic` (old Draft-29) and `/quic-v1`.
go-libp2p do supports listening with both QUIC versions on one single listener.
WebTransport has only supported QUICv1.
`/webtransport` now needs to be prefixed by a `/quic-v1` component instead of a `/quic` component.

`/webtransport` now also need to be prefixed by a `/quic-v1` instead of `/quic` component.
This has no protocol change, WebTransport only supports QUICv1 and were running on QUICv1 already.
Support for QUIC Draft-29 will be removed at some point in 2023 ([tracking issue](https://github.com/ipfs/kubo/issues/9496)). As a result, new deployements should use `/quic-v1` instead of `/quic`.

To support QUICv1 and WebTransport by default we run a new migration (n°`13`) which automatically add entries in addresses related fields in the config:
##### QUICv1 and WebTransport config migration
To support QUICv1 and WebTransport by default a new config migration (`v13`) is run which automatically adds entries in addresses-related fields:
- Replace all `/quic/webtransport` to `/quic-v1/webtransport`.
- For all `/quic` listener, keep the Draft-29 listener, and on the same ip and port, add `/quic-v1` and `/quic-v1/webtransport` listeners.

##### `/quic` (Draft-29) early-deprecation

We plan to remove support for QUIC Draft-29 in the mid to long term future.

You must not use `/quic` for new deployements and use `/quic-v1` instead.

#### Provider Record Republish and Expiration

Default `Reprovider.Interval` changed from 12h to 22h to match new defaults for the Provider Record Expiration (48h) in [go-libp2p-kad-dht v0.20.0](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.20.0).

Rationale for increasing this can be found in
[RFM 17: Provider Record Livenes Report](https://github.com/protocol/network-measurements/blob/master/results/rfm17-provider-record-liveness.md),
[kubo#9326](https://github.com/ipfs/kubo/pull/9326),
and the upstream DHT specifications at [libp2p/specs#451](https://github.com/libp2p/specs/pull/451).

Learn more: [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#reprovider)

#### Lowered `ConnMgr`

<!-- TODO: https://github.com/ipfs/kubo/pull/9483 -->


### Changelog

### Contributors
- For all `/quic` listeners, keep the Draft-29 listener, and on the same ip and port, add `/quic-v1` and `/quic-v1/webtransport` listeners.

#### Improving libp2p resource management integration
To help protect nodes from DoS (resource exhaustion) and eclipse attacks,
Kubo enabled the [go-libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager)
by default in [Kubo 0.17](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.17.md#libp2p-resource-management-enabled-by-default).

Introducing limits like this by default after the fact is tricky,
and various improvements have been made to improve the UX including:
1. [Dedicated docs concerning the resource manager integration](https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md). This is a great place to go to learn more or get your FAQs answered.
2. Increasing the default limits for the resource manager.
3. Enabling the [`Swarm.ConnMgr`](https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmconnmgr) by default and reducing it thresholds so it can intelligently prune connections in many cases before the indiscriminate resource manager kicks in.
4. Adjusted log messages and levels to make clear that the resource manager is likely doing your node a favor by bounding resources.
5. [Other miscellaneous config and command bugs reported by users](https://github.com/ipfs/kubo/issues/9442).

### 📝 Changelog

### 👨‍👩‍👧‍👦 Contributors