New Product proposal: IPFS Implementation feature parity checker

[SgtPooki]

Introduction

This is a work-in-progress proposal for creating a new product/tool that would benefit the IPFS ecosystem.

The idea is to create a tool that acts as a mashup of ipfs/interop & ipfs-shipyard/pinning-service-compliance where IPFS implementations (js-ipfs, iroh, kubo) can be displayed. Its intent wouldn't be to ensure interoperability but display interoperability gaps.

Goal

The goal of this new product/feature is multi-purpose:

1. A single location where all IPFS implementations and their functionality can be seen
2. Encourage/foster the creation, and performance improvements, of the different implementations
3. Support users' awareness of different implementations to simplify decision-making when deciding which implementation may be right for them
4. Encourage the adoption of certain features/functionality by an implementation via pressure from the community

Things that we shouldn't cover

• interoperability tests - ipfs/interop is a fairly large beast, and one thing I don't want this product to become is another huge maintenance burden. interoperability tests and insurance should remain in ipfs/interop, where they belong.
• infantile implementations - Implementations that do not pass a minimum bar (to be set) should not be included. We should have a minimum feature set that implementations must support to be included in the list. Without this barrier to entry, it would be too easy for this tool to become a list of incomplete and unmaintained implementations
• features that are too low-level - Displaying the support of different block encodings or other minutia could easily bog down the usefulness of this feature. The idea is not to cover every single thing an implementation can do, but instead to display which implementations support the most up-to-date and recommended things.
  • i.e. don't show nuance of dag-* support. Show that dag-cbor (or whatever is currently recommended) blocks can be added, removed, etc..

Proposal (TBD)

A lot of the work for this tool already exists elsewhere, so I don't want to duplicate that work. Instead, we should utilize existing tools to perform the functionality testing, and instead, focus this tool on rendering the resulting pass/fail for displayed features.

How we could consolidate the existing tests without duplicating work still needs to be fleshed out, but I wanted to ensure I wrote down my thoughts on this before investing too heavily.

Additional thoughts

• We could display benchmarking results to facilitate community pressure on certain implementations to speed up features & functionality that are slower than other implementations, but this could easily be its own tool.
• We may want to expand this tool to support minutia (dag-* and other microcosms of functionality) in the future, but I don't think we should start there.
• We could systematize ipfs/interop to be the foundation for this product, and make it easier to add implementations and feature tests.

[SgtPooki]

@lidel it would be useful to get your thoughts on how this could overlap with some of the specs work you're doing.

[lidel]

Hm.. need to decide on the resolution we want to operate at.

Something like https://libp2p.io/implementations/ with discrete features such as

• "Block" "CAR" "libp2p" "DCUtR" "Public DHT Client/Server aka kad 1.0.0", "MDSN", "TCP" "WebTransport" "QUIC", "Bitswap 1.x.x" "UnixFS", "UnixFS 1.5. Metadata" "UnixFS HAMT reading|auto-sharding", "DAG-JSON", "DAG-CBOR", "DNSLink", "IPNS V1", "IPNS V2 (Extensible Data Records)", "Gateway Path|Subdomain|DNSLink" "Delegated Routing with Reframe" etc etc

per implementation ?

• Is the idea here to create end-user presentation like https://caniuse.com we could include on https://ipfs.tech/implementations and link to from https://docs.ipfs.io/basics/ipfs-implementations/?
  • A simple version sounds doable, for MVP we don't need anything beyond static page that implementations can PR against when the close gaps.
  • This would be a really, really valuable thing to add to ipfs.tech website.
• A more like extreme low level version of constantly running test suite akin https://web-platform-tests.org / https://wpt.fyi/results/ feels premature.
  • Creating something like WPT for IPFS will be Herculean effort, requires both solid specs and implementation-agnostic testing tooling which we are still lacking. Future – sure!. I think for now we should focus on something similar to https://libp2p.io/implementations/ first.

[SgtPooki]

In gui triage, Lidel said that Robin Berjon will be in Lisbon, and he worked on web-platform-tests, so we could pick his brain during IPFS Camp.

[darobin]

I agree with @lidel that a good first step is something static that implementations can PR (we probably want them to PR some data rather than an HTML page directly, but that's a detail).

Building something like WPT certainly takes time, even if we will probably never need to get to the 1.8m tests that they have. All that the initial version did was bring together a few thousand tests in a central place and unified test runner that crashed browsers more often than not. There was a fair bit of time before that led to results being collected and reported on. I suspect we'll have to start with a similar MVP!