From 37917b41a4ea1b6bc293b22c03a02a9ef5ab447f Mon Sep 17 00:00:00 2001 From: Alex <445306+anorth@users.noreply.github.com> Date: Mon, 15 Feb 2021 16:07:20 +1100 Subject: [PATCH 1/6] Create IpfsGatewayHttpApi.md --- proposals/IpfsGatewayHttpApi.md | 134 ++++++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 proposals/IpfsGatewayHttpApi.md diff --git a/proposals/IpfsGatewayHttpApi.md b/proposals/IpfsGatewayHttpApi.md new file mode 100644 index 00000000..bdd97d2b --- /dev/null +++ b/proposals/IpfsGatewayHttpApi.md @@ -0,0 +1,134 @@ +# IPFS Gateway HTTP API +Authors: @anorth +Initial PR: TBD + +## Purpose & impact +#### Background & intent +_Describe the desired state of the world after this project? Why does that matter?_ + +Today, web and other clients must embed a full IPFS node implementation in order to interact with IPFS +on any level deeper than fetching flattened UnixFS file data from a gateway. +This embedding is heavyweight, and complex. +Maintaining a fully-featured js-ipfs implementation, as the only integration option, is a lot of work. +Delegating to a backend IPFS node directly using the RPC-style API is high friction. + +After this project, an HTTP Gateway API supports direct interaction with IPLD data using standard web/REST semantics, +but abstracts away the complexity of native protocol interactions (DHT, bitswap, graphsync) from clients. +A gateway client can fetch and verify arbitrary IPLD DAGs. +The API is application-level (DAG-at-a-time rather than node-at-a-time), maximising ease and minimising round-trips. +Scripts can fetch DAGs with curl to feed into other tools. + +Authenticated extensions to the API can push/pin IPLD data to the gateway for hosting. +The PL-operated gateway is read-only, but a Dapp provider can host their own for Dapp clients to interact with, or embed/extend it with application logic. + +"No API is the best API". +Related ideas are in [Verifiable HTTP Gateway Responses](https://github.com/ipfs/in-web-browsers/issues/128), +[CAR export from HTTP Gateways](https://github.com/ipfs/in-web-browsers/issues/170). + +#### Assumptions & hypotheses +_What must be true for this project to matter?_ + +- Dapp developers want to use IPFS, and there are lots of them +- Dapps and other applications need a more complex data model than UnixFS files +- Dapps and other integrations would find HTTP easier to interact with than low-level protocols +- Embedding a full go/js-ipfs node is a significant barrier to client use +- go/js-ipfs is slower and more difficult to interact with than an HTTP API +- Dapp developers are satisfied with the imperfect decentralization of using a gateway as an intermediate step. +- Having the same, more powerful HTTP API on localhost removes obstacles towards a deeper interaction with vendors like Brave + +#### User workflow example +_How would a developer or user use this new capability?_ + +A dapp developer builds a decentralized, semi-permissionless photo-sharing app. +They have a server that authenticates users but the rest of application logic is in the JS web client. +They embed an IPFS gateway in their server, which pins content posted by authenticated users +(the authentication protects the cost of pinning and third-party precache request abuse). +Their web app fetches DAGs describing the social graph, image metadata etc from HTTP API, as well as flattened files for the images themselves. +It’s fast, like a consumer app should be. + +#### Impact +_How directly important is the outcome to web3 dev stack product-market fit?_ + +🔥🔥🔥. If Dapp development is a strong market segment, but adoption is difficult, then reducing barriers to use is of highest importance. + +#### Leverage +_How much would nailing this project improve our knowledge and ability to execute future projects?_ + +🎯🎯 . The rating of 2 is motivated by the benefits of more rapid prototyping and iteration with low-friction APIs. +If we remove the need for learning intricate, non-intuitive HTTP API and leverage HTTP semantics so that JS “fetch” or curl is all you need, +we may see many more people experimenting with IPFS. +The [existing API](https://docs.ipfs.io/reference/http/api/) was not designed to be exposed on the web and introduces +significant overhead for people who want to use IPFS in something more than read-only mode. + +Success could result in freeing up significant resources from maintaining and developing js-ipfs as the only client integration point in the short term. + +#### Confidence +_How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. + +Very Low to Low. I don’t have the market data or direct user feedback that may support a higher confidence here. + +@lidel suggests they could provide a list of issues related to existing API being overly complex, +and then js-ipfs-http-client acting as a programmatic interface to cover those shortcomings hurting us even more +(breaking API changes, maintenance cost, people confused why they can't just use regular XHR/fetch and need a dedicated client lib). + + +## Project definition +#### Brief plan of attack + +- Design RESTful HTTP semantics on top of content-paths +- Settle on appropriate serial representation of DAGs (e.g. CAR, DAR, something else). +- Implement read-only API in HTTP gateway, and deploy it +- Implement JS client library: request wrappers and IPLD decoding/manipulation +- Implement publishing API (ie. HTTP POST), but deny from our public gateways +- Modularise HTTP gateway to permit dapp devs to embed it their back-ends + +#### What does done look like? +_What specific deliverables should completed to consider this project done?_ + +Our gateway serves arbitrary DAGs over simple HTTP requests. +Web-apps can fetch, manipulate, and render IPLD data with the standard JS XHR or [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch) without the need for any client library. +Backends that embed/extend the gateway also support publishing and pinning arbitrary IPLD data. + +#### What does success look like? +_Success means impact. How will we know we did the right thing?_ + +Dapp and other developers are creating clients that interact with IPLD data natively, with most application logic implemented client-side. +Our gateway metrics show growth in use of IPLD data. +Some dapps are running their own backends embedding a gateway, to support pinning data critical to their application. +Infura and others provide hosting services for gateways, as well as nodes. + +#### Counterpoints & pre-mortem +_Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ + +- Hypotheses about Dapps being an important market, or HTTP being a more accessible API might prove false +- API design could get stuck in a morass of discussion about standards +- Hard-core decentralization people object to the imperfect decentralization of using a gateway + +#### Alternatives +_How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_ + +- Just add more RPCs for DAG traversal to go-ipfs, that Infura nodes can then serve. Expose these through JS and other client libraries. +- Our gateways provide the read-only go-ipfs RPC API to the public, and establish that as standard +- Amazing technical work so that embedding js-ipfs is both easier and more performant than using an HTTP API + +#### Dependencies/prerequisites + +- JS-IPLD libraries capable of core encodings and IPLD representation +- IPLD selectors (for non-trivial DAG traversal) +- (For write path) Gateway<->go-ipfs write & pin integration + +#### Future opportunities + + +- Filecoin Gateway API: the same thing but for Filecoin blockchain state and deals + +## Required resources + +#### Effort estimate +Medium effort (3-5 weeks for a small team). +Some uncertainty until the API design is complete, but low uncertainty in execution thereafter. + +#### Roles / skills needed +- REST API design +- Backend (Go) development + From 1a5f975793d95990268998d927a8bc3126ef3637 Mon Sep 17 00:00:00 2001 From: Alex <445306+anorth@users.noreply.github.com> Date: Tue, 16 Feb 2021 16:37:31 +1100 Subject: [PATCH 2/6] Link to PR from body --- proposals/IpfsGatewayHttpApi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/IpfsGatewayHttpApi.md b/proposals/IpfsGatewayHttpApi.md index bdd97d2b..15bb1ff5 100644 --- a/proposals/IpfsGatewayHttpApi.md +++ b/proposals/IpfsGatewayHttpApi.md @@ -1,6 +1,6 @@ # IPFS Gateway HTTP API Authors: @anorth -Initial PR: TBD +Initial PR: https://github.com/protocol/web3-dev-team/pull/1 ## Purpose & impact #### Background & intent From a25ce2be27238141c271b1dbc1519b4f28efb2ea Mon Sep 17 00:00:00 2001 From: Alex <445306+anorth@users.noreply.github.com> Date: Wed, 17 Feb 2021 15:51:49 +1100 Subject: [PATCH 3/6] Link Writeable Gateway experiment Co-authored-by: Marcin Rataj --- proposals/IpfsGatewayHttpApi.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/IpfsGatewayHttpApi.md b/proposals/IpfsGatewayHttpApi.md index 15bb1ff5..ed9a4b9e 100644 --- a/proposals/IpfsGatewayHttpApi.md +++ b/proposals/IpfsGatewayHttpApi.md @@ -23,7 +23,7 @@ The PL-operated gateway is read-only, but a Dapp provider can host their own for "No API is the best API". Related ideas are in [Verifiable HTTP Gateway Responses](https://github.com/ipfs/in-web-browsers/issues/128), -[CAR export from HTTP Gateways](https://github.com/ipfs/in-web-browsers/issues/170). +[CAR export from HTTP Gateways](https://github.com/ipfs/in-web-browsers/issues/170), [Writable Gateway experiment](https://discuss.ipfs.io/t/writeable-http-gateways/210?u=lidel). #### Assumptions & hypotheses _What must be true for this project to matter?_ @@ -131,4 +131,3 @@ Some uncertainty until the API design is complete, but low uncertainty in execut #### Roles / skills needed - REST API design - Backend (Go) development - From 7d1d5a0602c33053b61f4fa200016fa2cd8499d1 Mon Sep 17 00:00:00 2001 From: Alex <445306+anorth@users.noreply.github.com> Date: Wed, 17 Feb 2021 15:52:11 +1100 Subject: [PATCH 4/6] Note IoT usage Co-authored-by: Marcin Rataj --- proposals/IpfsGatewayHttpApi.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/IpfsGatewayHttpApi.md b/proposals/IpfsGatewayHttpApi.md index ed9a4b9e..2eeb4617 100644 --- a/proposals/IpfsGatewayHttpApi.md +++ b/proposals/IpfsGatewayHttpApi.md @@ -35,6 +35,7 @@ _What must be true for this project to matter?_ - go/js-ipfs is slower and more difficult to interact with than an HTTP API - Dapp developers are satisfied with the imperfect decentralization of using a gateway as an intermediate step. - Having the same, more powerful HTTP API on localhost removes obstacles towards a deeper interaction with vendors like Brave +- Low-powered IoT devices can leverage gateways as less expensive way of moving data around, with true p2p kicking in only in LAN or as a fallback when centralized gateways are down. #### User workflow example _How would a developer or user use this new capability?_ From abba39f2775b20da449a50814932291cde8413a8 Mon Sep 17 00:00:00 2001 From: Alex <445306+anorth@users.noreply.github.com> Date: Wed, 17 Feb 2021 15:52:37 +1100 Subject: [PATCH 5/6] Done includes access control Co-authored-by: Marcin Rataj --- proposals/IpfsGatewayHttpApi.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/IpfsGatewayHttpApi.md b/proposals/IpfsGatewayHttpApi.md index 2eeb4617..9bfefcdb 100644 --- a/proposals/IpfsGatewayHttpApi.md +++ b/proposals/IpfsGatewayHttpApi.md @@ -89,6 +89,7 @@ _What specific deliverables should completed to consider this project done?_ Our gateway serves arbitrary DAGs over simple HTTP requests. Web-apps can fetch, manipulate, and render IPLD data with the standard JS XHR or [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch) without the need for any client library. Backends that embed/extend the gateway also support publishing and pinning arbitrary IPLD data. +Built-in mechanism for access control exist and makes it easy to limit POST/PUT/pinning capability to specific origins. #### What does success look like? _Success means impact. How will we know we did the right thing?_ From cae3c13f5be5462414097cc156916ee82d8d069e Mon Sep 17 00:00:00 2001 From: Alex <445306+anorth@users.noreply.github.com> Date: Wed, 17 Feb 2021 16:01:56 +1100 Subject: [PATCH 6/6] Note HTTP streaming limitations --- proposals/IpfsGatewayHttpApi.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/proposals/IpfsGatewayHttpApi.md b/proposals/IpfsGatewayHttpApi.md index 9bfefcdb..d31beace 100644 --- a/proposals/IpfsGatewayHttpApi.md +++ b/proposals/IpfsGatewayHttpApi.md @@ -1,5 +1,5 @@ # IPFS Gateway HTTP API -Authors: @anorth +Authors: @anorth, @lidel Initial PR: https://github.com/protocol/web3-dev-team/pull/1 ## Purpose & impact @@ -50,12 +50,13 @@ It’s fast, like a consumer app should be. #### Impact _How directly important is the outcome to web3 dev stack product-market fit?_ -🔥🔥🔥. If Dapp development is a strong market segment, but adoption is difficult, then reducing barriers to use is of highest importance. +If Dapp development is a strong market segment, but adoption is difficult, then reducing barriers to use is very important. +This API may not be the one ultimately used by complex projects, but an entry point that's powerful enough for a great many simple cases. #### Leverage _How much would nailing this project improve our knowledge and ability to execute future projects?_ -🎯🎯 . The rating of 2 is motivated by the benefits of more rapid prototyping and iteration with low-friction APIs. +More rapid prototyping and iteration with low-friction APIs (both us and community). If we remove the need for learning intricate, non-intuitive HTTP API and leverage HTTP semantics so that JS “fetch” or curl is all you need, we may see many more people experimenting with IPFS. The [existing API](https://docs.ipfs.io/reference/http/api/) was not designed to be exposed on the web and introduces @@ -102,9 +103,10 @@ Infura and others provide hosting services for gateways, as well as nodes. #### Counterpoints & pre-mortem _Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ -- Hypotheses about Dapps being an important market, or HTTP being a more accessible API might prove false -- API design could get stuck in a morass of discussion about standards -- Hard-core decentralization people object to the imperfect decentralization of using a gateway +- Hypotheses about Dapps being an important market, or HTTP being a more accessible API might prove false. +- API design could get stuck in a morass of discussion about standards. +- Hard-core decentralization people object to the imperfect decentralization of using a gateway. +- This API will likely be limited by HTTP capabilities: request/response pairs or possibly unidirectional streaming. May limit upload/download sizes, use of pubsub etc. Deeper IPFS integration may require an RPC-stype websocket API. If the capabilities of HTTP are too limiting for any project, this won't have much impact. #### Alternatives _How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_