Add idempotence/retry behavior to ILP-over-HTTP (#567)

* docs: idempotence/retry behavior for ILP-over-HTTP

* fix: separate request id and idempotency key

* fix: error if no corresponding ILP Prepare

* fix: caveat for retry behavior, clarifications

* fix: clarify support for sync mode

Co-Authored-By: David Fuelling <sappenin@gmail.com>

* docs: reorganize ilp-over-http async mode

- reorganize for clarity
- clarify async mode is optional, sync mode is still supported
- clarify behavior of sender and receiver for retries
- use 202 Accepted for async response
- retrying ILP Prepares is optional and must be negotiated out of band
- add timeout after Prepare expries before purging old idempotency keys

* docs: address comments

- clarify why ILP Prepares are retried, and why it needs to be negotiated
- minor edits
- remove 409 retry behavior
- clarify when idempotence should be used

* docs(http): callback header, omit Prepare retries

* docs(http): add prefer header, fix examples

* fix(http): require request id, callback headers

Co-authored-by: David Fuelling <sappenin@gmail.com>

0035-ilp-over-http/0035-ilp-over-http.md

@@ -5,6 +5,7 @@ draft: 2
 ---
 
 # ILP Over HTTP
+
 > A bilateral communication protocol for server-to-server connections
 
 ## Motivation
@@ -15,7 +16,7 @@ Scaling Interledger infrastructure to handle large volumes of ILP packets requir
 
 In an ILP Over HTTP connection, both peers run HTTP servers with accessible HTTPS endpoints. When peering, the peers exchange their respective URLs, authentication tokens or TLS certificates, ILP addresses, and settlement-related details.
 
-Each ILP Prepare packet is sent as the body of an HTTP request to the peer's server endpoint. ILP Fulfill or Reject packets are returned as the body of the HTTP response.
+Each ILP Prepare packet is sent as the body of an HTTP request to the peer's server endpoint. ILP Fulfill or Reject packets are returned as the body of the HTTP response in synchronous mode, or sent in a separate HTTP request in asynchronous mode.
 
 ## Specification
 
@@ -25,26 +26,37 @@ This is a minimal protocol built on HTTP. HTTP/2 is HIGHLY RECOMMENDED for perfo
 
 Peers MAY use any standard HTTP authentication mechanism to authenticate incoming requests. TLS Client Certificates are RECOMMENDED between peers for security and performance, though bearer tokens such as JSON Web Tokens (JWTs) or Macaroons MAY be used instead. Basic authentication (username and password) is NOT RECOMMENDED, because of the additional delay introduced by securely hashing the password.
 
-### Request
+### Send ILP Prepare
+
+#### Request
 
 ```http
 POST /ilp HTTP/x.x
-Host: example.com
+Host: bob.example
 Accept: application/octet-stream
 Content-Type: application/octet-stream
 Authorization: Bearer zxcljvoizuu09wqqpowipoalksdflksjdgxclvkjl0s909asdf
+Prefer: respond-async
+Callback-Url: https://alice.example/incoming/ilp
+Request-Id: 42ee09c8-a6de-4ae3-8a47-4732b0cbb07b
 
 < Body: Binary OER-Encoded ILP Prepare Packet >
 ```
 
-**Field Details:**
+- **Path** &mdash; A connector MAY specify any HTTP path for their peer to send ILP packets to.
+- **Host Header** &mdash; The standard [HTTP Host Header](https://tools.ietf.org/html/rfc2616#section-14.23) indicating the domain of the HTTP server the request is sent to.
+- **Content-Type / Accept Headers** &mdash; MUST be set to `application/octet-stream`.
+- **Body** &mdash; ILP Prepare encoded using OER, as specified in [RFC 27: Interledger Protocol V4](./0027-interledger-protocol-4/0027-interledger-protocol-4.md).
+
+Asynchronous mode uses these additional headers:
 
-- **Path** - A connector MAY specify any HTTP path for their peer to send ILP packets to.
-- **Host Header** - The standard [HTTP Host Header](https://tools.ietf.org/html/rfc2616#section-14.23) indicating the domain of the HTTP server the Request is sent to.
-- **Content-Type / Accept Headers** - MUST be set to `application/octet-stream`.
-- **Body** - ILP Packet encoded using OER, as specified in [RFC 27: Interledger Protocol V4](./0027-interledger-protocol-4/0027-interledger-protocol-4.md).
+- **Prefer** &mdash; MUST be set to `respond-async`. If omitted, the reply behavior defaults to synchronous mode.
+- **Request Id Header** &mdash; UUIDv4 to uniquely identify this ILP Prepare, and correlate the corresponding ILP Fulfill/Reject.
+- **Callback URL Header** &mdash; Callback URL of the origin connector to send an asynchronous HTTP request with the ILP Fulfill/Reject. Required unless peers exchange the callback URL out-of-band.
 
-### Response
+#### Response
+
+In synchronous mode, the raw OER-encoded ILP Fulfill or Reject is returned within the body of the response:
 
 ```http
 HTTP/x.x 200 OK
@@ -53,7 +65,60 @@ Content-Type: application/octet-stream
 < Body: Binary OER-Encoded ILP Fulfill or Reject Packet >
 ```
 
-All ILP Packets MUST be returned with the HTTP status code `200: OK`.
+If the request includes a `Prefer: respond-async` header, the recipient handling the ILP Prepare SHOULD choose to handle the packet asynchronously and return the corresponding ILP Fulfill/Reject in a separate outgoing HTTP request.
+
+If the request is semantically valid and the recipient chooses to handle it asynchronously, they MUST respond immediately that the ILP Prepare is accepted for processing, even if the packet will ultimately be rejected:
+
+```http
+HTTP/x.x 202 Accepted
+```
+
+### Async ILP Fulfill/Reject Reply
+
+#### Request
+
+```http
+POST /incoming/ilp HTTP/x.x
+Host: alice.example
+Content-Type: application/octet-stream
+Authorization: Bearer zxcljvoizuu09wqqpowipoalksdflksjdgxclvkjl0s909asdf
+Request-Id: 42ee09c8-a6de-4ae3-8a47-4732b0cbb07b
+
+< Body: Binary OER-Encoded ILP Fulfill or Reject Packet >
+```
+
+- **Path** &mdash; HTTP path from the callback URL in the original request carrying the ILP Prepare.
+- **Host Header** &mdash; The standard [HTTP Host Header](https://tools.ietf.org/html/rfc2616#section-14.23) indicating the domain of the HTTP server the Request is sent to.
+- **Content-Type Header** &mdash; MUST be set to `application/octet-stream`.
+- **Request Id Header** &mdash; Request ID from the corresponding ILP Prepare, which is a UUIDv4, matching this reply to the original request.
+- **Body** &mdash; ILP Packet encoded using OER, as specified in [RFC 27: Interledger Protocol V4](./0027-interledger-protocol-4/0027-interledger-protocol-4.md).
+
+#### Response
+
+```http
+HTTP/x.x 200 OK
+```
+
+If the request ID doesn't correspond to an in-flight ILP Prepare, or a reply was already processed, the connector should ignore it and return an error:
+
+```http
+HTTP/x.x 400 Bad Request
+```
+
+#### Retry Behavior
+
+If the recipient of the ILP Fulfill/Reject responds with a `5xx` status or no HTTP response is received within a given timeout, the sender of the ILP Fulfill/Reject SHOULD retry sending the request.
+
+The sender of the ILP Fulfill/Reject MUST conclude retrying after receiving a response with a `2xx` status or `4xx` error.
+
+The sender of the ILP Fulfill/Reject SHOULD ensure there are multiple attempts to deliver the reply packet to the peer before the corresponding ILP Prepare expires.
+
+#### Idempotence
+
+An ILP Fulfill packet corresponds to a commitment which affects financial accounting balances. If an HTTP request carrying the ILP reply fails, such as due to a network connection error, retrying delivery of the ILP reply with [idempotence](https://en.wikipedia.org/wiki/Idempotence) can prevent balance inconsistencies between peers.
+
+If the sender of an ILP Prepare expects an asynchronous reply, they should only process the first ILP reply they receive corresponding to the in-flight ILP Prepare.
 
-An endpoint MAY return standard HTTP errors, including but not limited to: a malformed or unauthenticated request, rate limiting, or an unresponsive upstream service. Connectors SHOULD either retry the request, if applicable, or relay an ILP Reject packet back to the original sender with an appropriate [Final or Temporary error code](./0027-interledger-protocol-4/0027-interledger-protocol-4#error-codes). Server errors (status codes 500-599) SHOULD be translated into ILP Reject packets with `T00: Temporary Error` codes.
+### Error Handling
 
+An endpoint MAY return standard HTTP errors, including but not limited to: a malformed or unauthenticated request, rate limiting, or an unresponsive upstream service. Connectors SHOULD relay an ILP Reject packet back to the original sender with an appropriate [Final or Temporary error code](./0027-interledger-protocol-4/0027-interledger-protocol-4#error-codes). Server errors (status codes 500-599) SHOULD be translated into ILP Reject packets with `T00: Temporary Error` codes.