fix: content-type response header hints how to process response

packages/verified-fetch/README.md

@@ -126,7 +126,7 @@ const json = await resp.json()
 
 ### Custom content-type parsing
 
-By default, `@helia/verified-fetch` sets the `Content-Type` header as `application/octet-stream` - this is because the `.json()`, `.text()`, `.blob()`, and `.arrayBuffer()` methods will usually work as expected without a detailed content type.
+By default, if the response can be parsed as JSON, `@helia/verified-fetch` sets the `Content-Type` header as `application/json`, otherwise it sets it as `application/octet-stream` - this is because the `.json()`, `.text()`, `.blob()`, and `.arrayBuffer()` methods will usually work as expected without a detailed content type.
 
 If you require an accurate content-type you can provide a `contentTypeParser` function as an option to `createVerifiedFetch` to handle parsing the content type.
 
@@ -140,14 +140,198 @@ import { fileTypeFromBuffer } from '@sgtpooki/file-type'
 
 const fetch = await createVerifiedFetch({
  gateways: ['https://trustless-gateway.link'],
- routers: ['http://delegated-ipfs.dev'],
+ routers: ['http://delegated-ipfs.dev']
+}, {
  contentTypeParser: async (bytes) => {
    // call to some magic-byte recognition library like magic-bytes, file-type, or your own custom byte recognition
-   return fileTypeFromBuffer(bytes)?.mime
+   const result = await fileTypeFromBuffer(bytes)
+   return result?.mime
  }
 })
 ```
 
+### IPLD codec handling
+
+IPFS supports several data formats and `@helia/verified-fetch` attempts to abstract away some of the details.
+
+#### DAG-PB
+
+[DAG-PB](https://ipld.io/docs/codecs/known/dag-pb/) is the codec we are most likely to encounter, it is what [UnixFS](https://github.com/ipfs/specs/blob/main/UNIXFS.md) uses under the hood.
+
+##### Using the DAG-PB codec as a Blob
+
+```TypeScript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://Qmfoo')
+const blob = await res.blob()
+
+console.info(blob) // Blob { size: x, type: 'application/octet-stream' }
+```
+
+##### Using the DAG-PB codec as an ArrayBuffer
+
+```TypeScript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://Qmfoo')
+const buf = await res.arrayBuffer()
+
+console.info(buf) // ArrayBuffer { [Uint8Contents]: < ... >, byteLength: x }
+```
+
+##### Using the DAG-PB codec as a stream
+
+```TypeScript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://Qmfoo')
+const reader = res.body?.getReader()
+
+while (true) {
+  const next = await reader.read()
+
+  if (next?.done === true) {
+    break
+  }
+
+  if (next?.value != null) {
+    console.info(next.value) // Uint8Array(x) [ ... ]
+  }
+}
+```
+
+##### Content-Type
+
+When fetching `DAG-PB` data, the content type will be set to `application/octet-stream` unless a custom content-type parser is configured.
+
+#### JSON
+
+The JSON codec is a very simple codec, a block parseable with this codec is a JSON string encoded into a `Uint8Array`.
+
+##### Using the JSON codec
+
+```TypeScript
+import * as json from 'multiformats/codecs/json'
+
+const block = new TextEncoder().encode('{ "hello": "world" }')
+const obj = json.decode(block)
+
+console.info(obj) // { hello: 'world' }
+```
+
+##### Content-Type
+
+When the `JSON` codec is encountered, the `Content-Type` header of the response will be set to `application/json`.
+
+### DAG-JSON
+
+[DAG-JSON](https://ipld.io/docs/codecs/known/dag-json/) expands on the `JSON` codec, adding the ability to contain [CID](https://docs.ipfs.tech/concepts/content-addressing/)s which act as links to other blocks, and byte arrays.
+
+`CID`s and byte arrays are represented using special object structures with a single `"/"` property.
+
+Using `DAG-JSON` has two important caveats:
+
+1. Your `JSON` structure cannot contain an object with only a `"/"` property, as it will be interpreted as a special type.
+2. Since `JSON` has no technical limit on number sizes, `DAG-JSON` also allows numbers larger than `Number.MAX_SAFE_INTEGER`. JavaScript requires use of `BigInt`s to represent numbers larger than this, and `JSON.parse` does not support them, so precision will be lost.
+
+Otherwise this codec follows the same rules as the `JSON` codec.
+
+##### Using the DAG-JSON codec
+
+```TypeScript
+import * as dagJson from '@ipld/dag-json'
+
+const block = new TextEncoder().encode(`{
+  "hello": "world",
+  "cid": {
+    "/": "baeaaac3imvwgy3zao5xxe3de"
+  },
+  "buf": {
+    "/": {
+      "bytes": "AAECAwQ"
+    }
+  }
+}`)
+
+const obj = dagJson.decode(block)
+
+console.info(obj)
+// {
+// hello: 'world',
+// cid: CID(baeaaac3imvwgy3zao5xxe3de),
+// buf: Uint8Array(5) [ 0, 1, 2, 3, 4 ]
+// }
+```
+
+##### Content-Type
+
+When the `DAG-JSON` codec is encountered, the `Content-Type` header of the response will be set to `application/json`.
+
+`DAG-JSON` data can be parsed from the response by using the `.json()` function, which will return `CID`s/byte arrays as plain `{ "/": ... }` objects:
+
+```TypeScript
+import { verifiedFetch } from '@helia/verified-fetch'
+import * as dagJson from '@ipld/dag-json'
+
+const res = await verifiedFetch('ipfs://bafyDAGJSON')
+
+// either:
+const obj = await res.json()
+console.info(obj.cid) // { "/": "baeaaac3imvwgy3zao5xxe3de" }
+console.info(obj.buf) // { "/": { "bytes": "AAECAwQ" } }
+```
+
+Alternatively or it can be decoded using the `@ipld/dag-json` module and the `.arrayBuffer()` method, in which case you will get `CID` objects and `Uint8Array`s:
+
+```TypeScript
+import { verifiedFetch } from '@helia/verified-fetch'
+import * as dagJson from '@ipld/dag-json'
+
+const res = await verifiedFetch('ipfs://bafyDAGJSON')
+
+// or:
+const obj = dagJson.decode(new Uint8Array(await res.arrayBuffer()))
+console.info(obj.cid) // CID(baeaaac3imvwgy3zao5xxe3de)
+console.info(obj.buf) // Uint8Array(5) [ 0, 1, 2, 3, 4 ]
+```
+
+#### DAG-CBOR
+
+[DAG-CBOR](https://ipld.io/docs/codecs/known/dag-cbor/) uses the [Concise Binary Object Representation](https://cbor.io/) format for serialization instead of JSON.
+
+This supports more datatypes in a safer way than JSON and is smaller on the wire to boot so is usually preferable to JSON or DAG-JSON.
+
+##### Content-Type
+
+Not all data types supported by `DAG-CBOR` can be successfully turned into JSON and back.
+
+When a decoded block can be round-tripped to JSON, the `Content-Type` will be set to `application/json`. In this case the `.json()` method on the `Response` object can be used to obtain an object representation of the response.
+
+When it cannot, the `Content-Type` will be `application/octet-stream` - in this case the `@ipld/dag-json` module must be used to deserialize the return value from `.arrayBuffer()`.
+
+##### Detecting JSON-safe DAG-CBOR
+
+If the `Content-Type` header of the response is `application/json`, the `.json()` method may be used to access the response body in object form, otherwise the `.arrayBuffer()` method must be used to decode the raw bytes using the `@ipld/dag-cbor` module.
+
+```TypeScript
+import { verifiedFetch } from '@helia/verified-fetch'
+import * as dagCbor from '@ipld/dag-cbor'
+
+const res = await verifiedFetch('ipfs://bafyDagCborCID')
+let obj
+
+if (res.headers.get('Content-Type') === 'application/json') {
+  // DAG-CBOR data can be safely decoded as JSON
+  obj = await res.json()
+} else {
+  // response contains non-JSON friendly data types
+  obj = dagCbor.decode(new Uint8Array(await res.arrayBuffer()))
+}
+
+console.info(obj) // ...
+```
+
 ## Comparison to fetch
 
 This module attempts to act as similarly to the `fetch()` API as possible.

packages/verified-fetch/package.json

@@ -142,8 +142,6 @@
   },
   "dependencies": {
     "@helia/block-brokers": "^2.0.1",
-    "@helia/dag-cbor": "^3.0.0",
-    "@helia/dag-json": "^3.0.0",
     "@helia/http": "^1.0.1",
     "@helia/interface": "^4.0.0",
     "@helia/ipns": "^6.0.0",
@@ -155,12 +153,15 @@
     "@ipld/dag-pb": "^4.0.8",
     "@libp2p/interface": "^1.1.2",
     "@libp2p/peer-id": "^4.0.5",
+    "cborg": "^4.0.9",
     "hashlru": "^2.3.0",
     "ipfs-unixfs-exporter": "^13.5.0",
     "multiformats": "^13.0.1",
     "progress-events": "^1.0.0"
   },
   "devDependencies": {
+    "@helia/dag-cbor": "^3.0.0",
+    "@helia/dag-json": "^3.0.0",
     "@helia/utils": "^0.0.1",
     "@libp2p/logger": "^4.0.5",
     "@libp2p/peer-id-factory": "^4.0.5",
@@ -171,6 +172,7 @@
     "datastore-core": "^9.2.8",
     "helia": "^4.0.1",
     "it-last": "^3.0.4",
+    "it-to-buffer": "^4.0.5",
     "magic-bytes.js": "^1.8.0",
     "sinon": "^17.0.1",
     "sinon-ts": "^2.0.0",