chore: use raw nodes for leaf data

[achingbrain]

go uses raw unixfs nodes for leaf data whereas this module uses file nodes - that causes CIDs to differ between the two implementations. I've added a rawLeafNodes option (false by default) that if set to true will cause this module to also use raw nodes for leaf data.

It also seems that setting the reduceSingleLeafToSelf option to false in order to disable that optimisation was being ignored if the optimisation was possible so I made it true by default and respected if explicitly set to false.

[achingbrain]

Requires the following to be merged:

ipfs/js-ipfs-unixfs#24
ipfs/js-ipfs-unixfs#25

[daviddias]

@achingbrain I remember that go-ipfs used file for raw nodes when I implemented this and then things might have changed with the introduction of IPLD to IPLD raw nodes. I know that there are incoherence between regular ipfs add, ipfs files write and when using CIDv1 vs CIDv0.

Can you create a series of interop tests at http://github.com/ipfs/interop to ensure that these things are well captured?

[achingbrain]

@diasdavid aha, I didn't know about that repo. Is it run as part of CI anywhere?

[achingbrain]

@diasdavid

I've added a test here: ipfs/interop#22 - of course, it won't pass for js-ipfs until the mfs functionality has been merged and released.

I found that ipfs add results in file leaf nodes but cat data.file | ipfs files write --create /somepath results in raw leaf nodes.

I asked in the go-ipfs repo about this behaviour and it seems to be because go-ipfs exclusively uses trickle-dags for mfs.

[daviddias]

I asked in the go-ipfs repo about this behaviour and it seems to be because go-ipfs exclusively uses trickle-dags for mfs.

This is a massive loss for deduping. @achingbrain see https://github.com/ipfs/js-ipfs-unixfs-engine/issues/119 and ipfs/kubo#3576

[achingbrain]

I see what you mean, but logically if a file is split up into a bunch of chunks, each chunk is just raw data and not a file so probably shouldn't be marked as such.

That said, it's just a file because it's a named stream of bytes stored in a directory so it could all just be raw and interpreted as a file if we got to the node by following a named DAGLink.

E.g. to extend the example from the ipld spec with a type field:

{
  "files": {
    "cat.jpg": { // give links properties wrapping them in another object
      "link": {"/": "/ipfs/QmUmg7BZC1YP1ca66rRtWKxpXp77WgVHrnv263JtDuvs2k"},
      "mode": 0755,
      "owner": "jbenet",
      "type": "file" // or "symlink" or "directory", etc
    }
  }
}

🤷‍♂️ Just thinking out loud.

---

ipfs/kubo#3576 has been open for well over a year. We can't really enforce interop without replicating go's behaviour, even if it's wrong. How do you want to proceed?

[daviddias]

@achingbrain target interop first and outline a plan for migration.

[achingbrain]

In that case we're going to have to take the hit on deduping...

[alanshaw]

ping @achingbrain what's left to do on this one - flip the rawLeafNodes option default? Code looks sensible, would be nice to have another test or two 😉

[achingbrain]

@alanshaw from what I can see the go implementation uses different importers depending on what you are doing.

ipfs add marks leaf nodes as file (which jsipfs has replicated), ipfs files add marks them as raw.

Until someone decides which is correct we should probably leave the defaults as they are.

[achingbrain]

I've added some tests and updated the readme too. The Mac Jenkins slaves are preventing the build from passing as they are out of disk space so I'm going to merge this.