Add WriteStream, ReadStream, Converter and HexHelper

[cpl121]

Description

Currently WriteStream, ReadStream, Converter and HexHelper iota.js dependencies are being used in Firefly and they need to be added in the sdk

Also, the encodings have been modified to make transactions between layer 1 and layer 2, as can be seen in the evm-toolkit.
To fit these changes, a new type SpecialStream has been created in Firefly. It would be necessary to add this class together with size64Encode.
On the other hand, although it is still in development, it would also be necessary to add the SpecialStream to be able to read these new transactions, as well as their decode

Motivation

The iota.js dependencies used in Firefly need to be incorporated into the sdk, and the way to encode/decode transactions from layer 1 to layer 2 has been updated, thus the WriteSpecialStream/ReadSpecialStream classes needs to be added too

Requirements

1. Add WriteStream and WriteSpecialStream
2. Add ReadStream and ReadSpecialStream
3. Add Converter
4. Add HexHelper

Are you planning to do it yourself in a pull request?

No.

[kwek20]

Hi @cpl121 is this still needed? If so, why? Adding a whole bunch of "custom" serialization really defeats the purpose of bindings and single source of truth. Is there a way we can change this so you dont need any of the streams?

As for Converter and HexHelper, we should have added all the missing utilities to our utils package. Let me know specifically what method you are missing and ill add it.

[kwek20]

Meeting notes:
HexHelper: Not used
Converter: Methods have been moved to utils
Read/write streams: Possibly provide a read/write() that takes ReadStream / WriteStream and Serialize / deserialize bytes from there. (read takes Uint8Array and returns object, write takes object and returns Uint8Array)

• Q: How do we know what were serializing/deserializing? Is there a default entry point like "Transaction"?

Specialstream: Custom u64 encoding from wasp

• Q: What makes it special, how do we name it better?
• Smart contracts and Governance requires this (TODO: Confirm)
• https://github.com/iotaledger/firefly/blob/develop/packages/shared/lib/core/layer-2/classes/special-stream.class.ts

Need a way to parse NewTransactionDetails for L1-L2 transfer https://github.com/iotaledger/firefly/blob/develop/packages/shared/lib/core/layer-2/utils/getLayer2MetadataForTransfer.ts#L9

Q: Is the reason for these streams that svelte stores requires them?

[kwek20]

@begonaalvarezd Your feedback is welcome please :)

[begonaalvarezd]

After seeing our dependencies in Firefly, apart from utils needed for L2 which should come from a wasp client util libary (so out of the scope in this issue), we use iota-js for 2 things in Firefly that perhaps could be provided by the sdk somehow

• Format NFT metadata https://github.com/iotaledger/firefly/blob/develop/packages/shared/lib/core/nfts/utils/convertAndFormatNftMetadata.ts#L5
• Parse governance metadata https://github.com/iotaledger/firefly/blob/develop/packages/shared/lib/core/wallet/utils/parseGovernanceMetadata.ts#L10

[kwek20]

@begonaalvarezd the hex conversion we already have added in utils/utf8.ts: hexToUtf8 and utf8tobytes.
I will add a hexToBytes so we dont need to do a 2 step

So that leaves us with the ReadStream, which i think we can already do in standard js code:

import { Readable } from "stream";

export async function parseGovernanceMetadata(metadata: string): Promise<IParticipation[]> {
    const read = async (a: Readable, b: number) => {
        let c = [b];
        for (let i = 0; i < b; i++) {
            c[i] = await a.read();
        }
        return c;
    };
    
    const readStream = Readable.from(hexToBytes(metadata));
    const participations: IParticipation[] = []
    const amountParticipations = await readStream.read();
    for (let index = 0; index < amountParticipations; index++) {
        const eventId = bytesToHex(await read(readStream, 32));
        const amountAnswers = await readStream.read();

        const answers: number[] = []
        for (let index = 0; index < amountAnswers; index++) {
            const answer = await readStream.read();
            answers.push(answer)
        }

        participations.push({ eventId, answers })
    }

    return participations;
}

What we lose with this is the error logging name when reading, which we had when we specified the name in the readUInt8('amountParticipations').

[thibault-martinez]

Closing this as part of it has been addressed and it's been decided that the SDK won't contain anything EVM related. Please (re)open issues if you feel like you still miss something.