13/WAKU2-STORE / 14/WAKU2-MESSAGE: Allow messages to be marked as "don't store"

[cammellos]

Sometimes the application might produce messages that don't need to be stored, an example would be a ping to show that you are online, or for example messages that are time sensitive (say a push notification, through our current push notification system, which would be ignored if retrieved at a later date).

In those cases it would be useful to mark them as "don't store", so that they are not stored by store nodes. This would improve bandwidth performance of the overall network and of the clients, at the cost of some metadata leaking (though can be optional).

Not a crucial feature, but there would be a few use cases for us.

[oskarth]

Good idea! Would a ephemeral field in Waku Message spec https://rfc.vac.dev/spec/14/ work? Then by convention store nodes wouldn't store those messages.

Similar to MVDS metadata bool ephemeral = 2;

[oskarth]

Also cc @staheri14 fyi

[cammellos]

@oskarth yes, precisely that would work, thanks!

[cammellos]

Another option, it could be a bit more generic, say to allow a StoreTTL, after which point messages are not sent to users from mailservers, so you can still pull them if you just arrived online, and cover both cases.
Say something like:

StoreTTL=x means that if I fetch messages those that have SenderTime + TTL < RequestTimeFrom < RequestTimeTo

In this case it covers the case outlined above and also avoids the scenario of a user coming back online and just missing a useful message.

cc @richard-ramos

[staheri14]

Good idea! Would a ephemeral field in Waku Message spec https://rfc.vac.dev/spec/14/ work? Then by convention store nodes wouldn't store those messages.

Similar to MVDS metadata bool ephemeral = 2;

@oskarth
That is definitely doable and is a good idea.

[staheri14]

StoreTTL=x means that if I fetch messages those that have SenderTime + TTL < RequestTimeFrom < RequestTimeTo

In this case it covers the case outlined above and also avoids the scenario of a user coming back online and just missing a useful message.

@cammellos
I understand the message should be persisted only for StoreTTL e.g., 2 minutes since its creation time. But I could not get the use of RequestTimeFrom and RequestTimeTo? can you please give an example?

[cammellos]

@staheri14 sorry, my bad, I didn't apply the reasoning to the current specs.

Looking at the store docs, I noticed there's a feature that we are missing (maybe though is there and I missed it :) ) https://rfc.vac.dev/spec/13/

Basically when querying store, we want also to specify a range.

The way we currently do is using two parameters, From and To (on top of cursor, which works mostly as in the waku 2 specs).

In a few scenarios we want to query for a time range, for example:

• The user has been offline for x minutes
• We want to "fill a gap" in the past

In the first case, we send a query:

From: {now} To: {last-online} (we actually don't specify From and let the server serves us anything they have recent).
This avoids downloading unnecessary data (if you have been offline for 30 seconds, you only fetch that time range).

The other case is similar:

From: {Start-gap} To: {end-gap}
So we can fetch a particular gap (which again might be only a few minutes, so if we cap the time we don't download unnecessary data).

Also we combine cursor with To, again in order not to download too much data:

Cursor: {x}, To:{y} which will fetch from the cursor to the timestamp y

I could not find a way to do that in the current specs, hope I am looking at the right place.

Long story short, RequestTimeFrom is the same as upper in the current mailserver specs and RequestTimeTo is the same as lower in the current mailserver specs.
https://rfc.vac.dev/spec/8/

This feature I can't see in the docs. Also we actually construct the cursor, though that's not the cleanest choice as it's a bit of an implementation detail, other APIs have separat

[staheri14]

Basically when querying store, we want also to specify a range.

Thanks a lot for the clarification @cammellos!

I think what you want is a time-based history query which is covered in the following specs i.e., Waku FT-store. The FT-Store protocol also supports fetching the gap in the history (and the initial idea was exactly to allow healing the history gaps for offline nodes).
https://rfc.vac.dev/spec/21/

There is also the Resume method in the current Nim API of waku2 which "retrieves and persists the message history since the node's last online time".

Also regarding the cursor, we do have pagination in the store protocol:
https://rfc.vac.dev/spec/13/#paginginfo
Using pagination, you can specify a pageSize as a limit for the number of fetched messages. Then, for example, you can fetch the first pageSize number of messages whose timestamps fall between start_time and end_time. As the result of the query, a pagingInfo is returned to the querying node which can use to fetch the rest of the messages.
A cursor in our design is of the Index type and signifies the index of a persisted WakuMessage.

I think the only missing feature is the TTL for persisted messages.
Hope that my explanation helps! :) let me know if there are other questions

[jm-clius]

Any further action needed on this @cammellos? Moving to backlog for now.