TTL on rows

[gwik]

Hi,

I'd very much like badger to provide a time to live on rows.
The rows with an expired time to live would be evicted at compaction.
Is there any current plan for this ? Would it require architectural changes ?
Maybe this could be done on the user side using transactions. A reverse index per TTL and scan by range to evict. However eviction would be manually triggered and compaction seems a more efficient, natural way to deal with it, and arguably less error-prone.

RocksDB implements it although it seems a little bit of hack. They suffix keys with TTL, but maybe that'll be good enough.

https://github.com/facebook/rocksdb/wiki/Time-to-Live

[manishrjain]

This can be done easily and should fit in right with the way we're doing things. It would add a small overhead to each key-value pair in the LSM tree (a few bytes to store the timestamp), but that's almost negligible compared to the extra space consumed by MVCC.

To understand this right, a user can set an expiry while setting a key. This can be converted into an exact timestamp after which that key would be considered expired / deleted. Value log GC can then use this information while reclaiming space.

Note that to determine whether a key is expired, the time used would be the time of the local server -- if the server time is out of whack, and data is relocated to another server, keys could either expire too early, or too late. This would be a rare but fundamental problem that I presume is accepted for TTL?

The biggest issue that I see, not specific to this feature request, is the need for the disk format to change. So, if we do this, we must do this before v1.0 release.

I see that this same feature has been requested off BoltDB:
boltdb/bolt#541

And of course, RocksDB has this as well. Still, to support your case, can you please expand how do you intend to use this feature?

[manishrjain]

Okay, it wasn't going to take long, so decided to build this. The APIs are going to change a bit due to this change -- actually, the most common case of setting a key-value would be slightly simplified.

Expect this to hit master today or tomorrow.

[gwik]

Yeah, thanks !

[gwik]

Sorry, I didn't see your first reply, It's awesome that you got this merged so fast.

Note that to determine whether a key is expired, the time used would be the time of the local server -- if the server time is out of whack, and data is relocated to another server, keys could either expire too early, or too late. This would be a rare but fundamental problem that I presume is accepted for TTL?

I intend to use it embedded in a iOS and Android app where time is a sensitive thing we've already be bitten by.
Could the user provide the time function as an option (for GC expiry and SetWithTTL) ? Default would be time.Now but one can override it ? Should I create another issue ?
Actually I'd be happy to contribute, I'll write it if you like.

Thanks again.

[deepakjois]

Hi @gwik I think it is better if you create a separate issue for it.

It would be even better if you actually want to contribute the change yourselves. But let’s discuss it first. The existing mechanism provides a TTLs that have a granularity of 1 second. There are some changes under discussion to allow sub-second TTLs, but it is not straightforward because it is backwards incompatible if we just implement it on top of the existing code. See #339 and #342 for more context.