Great work! Just did the first pass. One thing I'm little bit concerned about is if we mix timestamps in milliseconds with timestamp in nanoseconds, we can get wrong results during search. For example, if we have two log messages {"timestamp": "1707026480000"} and {"timestamp": "1707026455.123456789"}, and the query is timestamp > date(1707026480000), we should return nothing but actually we will return the second log message.

Right this is a good point. I modified how we parse literals supplied inside of date() so that its at least consistent with how we parse the log. E.g. if we have timestamps like "1707026455.123456789" in the log and you copy and paste timestamps to specify a filter like timestamp: date(1707026455.123456789) then the timestamp will match as expected. You are right though that when the user starts mixing formats they can get unexpected results.

The core of the problem is that our various timestamp formats already implicitly have different levels of precision (second, millisecond, nanosecond) but we haven't really reconciled these different precisions at the storage, search, or archive metadata levels.

We sort of deal with this problem halfway in TimestampPattern as timestamps which specify second-level precision get stored with millisecond precision, but the same can't be said for raw integer or floating point timestamps. One way to handle this might be to transform all timestamps into nanosecond precision timestamps no matter how they're parsed -- this would make storage, search, and archive metadata consistent in an easy to reason about way. The downside is that this creates an encoding challenge for raw int and float timestamp columns if we want to decompress in a 1:1 way.

The other issue is that "store everything as nanosecond precision" is a bit loaded because it might sometimes be ambiguous/hard to figure out what the precision of some timestamp is supposed to be, so there's still a need to let users disambiguate via configuration for a robust solution (granted for most timestamps it should be pretty obvious).

Anyway if we do something like this (which we should at some point) I'm not sure if we want to include it in this PR (even though its slowly becoming the fix-every-timestamp-issue-we-notice PR).

And can we delete TimestampEntry::TimestampEncoding and associated ingest_timestamp(double)?

And can we delete TimestampEntry::TimestampEncoding and associated ingest_timestamp(double)?

Not yet because we still allow raw floating point columns to be timestamps.

And can we delete TimestampEntry::TimestampEncoding and associated ingest_timestamp(double)?

Not yet because we still allow raw floating point columns to be timestamps.

Oh, that's right