[pkg/ottl]: Support coercing too-new timestamps

[wbh1]

Component(s)

exporter/loki

Is your feature request related to a problem? Please describe.

Loki will refuse to receive logs that have a timestamp newer than the time on the Loki ingester process.

Example output:

Sep  7 12:47:22 otelgw-3 otelcol-contrib[1305727]: 2023-09-07T12:47:22.483Z#011error#011exporterhelper/queued_retry.go:401#011Exporting failed. The error is not retryable. Dropping data.#011{"kind": "exporter", "data_type": "logs", "name": "loki/prod", "error": "Permanent error: HTTP 400 \"Bad Request\": entry for stream '{component=\"hosts\", datacenter=\"shg1\", environment=\"production\", exporter=\"OTLP\", instance=\"REDACTED\", path=\"/var/log/auth.log\"}' has timestamp too new: 2023-09-07T13:23:01Z", "dropped_items": 68}

This happens sometimes in our environment when NTP gets messed up on a server and its time is actually ahead of the real time. Then, we're alerted to an increase in exporter failures.

Describe the solution you'd like

Ideally, it'd be nice to have an optional feature to coerce timestamps before exporting to Loki (maybe here)?

Logic would look like:

func timestampFromLogRecord(lr plog.LogRecord) time.Time {
	if lr.Timestamp() != 0 {
                if lr.Timestamp() > timeNow() {
                    return time.Unix(0, int64(pcommon.NewTimestampFromTime(timeNow())))
                }
		return time.Unix(0, int64(lr.Timestamp()))
	}

	if lr.ObservedTimestamp() != 0 {
                if lr.ObservedTimestamp() > timeNow() {
                    return time.Unix(0, int64(pcommon.NewTimestampFromTime(timeNow())))
                }
		return time.Unix(0, int64(lr.ObservedTimestamp()))
	}

	return time.Unix(0, int64(pcommon.NewTimestampFromTime(timeNow())))
}

Describe alternatives you've considered

We've tried using OTTL to transform the timestamp/observed timestamp fields, but there doesn't seem to be a way to get the current system time.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

It would be more appropriate to have it as part of OTTL and the transform processor instead of something inside the Loki exporter.

cc @TylerHelmuth

[wbh1]

I agree. How do I go about making this a feature request for OTTL, then?

[jpkrohling]

I changed the labels for this, so that it's now about OTTL instead of Loki exporter. The code owners should be pinged shortly.

[github-actions]

Pinging code owners for pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

We are doing a lot of work to improve time manipulation in OTTL (and it is almost done), and you're right that we need a Now() function. Would you like to submit a PR with that new function?

[wbh1]

Certainly! I can take a crack at it. The only other OTTL func I could see that is similar in only producing an output without an input is UUID. Is that along the lines of what you're looking for?

A Now() function doesn't really seem like a "Converter" type function, but neither does UUID() and idk where else it would go.

[TylerHelmuth]

Ya it would be similar to UUID(). It should take nothing and return the value of time.Now().

[TylerHelmuth]

@wbh1 are you still interested in contributing the function? I'd like to get this capability in before the next release. Let me know if that is feasible for you. If not, I can take over.

[wbh1]

Hey @TylerHelmuth, yes I am! I actually picked it up today and should be able to get a PR up tomorrow today -- PR is up!

[wbh1]

For future internet explorers, a solution to fix Loki rejecting logs from otelcol for having a timestamp that is too new is to apply a transform processor to them that uses this new Now() function:

      transform/timestamps:
        error_mode: ignore
        log_statements:
          - context: log
            statements:
              # If the timestamp attached to the log is from the future, set it to the time that the otelgw received it
              # to avoid having the log rejected from Loki
              - set(time, Now()) where time > Now()