Reported lesser number of fluentd_output_status_error count metric than expected count

[preeti-16]

Hi,

Since client pod is down so when fluentd_output plugin is trying to send data to the output destination since destination is not reachable so error count got reported.
As retry_wait inside buffer section configuration is set to 10s so ideally retry should happened after 10sec but some additional seconds it took due to which there is a delay in reporting the error.

Below are some snippets of log messages:

{"log":{"message":"failed to flush the buffer. retry_times=0 next_retry_time=2024-09-26 04:20:04
{"log":{"message":"failed to flush the buffer. retry_times=1 next_retry_time=2024-09-26 04:20:15
{"log":{"message":"failed to flush the buffer. retry_times=2 next_retry_time=2024-09-26 04:20:25
{"log":{"message":"failed to flush the buffer. retry_times=3 next_retry_time=2024-09-26 04:20:37
{"log":{"message":"failed to flush the buffer. retry_times=4 next_retry_time=2024-09-26 04:20:48
{"log":{"message":"failed to flush the buffer. retry_times=5 next_retry_time=2024-09-26 04:21:00
{"log":{"message":"failed to flush the buffer. retry_times=6 next_retry_time=2024-09-26 04:21:13
{"log":{"message":"failed to flush the buffer. retry_times=7 next_retry_time=2024-09-26 04:21:24

Can you check this issue on priority.

Thanks

[github-actions]

@preeti-16 this issue was automatically closed because it did not follow the issue template.

[daipom]

You need to set false to retry_randomize.

https://docs.fluentd.org/configuration/buffer-section#retries-parameters