Skip to content

Commit

Permalink
Clarify comment and doc
Browse files Browse the repository at this point in the history
Signed-off-by: Marco Pracucci <marco@pracucci.com>
  • Loading branch information
pracucci committed Jul 2, 2024
1 parent d2bd338 commit 0cf787f
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 3 deletions.
2 changes: 1 addition & 1 deletion docs/sources/mimir/manage/mimir-runbooks/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -1390,7 +1390,7 @@ How it **works**:

- When ingester is starting, it needs to fetch and process records from Kafka until preconfigured consumption lag is honored. The are two configuration options controlling the desired and tolerated lag before a ingester is considered to have caught up reading from a partition at startup:
- `-ingest-storage.kafka.max-consumer-lag-at-startup`: this is the guaranteed maximum lag before a ingester is considered to have caught up. The ingester doesn't become ACTIVE in the hash ring and doesn't passes the readiness check until the measured lag is below this setting.
- `-ingest-storage.kafka.target-consumer-lag-at-startup`: this is the desired maximum lag that a ingester should try to achieve at startup. This setting is a best-effort. The ingester is granted a "grace period" to have the measured lag below this setting, but the ingester starts anyway if the target lag hasn't been reached within the "grace period", as far as the max lag is honored.
- `-ingest-storage.kafka.target-consumer-lag-at-startup`: this is the desired maximum lag that a ingester should try to achieve at startup. This setting is a best-effort. The ingester is granted a "grace period" to have the measured lag below this setting, but the ingester starts anyway if the target lag hasn't been reached within the "grace period", as far as the max lag is honored. The "grace period" is equal to the configured `-ingest-storage.kafka.max-consumer-lag-at-startup`.
- Each record has a timestamp when it was sent to Kafka by the distributor. When ingester reads the record, it computes "receive delay" as a difference between current time (when record was read) and time when record was sent to Kafka. This receive delay is reported in the metric `cortex_ingest_storage_reader_receive_delay_seconds`. You can see receive delay on `Mimir / Writes` dashboard, in section "Ingester (ingest storage – end-to-end latency)".
- Under normal conditions when ingester is processing records faster than records are appearing, receive delay should be decreasing, until `-ingest-storage.kafka.max-consumer-lag-at-startup` is honored.
- When ingester is starting, and observed "receive delay" is increasing, alert is raised.
Expand Down
6 changes: 4 additions & 2 deletions pkg/storage/ingest/reader.go
Original file line number Diff line number Diff line change
Expand Up @@ -207,8 +207,10 @@ func (r *PartitionReader) processNextFetchesUntilTargetOrMaxLagHonored(ctx conte

// If the target lag hasn't been reached with the first attempt (which stops once at least the max lag
// is honored) then we try to reach the (lower) target lag within a fixed time (best-effort).
// The timeout is equal to the max lag. This is done because we expect at least a 1x replay speed
// from Kafka (which means at most it takes 1s to ingest 1s of data).
// The timeout is equal to the max lag. This is done because we expect at least a 2x replay speed
// from Kafka (which means at most it takes 1s to ingest 2s of data): assuming new data is continuously
// written to the partition, we give the reader maxLag time to replay the backlog + ingest the new data
// written in the meanwhile.
func() (time.Duration, error) {
timedCtx, cancel := context.WithTimeoutCause(ctx, maxLag, errWaitTargetLagDeadlineExceeded)
defer cancel()
Expand Down

0 comments on commit 0cf787f

Please sign in to comment.