Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bytes aggregations #2150

Merged
merged 11 commits into from
Jun 2, 2020
Merged

Bytes aggregations #2150

merged 11 commits into from
Jun 2, 2020

Conversation

cyriltovena
Copy link
Contributor

@cyriltovena cyriltovena commented May 29, 2020

This adds two new range aggregations:

  • bytes_rate which calculate the per second bytes rate of log streams.
  • bytes_over_time which returns the bytes size of each log streams.

This can be used with filters, for example this query:

bytes_over_time({cluster="us-central1"} |= "err"[5m]) gives you the total bytes of logs containing the word err in the last 5 minutes.

Those two new aggregations are super useful to determine the size of streams sent to Loki.

Note for the reviewer: This PR brings a lot of code for extracting timeseries from stream which will be reused for future LogQL features.

Fixes #2077

Still need to works on test.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@codecov-commenter
Copy link

Codecov Report

Merging #2150 into master will increase coverage by 0.14%.
The diff coverage is 90.38%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2150      +/-   ##
==========================================
+ Coverage   61.50%   61.65%   +0.14%     
==========================================
  Files         149      151       +2     
  Lines       11935    12006      +71     
==========================================
+ Hits         7341     7402      +61     
- Misses       4012     4019       +7     
- Partials      582      585       +3     
Impacted Files Coverage Δ
pkg/logql/ast.go 89.42% <ø> (ø)
pkg/logql/engine.go 90.00% <ø> (-0.33%) ⬇️
pkg/logql/lex.go 100.00% <ø> (ø)
pkg/logql/evaluator.go 90.56% <81.81%> (-0.90%) ⬇️
pkg/logql/functions.go 87.87% <87.87%> (ø)
pkg/logql/series_extractor.go 94.28% <94.28%> (ø)
pkg/logql/range_vector.go 97.64% <100.00%> (ø)
pkg/logql/shardmapper.go 80.00% <100.00%> (ø)
pkg/logql/test_utils.go 89.13% <100.00%> (ø)
... and 1 more

Copy link
Member

@owen-d owen-d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great, really looking forward to merging after addressing a few comments.

pkg/logql/functions.go Outdated Show resolved Hide resolved
pkg/logql/series_extractor.go Show resolved Hide resolved
pkg/logql/series_extractor.go Show resolved Hide resolved
pkg/logql/series_extractor_test.go Outdated Show resolved Hide resolved
pkg/logql/series_extractor_test.go Show resolved Hide resolved
pkg/logql/shardmapper.go Outdated Show resolved Hide resolved
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@owen-d owen-d merged commit 18f1b06 into grafana:master Jun 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bytes Log Aggregations
3 participants