Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade golangci-lint and fix linting errors #9601

Merged
merged 26 commits into from
Jun 12, 2023

Conversation

trevorwhitney
Copy link
Collaborator

@trevorwhitney trevorwhitney commented Jun 1, 2023

What this PR does / why we need it:

I discovered out linting was a bit out of date when I ran make lint locally with a slightly newer version of golangci-lint installed (1.52.2 instead of 1.51.2). Going above 1.51.2 produced a sea of red using the same linters we currently have enabled.

go version
go version go1.20.2 linux/amd64
golangci-lint version
golangci-lint has version 1.52.2 built with go1.20.2 from v1.52.2 on 19700101-00:00:00
GO111MODULE=on golangci-lint run -v
pkg/storage/stores/series/index/schema.go:294:95: unused-parameter: parameter 'chunkID' seems to be unused, consider removing or renaming it as _ (revive)
func (v9Entries) GetLabelWriteEntries(bucket Bucket, metricName string, labels labels.Labels, chunkID string) ([]Entry, error) {
                                                                                              ^
pkg/storage/stores/series/index/schema.go:325:54: unused-parameter: parameter 'metricName' seems to be unused, consider removing or renaming it as _ (revive)
func (v9Entries) GetChunkWriteEntries(bucket Bucket, metricName string, labels labels.Labels, chunkID string) ([]Entry, error) {
                                                     ^
pkg/storage/stores/series/index/schema.go:386:53: unused-parameter: parameter 'shard' seems to be unused, consider removing or renaming it as _ (revive)
func (v9Entries) FilterReadQueries(queries []Query, shard *astmapper.ShardAnnotation) []Query {
                                                    ^
pkg/storage/stores/series/index/schema.go:395:98: unused-parameter: parameter 'chunkID' seems to be unused, consider removing or renaming it as _ (revive)
func (s v10Entries) GetLabelWriteEntries(bucket Bucket, metricName string, labels labels.Labels, chunkID string) ([]Entry, error) {
                                                                                                 ^
pkg/storage/stores/series/index/schema.go:429:55: unused-parameter: parameter 'metricName' seems to be unused, consider removing or renaming it as _ (revive)
func (v10Entries) GetChunkWriteEntries(bucket Bucket, metricName string, labels labels.Labels, chunkID string) ([]Entry, error) {
                                                      ^
pkg/storage/stores/series/index/schema.go:529:98: unused-parameter: parameter 'chunkID' seems to be unused, consider removing or renaming it as _ (revive)
func (s v11Entries) GetLabelWriteEntries(bucket Bucket, metricName string, labels labels.Labels, chunkID string) ([]Entry, error) {
                                                                                                 ^
pkg/storage/stores/shipper/util/queries_test.go:19:41: unused-parameter: parameter 'ctx' seems to be unused, consider removing or renaming it as _ (revive)
func (m *mockTableQuerier) MultiQueries(ctx context.Context, queries []index.Query, callback index.QueryPagesCallback) error {
                                        ^
pkg/storage/chunk/client/cassandra/table_client.go:24:21: unused-parameter: parameter 'ctx' seems to be unused, consider removing or renaming it as _ (revive)
func NewTableClient(ctx context.Context, cfg Config, registerer prometheus.Registerer) (index.TableClient, error) {
                    ^
pkg/storage/chunk/client/cassandra/authenticator.go:41:46: unused-parameter: parameter 'data' seems to be unused, consider removing or renaming it as _ (revive)
func (p CustomPasswordAuthenticator) Success(data []byte) error {
                                             ^
pkg/storage/chunk/client/cassandra/instrumentation.go:31:30: unused-parameter: parameter 'ctx' seems to be unused, consider removing or renaming it as _ (revive)
func (observer) ObserveBatch(ctx context.Context, b gocql.ObservedBatch) {
                             ^
pkg/storage/chunk/client/cassandra/storage_client.go:690:35: unused-parameter: parameter 'host' seems to be unused, consider removing or renaming it as _ (revive)
func (noopConvictionPolicy) Reset(host *gocql.HostInfo) {}
                                  ^
pkg/storage/chunk/client/cassandra/instrumentation.go:35:30: unused-parameter: parameter 'cts' seems to be unused, consider removing or renaming it as _ (revive)
func (observer) ObserveQuery(cts context.Context, q gocql.ObservedQuery) {
                             ^
clients/pkg/logentry/metric/metricvec.go:36:30: unused-parameter: parameter 'ch' seems to be unused, consider removing or renaming it as _ (revive)
func (c *metricVec) Describe(ch chan<- *prometheus.Desc) {}
                             ^
clients/pkg/logentry/metric/histograms.go:16:30: unused-parameter: parameter 'config' seems to be unused, consider removing or renaming it as _ (revive)
func validateHistogramConfig(config *HistogramConfig) error {
                             ^
pkg/util/pool/bytesbuffer.go:61:2: redefines-builtin-id: redefinition of the built-in function cap (revive)
	cap := s.Cap()
	^
clients/pkg/promtail/wal/reader.go:17:2: redefines-builtin-id: redefinition of the built-in function close (revive)
	reader, close, err := walUtils.NewWalReader(dir, -1)
	^
clients/pkg/promtail/wal/watcher_test.go:29:62: unused-parameter: parameter 'i' seems to be unused, consider removing or renaming it as _ (revive)
func (t *testWriteTo) StoreSeries(series []record.RefSeries, i int) {
                                                             ^
clients/pkg/promtail/wal/wal.go:89:2: if-return: redundant if ...; err != nil check, just return error instead. (revive)
	if err := w.wal.Log(*seriesBuf, *entriesBuf); err != nil {
		return err
	}
clients/pkg/promtail/wal/writer.go:243:60: unused-parameter: parameter 'logger' seems to be unused, consider removing or renaming it as _ (revive)
func (ew *entryWriter) WriteEntry(entry api.Entry, wl WAL, logger log.Logger) error {
                                                           ^
pkg/storage/chunk/client/aws/mock.go:270:72: unused-parameter: parameter 'input' seems to be unused, consider removing or renaming it as _ (revive)
func (m *mockDynamoDBClient) ListTablesPagesWithContext(_ aws.Context, input *dynamodb.ListTablesInput, fn func(*dynamodb.ListTablesOutput, bool) bool, _ ...request.Option) error {
                                                                       ^
pkg/storage/chunk/client/aws/dynamodb_table_client.go:93:11: superfluous-else: if block ends with a continue statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (revive)
			} else {
				return err
			}
pkg/ruler/storage/wal/util.go:130:45: unused-parameter: parameter 'histograms' seems to be unused, consider removing or renaming it as _ (revive)
func (c *walDataCollector) AppendHistograms(histograms []record.RefHistogramSample) bool {
                                            ^
pkg/ruler/storage/wal/util.go:135:50: unused-parameter: parameter 'histograms' seems to be unused, consider removing or renaming it as _ (revive)
func (c *walDataCollector) AppendFloatHistograms(histograms []record.RefFloatHistogramSample) bool {
                                                 ^
pkg/ruler/storage/wal/util.go:140:48: unused-parameter: parameter 'series' seems to be unused, consider removing or renaming it as _ (revive)
func (c *walDataCollector) UpdateSeriesSegment(series []record.RefSeries, index int) {}
                                               ^
pkg/util/spanlogger/noop.go:23:45: unused-parameter: parameter 'handler' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpanContext) ForeachBaggageItem(handler func(k, v string) bool) {}
                                            ^
pkg/util/spanlogger/noop.go:26:34: unused-parameter: parameter 'key' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) SetBaggageItem(key, val string) opentracing.Span        { return defaultNoopSpan }
                                 ^
pkg/util/spanlogger/noop.go:27:31: unused-parameter: parameter 'key' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) BaggageItem(key string) string                          { return emptyString }
                              ^
pkg/util/spanlogger/noop.go:28:26: unused-parameter: parameter 'key' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) SetTag(key string, value interface{}) opentracing.Span  { return n }
                         ^
pkg/util/spanlogger/noop.go:29:29: unused-parameter: parameter 'fields' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) LogFields(fields ...log.Field)                          {}
                            ^
pkg/util/spanlogger/noop.go:30:25: unused-parameter: parameter 'keyVals' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) LogKV(keyVals ...interface{})                           {}
                        ^
pkg/util/spanlogger/noop.go:32:37: unused-parameter: parameter 'opts' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) FinishWithOptions(opts opentracing.FinishOptions)       {}
                                    ^
pkg/util/spanlogger/noop.go:33:36: unused-parameter: parameter 'operationName' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) SetOperationName(operationName string) opentracing.Span { return n }
                                   ^
pkg/util/spanlogger/noop.go:35:28: unused-parameter: parameter 'event' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) LogEvent(event string)                                  {}
                           ^
pkg/util/spanlogger/noop.go:36:39: unused-parameter: parameter 'event' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) LogEventWithPayload(event string, payload interface{})  {}
                                      ^
pkg/util/spanlogger/noop.go:37:23: unused-parameter: parameter 'data' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopSpan) Log(data opentracing.LogData)                           {}
                      ^
pkg/util/spanlogger/noop.go:40:31: unused-parameter: parameter 'operationName' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopTracer) StartSpan(operationName string, opts ...opentracing.StartSpanOption) opentracing.Span {
                              ^
pkg/util/spanlogger/noop.go:45:28: unused-parameter: parameter 'sp' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopTracer) Inject(sp opentracing.SpanContext, format interface{}, carrier interface{}) error {
                           ^
pkg/util/spanlogger/noop.go:50:29: unused-parameter: parameter 'format' seems to be unused, consider removing or renaming it as _ (revive)
func (n noopTracer) Extract(format interface{}, carrier interface{}) (opentracing.SpanContext, error) {
                            ^
pkg/logproto/compat.go:298:2: redefines-builtin-id: redefinition of the built-in function new (revive)
	new := *m
	^
pkg/logproto/compat.go:306:2: redefines-builtin-id: redefinition of the built-in function new (revive)
	new := *m
	^
pkg/storage/chunk/json_helpers.go:83:23: unused-parameter: parameter 'ptr' seems to be unused, consider removing or renaming it as _ (revive)
func modelTimeIsEmpty(ptr unsafe.Pointer) bool {
                      ^
pkg/storage/chunk/bigchunk.go:94:28: unused-parameter: parameter 'start' seems to be unused, consider removing or renaming it as _ (revive)
func (b *bigchunk) Rebound(start, end model.Time, filter filter.Func) (Data, error) {
                           ^
pkg/ruler/storage/instance/marshal.go:31:52: unused-parameter: parameter 'scrubSecrets' seems to be unused, consider removing or renaming it as _ (revive)
func MarshalConfigToWriter(c *Config, w io.Writer, scrubSecrets bool) error {
                                                   ^
pkg/ruler/storage/instance/instance_test.go:217:48: unused-parameter: parameter 'f' seems to be unused, consider removing or renaming it as _ (revive)
func (s *mockWalStorage) WriteStalenessMarkers(f func() int64) error { return nil }
                                               ^
pkg/ruler/storage/instance/instance_test.go:219:35: unused-parameter: parameter 'mint' seems to be unused, consider removing or renaming it as _ (revive)
func (s *mockWalStorage) Truncate(mint int64) error                  { return nil }
                                  ^
pkg/ruler/storage/instance/instance_test.go:237:45: unused-parameter: parameter 't' seems to be unused, consider removing or renaming it as _ (revive)
func (a *mockAppender) Add(l labels.Labels, t int64, v float64) (storage.SeriesRef, error) {
                                            ^
pkg/ruler/storage/instance/manager.go:366:36: unused-parameter: parameter 'name' seems to be unused, consider removing or renaming it as _ (revive)
func (m MockManager) InstanceReady(name string) bool {
                                   ^
pkg/ruler/storage/instance/instance_test.go:247:55: unused-parameter: parameter 't' seems to be unused, consider removing or renaming it as _ (revive)
func (a *mockAppender) AddFast(ref storage.SeriesRef, t int64, v float64) error {
                                                      ^
pkg/ruler/storage/instance/instance_test.go:259:39: unused-parameter: parameter 'ref' seems to be unused, consider removing or renaming it as _ (revive)
func (a *mockAppender) AppendExemplar(ref storage.SeriesRef, l labels.Labels, e exemplar.Exemplar) (storage.SeriesRef, error) {
                                      ^
pkg/ruler/storage/instance/instance_test.go:263:39: unused-parameter: parameter 'ref' seems to be unused, consider removing or renaming it as _ (revive)
func (a *mockAppender) UpdateMetadata(ref storage.SeriesRef, l labels.Labels, m metadata.Metadata) (storage.SeriesRef, error) {
                                      ^

The majority of these errors are unused parameter errors from the revive linter.

This PR upgrades to the latest golangci-lint and fixes all the errors.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR

@trevorwhitney trevorwhitney requested a review from a team as a code owner June 1, 2023 22:00
@trevorwhitney trevorwhitney changed the title fix linting rules and run golangci-lint --fix fix linting rules and run golangci-lint run --fix Jun 1, 2023
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jun 1, 2023
@periklis
Copy link
Collaborator

periklis commented Jun 2, 2023

@trevorwhitney Can we exclude the operator subdirectory for now? I believe we have two deviating configurations for golangci-lint in action, one in the root-level Loki directory and one in the operator subdirectory.

@trevorwhitney
Copy link
Collaborator Author

@periklis can do 👍

Copy link
Contributor

@jeschkies jeschkies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome. Did you run the formatter? If so could change make check-format so that it runs on all files? See

loki/Makefile

Line 757 in 2c7309e

check-format: format

I'd not ignore the unused parameter. It catches bugs.

I know it's a lot of work but ideally they are replaced with _.

@trevorwhitney
Copy link
Collaborator Author

I successfully recreated the errors I was seeing locally in commit 0eddf36 by upgrading the version of golangci-lint we are using in CI: https://drone.grafana.net/grafana/loki/24068/3/9

I will now commit the changes to make the linter pass.

@trevorwhitney trevorwhitney changed the title fix linting rules and run golangci-lint run --fix Upgrade golangci-lint and fix linting errors Jun 7, 2023
Copy link
Collaborator

@slim-bean slim-bean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -50,7 +50,7 @@ GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD)

# We don't want find to scan inside a bunch of directories, to accelerate the
# 'make: Entering directory '/src/loki' phase.
DONT_FIND := -name tools -prune -o -name vendor -prune -o -name .git -prune -o -name .cache -prune -o -name .pkg -prune -o
DONT_FIND := -name tools -prune -o -name vendor -prune -o -name operator -prune -o -name .git -prune -o -name .cache -prune -o -name .pkg -prune -o
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other there any implications here for removing the operator directory? like do they lose linting?

Copy link
Collaborator Author

@trevorwhitney trevorwhitney Jun 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does have it's own .golangci-lint.yml, and the concern was we may need to make formatting decisions based on the top level linting that won't agree with operator linting. that being said this may be YAGNI, so I'll remove it for now and wait for it to be a problem.

Makefile Outdated
@@ -298,6 +298,7 @@ publish: packages
lint: ## run linters
go version
golangci-lint version
GO111MODULE=on golangci-lint cache clean -v
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

won't this make linting incredibly slow if we always clean the cache?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one argument is that linting should be slow on CI in order to be complete. I don't have a strong opinion though. I added this when I was struggling with some differences I was seeing on CI vs local but it ended up being something different. I'm happy to remove.

Makefile Outdated
@@ -755,7 +755,7 @@ format:

GIT_TARGET_BRANCH ?= main
check-format: format
git diff --name-only HEAD origin/$(GIT_TARGET_BRANCH) -- "*.go" | xargs --no-run-if-empty git diff --exit-code -- \
find . $(DONT_FIND) -name operator -prune -o -type f -name "*.go" | xargs --no-run-if-empty git diff --exit-code -- \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This excludes the operator files but not the folder 🙈 So the git diff runs on ./operator. You need to exclude it somehow.

@github-actions github-actions bot added area/helm type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories labels Jun 12, 2023
commit 065bee7
Author: Travis Patterson <travis.patterson@grafana.com>
Date:   Mon Jun 12 10:21:58 2023 -0600

    Label Volume Endpoint (#9588)

    For a given set of matchers, returns the top N associated label/value
    pairs by volume. A query for `{cluster=prod}` will return

    ```
    cluster=prod: size (total logs matching this matcher)
     .
     .
     .
    nth-label=nth-value
    ```

    This is to service use cases where users want to understand where their
    log volume has come from by label without making multiple requests to
    the stats endpoint.

    Note: This PR is a monster but it's mostly plumbing. I've pointed out
    the most interesting bits that actually get the volumes from
    ingesters/indexs

commit 4d997a5
Author: Piotr <17101802+thampiotr@users.noreply.github.com>
Date:   Mon Jun 12 16:24:26 2023 +0100

    Fix promtail cluster template not finding all clusters. (#9684)

    **What this PR does / why we need it**:
    In promtail-mixin, the dropdown template for clusters would only include
    clusters that run loki. So if a cluster only run promtail and not loki,
    it doesn't appear.

commit 57f9452
Author: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>
Date:   Mon Jun 12 15:21:08 2023 +0200

    Revert 9217 chaudum/tsdb chunkrefs pool (#9685)

    Revert #9217 (potential bug in query result)

    ---------

    Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

commit 73ac208
Author: Salva Corts <salva.corts@grafana.com>
Date:   Mon Jun 12 10:46:30 2023 +0200

    Improve docs for empty value in cache compression config (#9649)

    **What this PR does / why we need it**:
    Follow up PR for
    #9535 (comment)

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [ ] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [ ] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

commit f239435
Author: Christophe Collot <52134228+CCOLLOT@users.noreply.github.com>
Date:   Fri Jun 9 15:00:31 2023 +0200

    feat(lambda-promtail): add cloudfront log file ingestion support (#9573)

    **What this PR does / why we need it**:

    This PR enables ingesting logs from Cloudfront log files stored in s3
    (batch).
    The current setup only supports streaming Cloudfront logs through AWS
    Kinesis, whereas this PR implements the same flow as for VPC Flow logs,
    Load Balancer logs, and Cloudtrail logs (s3 --> SQS (optional) -->
    Lambda Promtail --> Loki)

    **Special notes for your reviewer**:
    + The Cloudfront log file format is different from the already
    implemented services, meaning we had to build yet another regex. AWS
    never bothered making all services follow the same log file naming
    convention but the "good" thing is that it's now very unlikely they will
    change it in the future.
    + The Cloudfront file name does not have any mention of the AWS account
    or the time of log it contains, it means we have to infer the log type
    from the filename format instead of finding the exact string
    "cloudfront" in the filename. This is why in `getLabels`, if no `type`
    parameter is found in the regex, we use the key corresponding to the
    name of the matching parser.
    + I introduced a new `parser` struct to group together several
    parameters specific to a type of log (and avoid relying too much on map
    key string matching and / or if statements for specific use cases)
    + I've been successfully running this code in several AWS environments
    for days.
    + I corrected a typo from my previous PR #9497 (wrong PR number in
    Changelog.md)
    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [x] Tests updated
    - [x] `CHANGELOG.md` updated
    - [x] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [x] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

    ---------

    Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>

commit c6fbff2
Author: Salva Corts <salva.corts@grafana.com>
Date:   Fri Jun 9 14:40:36 2023 +0200

    Add config to avoid caching stats for recent data (#9537)

    **What this PR does / why we need it**:

    When we query the stats for recent data, we query both the ingesters and
    the index gateways for the stats.

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/async_store.go#L112-L114

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/async_store.go#L126-L127

    Then we merge all the responses, which means summing up all the stats

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/async_store.go#L157-L158

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/stores/index/stats/stats.go#L23-L26

    Because we have a replication factor of 3, this means that we will get
    the stats from the ingesters repeated up to 3 times, hence inflating the
    stats.

    In the stats cache, we store the stats for a given matcher set for the
    whole day, then we extract the stats from the cache by the factor of
    time from the request that is stored in the cache:

    https://github.com/grafana/loki/blob/336283acadb34f5fda9abce4e6fcef1dca9965d8/pkg/querier/queryrange/index_stats_cache.go#L33

    https://github.com/grafana/loki/blob/336283acadb34f5fda9abce4e6fcef1dca9965d8/pkg/querier/queryrange/index_stats_cache.go#L40

    Inflated stats for recent data will be cached, so subsequent stats
    extracted from the cache will be inflated regardless of the time.

    This PR adds a new per-tenant limit `max_stats_cache_freshness` to not
    cache requests with an end time that falls within Now minus this
    duration.

    Here's a scenario illustrating this. The graphs below show the bytes
    stats queried in the sharding middleware. We are running a log filter
    query that won't match any log, every 5 seconds with a length of 3h.

    ![image](https://github.com/grafana/loki/assets/8354290/45c2e6e9-185c-4a18-b290-47da27fc3e39)

    As can be seen, after enabling the stats cache and
    configuring`do_not_cache_request_within` to not cache stats for requests
    within 30m, the bytes stats used in the sharding middleware stopped
    increasing.

    In both cases the stats cache hit ration was 100%.

    ![image](https://github.com/grafana/loki/assets/8354290/cd35bcb8-0c77-4693-a06b-502741fd6e23)

    **Special notes for your reviewer**:
    - Blocked by #9535
    - Note that this PR doesn't fix the root issue of inflated stats form
    the ingesters, but rather buys us some time to work on that.

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [x] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [ ] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

commit 22779e1
Author: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
Date:   Fri Jun 9 13:33:15 2023 +0100

    Fix date template function with epoch times (#8886)

    **What this PR does / why we need it**:

    Adds new toUnixEpoch... functions to convert from a string with a
    Unix/Epoch time to an integer that can be used in the existing `toDate`
    function.

    Note that these are the opposites of some of the functions introduced in
    #8774.

    **Which issue(s) this PR fixes**:
    Fixes #8624.

    **Special notes for your reviewer**:

    **Checklist**
    - [ ] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [X] Documentation added
    - [X] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`

    ---------

    Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

commit 1b410db
Author: Bruno FERNANDO <bruno.fernando@jobteaser.com>
Date:   Fri Jun 9 13:48:42 2023 +0200

    feat(promtail): add CF ClientRequestSource field (#9669)

    **What this PR does / why we need it**:

    Hey folks 👋

    Little contribution here to add a useful log field for cloudflare users.
    Indeed I add the [ClientRequestSource
    field](https://developers.cloudflare.com/logs/reference/clientrequestsource/
    ) which is pretty useful when debugging some specific traffic handled by
    cloudflare

    Extra: Since I was on the documentation I fixed an indentation issue
    that I spotted

    Don't hesitate to reach me if you have any questions

    Cheers 😉

    **Which issue(s) this PR fixes**:
    Fixes #<issue number>

    **Special notes for your reviewer**:

    Loki rocks 🚀

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [ ] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [ ] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

commit b1917a6
Author: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Date:   Fri Jun 9 13:38:21 2023 +0200

    add "alignLeft" and "alignRight" functions (#9672)

    Fixes #9667

commit 98d1307
Author: Ashwanth <iamashwanth@gmail.com>
Date:   Fri Jun 9 12:46:38 2023 +0530

    config: ensure storage config defaults apply to named stores (#9650)

    **What this PR does / why we need it**:
    Since named store config does not register any flags, storage configs
    defined under it do not get the defaults.
    For example
    [aws_storage_config](https://grafana.com/docs/loki/latest/configuration/#aws_storage_config)
    sets the default `storage_class` to `STANDARD`, but the same doesn't get
    applied by default when using named stores.

    This PR ensures that named storage configs are always assigned default
    values when they are unmarshalled by implementing `yaml.Unmarshaler`
    interface

commit 4cebc2d
Author: Pepe Cano <825430+ppcano@users.noreply.github.com>
Date:   Thu Jun 8 21:44:00 2023 +0200

    Docs: replace `k6 Cloud` mention (#9599)

    k6 is now available as a managed service on Grafana Cloud.

    This is a small doc changes to remove the mention of `k6 Cloud`.

    ---------

    Co-authored-by: J Stickler <julie.stickler@grafana.com>

commit 1db560f
Author: Danny Kopping <danny.kopping@grafana.com>
Date:   Thu Jun 8 14:19:58 2023 +0200

    Adding background cache (en|de)queue counters (#9665)

    **What this PR does / why we need it**:
    The background writeback cache exposes gauge metric currently for the
    current queue size. Gauges can be useful, but they are susceptible to
    sample errors because they only represent the point in time as the time
    of the scrape.

    Exposing counters for the bytes (en|de)queued to/from the cache will be
    more useful because they can be aggregated.

    Signed-off-by: Danny Kopping <danny.kopping@grafana.com>

commit 609bc22
Author: Dylan Guedes <djmgguedes@gmail.com>
Date:   Thu Jun 8 09:04:45 2023 -0300

    Distributor: Make  key configurable when logging failures (#9659)

    **What this PR does / why we need it**:
    Make appending `insight=true` key-value pair to log failures
    configurable.

    **Which issue(s) this PR fixes**:
    N/A

commit d581258
Author: Nils Griebner <nils@nils-griebner.de>
Date:   Thu Jun 8 11:30:52 2023 +0200

    Make table manager retention options configurable in helm chart values (#9647)

    **What this PR does / why we need it**:

    Configuration options for table manager retention are hard-coded in the
    helm chart values at the moment so that it's not possible to enable
    retention deletes.

    **Which issue(s) this PR fixes**:
    Fixes #8676

    **Special notes for your reviewer**:
    -

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [x] Tests updated
    - [x] `CHANGELOG.md` updated
    - [x] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [x] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

    Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
commit 065bee7
Author: Travis Patterson <travis.patterson@grafana.com>
Date:   Mon Jun 12 10:21:58 2023 -0600

    Label Volume Endpoint (#9588)

    For a given set of matchers, returns the top N associated label/value
    pairs by volume. A query for `{cluster=prod}` will return

    ```
    cluster=prod: size (total logs matching this matcher)
     .
     .
     .
    nth-label=nth-value
    ```

    This is to service use cases where users want to understand where their
    log volume has come from by label without making multiple requests to
    the stats endpoint.

    Note: This PR is a monster but it's mostly plumbing. I've pointed out
    the most interesting bits that actually get the volumes from
    ingesters/indexs

commit 4d997a5
Author: Piotr <17101802+thampiotr@users.noreply.github.com>
Date:   Mon Jun 12 16:24:26 2023 +0100

    Fix promtail cluster template not finding all clusters. (#9684)

    **What this PR does / why we need it**:
    In promtail-mixin, the dropdown template for clusters would only include
    clusters that run loki. So if a cluster only run promtail and not loki,
    it doesn't appear.

commit 57f9452
Author: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>
Date:   Mon Jun 12 15:21:08 2023 +0200

    Revert 9217 chaudum/tsdb chunkrefs pool (#9685)

    Revert #9217 (potential bug in query result)

    ---------

    Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

commit 73ac208
Author: Salva Corts <salva.corts@grafana.com>
Date:   Mon Jun 12 10:46:30 2023 +0200

    Improve docs for empty value in cache compression config (#9649)

    **What this PR does / why we need it**:
    Follow up PR for
    #9535 (comment)

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [ ] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [ ] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

commit f239435
Author: Christophe Collot <52134228+CCOLLOT@users.noreply.github.com>
Date:   Fri Jun 9 15:00:31 2023 +0200

    feat(lambda-promtail): add cloudfront log file ingestion support (#9573)

    **What this PR does / why we need it**:

    This PR enables ingesting logs from Cloudfront log files stored in s3
    (batch).
    The current setup only supports streaming Cloudfront logs through AWS
    Kinesis, whereas this PR implements the same flow as for VPC Flow logs,
    Load Balancer logs, and Cloudtrail logs (s3 --> SQS (optional) -->
    Lambda Promtail --> Loki)

    **Special notes for your reviewer**:
    + The Cloudfront log file format is different from the already
    implemented services, meaning we had to build yet another regex. AWS
    never bothered making all services follow the same log file naming
    convention but the "good" thing is that it's now very unlikely they will
    change it in the future.
    + The Cloudfront file name does not have any mention of the AWS account
    or the time of log it contains, it means we have to infer the log type
    from the filename format instead of finding the exact string
    "cloudfront" in the filename. This is why in `getLabels`, if no `type`
    parameter is found in the regex, we use the key corresponding to the
    name of the matching parser.
    + I introduced a new `parser` struct to group together several
    parameters specific to a type of log (and avoid relying too much on map
    key string matching and / or if statements for specific use cases)
    + I've been successfully running this code in several AWS environments
    for days.
    + I corrected a typo from my previous PR #9497 (wrong PR number in
    Changelog.md)
    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [x] Tests updated
    - [x] `CHANGELOG.md` updated
    - [x] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [x] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

    ---------

    Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>

commit c6fbff2
Author: Salva Corts <salva.corts@grafana.com>
Date:   Fri Jun 9 14:40:36 2023 +0200

    Add config to avoid caching stats for recent data (#9537)

    **What this PR does / why we need it**:

    When we query the stats for recent data, we query both the ingesters and
    the index gateways for the stats.

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/async_store.go#L112-L114

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/async_store.go#L126-L127

    Then we merge all the responses, which means summing up all the stats

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/async_store.go#L157-L158

    https://github.com/grafana/loki/blob/ebdb2b18007e024d56105afc5230383165ca1650/pkg/storage/stores/index/stats/stats.go#L23-L26

    Because we have a replication factor of 3, this means that we will get
    the stats from the ingesters repeated up to 3 times, hence inflating the
    stats.

    In the stats cache, we store the stats for a given matcher set for the
    whole day, then we extract the stats from the cache by the factor of
    time from the request that is stored in the cache:

    https://github.com/grafana/loki/blob/336283acadb34f5fda9abce4e6fcef1dca9965d8/pkg/querier/queryrange/index_stats_cache.go#L33

    https://github.com/grafana/loki/blob/336283acadb34f5fda9abce4e6fcef1dca9965d8/pkg/querier/queryrange/index_stats_cache.go#L40

    Inflated stats for recent data will be cached, so subsequent stats
    extracted from the cache will be inflated regardless of the time.

    This PR adds a new per-tenant limit `max_stats_cache_freshness` to not
    cache requests with an end time that falls within Now minus this
    duration.

    Here's a scenario illustrating this. The graphs below show the bytes
    stats queried in the sharding middleware. We are running a log filter
    query that won't match any log, every 5 seconds with a length of 3h.

    ![image](https://github.com/grafana/loki/assets/8354290/45c2e6e9-185c-4a18-b290-47da27fc3e39)

    As can be seen, after enabling the stats cache and
    configuring`do_not_cache_request_within` to not cache stats for requests
    within 30m, the bytes stats used in the sharding middleware stopped
    increasing.

    In both cases the stats cache hit ration was 100%.

    ![image](https://github.com/grafana/loki/assets/8354290/cd35bcb8-0c77-4693-a06b-502741fd6e23)

    **Special notes for your reviewer**:
    - Blocked by #9535
    - Note that this PR doesn't fix the root issue of inflated stats form
    the ingesters, but rather buys us some time to work on that.

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [x] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [ ] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

commit 22779e1
Author: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
Date:   Fri Jun 9 13:33:15 2023 +0100

    Fix date template function with epoch times (#8886)

    **What this PR does / why we need it**:

    Adds new toUnixEpoch... functions to convert from a string with a
    Unix/Epoch time to an integer that can be used in the existing `toDate`
    function.

    Note that these are the opposites of some of the functions introduced in
    #8774.

    **Which issue(s) this PR fixes**:
    Fixes #8624.

    **Special notes for your reviewer**:

    **Checklist**
    - [ ] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [X] Documentation added
    - [X] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`

    ---------

    Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

commit 1b410db
Author: Bruno FERNANDO <bruno.fernando@jobteaser.com>
Date:   Fri Jun 9 13:48:42 2023 +0200

    feat(promtail): add CF ClientRequestSource field (#9669)

    **What this PR does / why we need it**:

    Hey folks 👋

    Little contribution here to add a useful log field for cloudflare users.
    Indeed I add the [ClientRequestSource
    field](https://developers.cloudflare.com/logs/reference/clientrequestsource/
    ) which is pretty useful when debugging some specific traffic handled by
    cloudflare

    Extra: Since I was on the documentation I fixed an indentation issue
    that I spotted

    Don't hesitate to reach me if you have any questions

    Cheers 😉

    **Which issue(s) this PR fixes**:
    Fixes #<issue number>

    **Special notes for your reviewer**:

    Loki rocks 🚀

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [ ] Tests updated
    - [ ] `CHANGELOG.md` updated
    - [ ] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [ ] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

commit b1917a6
Author: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Date:   Fri Jun 9 13:38:21 2023 +0200

    add "alignLeft" and "alignRight" functions (#9672)

    Fixes #9667

commit 98d1307
Author: Ashwanth <iamashwanth@gmail.com>
Date:   Fri Jun 9 12:46:38 2023 +0530

    config: ensure storage config defaults apply to named stores (#9650)

    **What this PR does / why we need it**:
    Since named store config does not register any flags, storage configs
    defined under it do not get the defaults.
    For example
    [aws_storage_config](https://grafana.com/docs/loki/latest/configuration/#aws_storage_config)
    sets the default `storage_class` to `STANDARD`, but the same doesn't get
    applied by default when using named stores.

    This PR ensures that named storage configs are always assigned default
    values when they are unmarshalled by implementing `yaml.Unmarshaler`
    interface

commit 4cebc2d
Author: Pepe Cano <825430+ppcano@users.noreply.github.com>
Date:   Thu Jun 8 21:44:00 2023 +0200

    Docs: replace `k6 Cloud` mention (#9599)

    k6 is now available as a managed service on Grafana Cloud.

    This is a small doc changes to remove the mention of `k6 Cloud`.

    ---------

    Co-authored-by: J Stickler <julie.stickler@grafana.com>

commit 1db560f
Author: Danny Kopping <danny.kopping@grafana.com>
Date:   Thu Jun 8 14:19:58 2023 +0200

    Adding background cache (en|de)queue counters (#9665)

    **What this PR does / why we need it**:
    The background writeback cache exposes gauge metric currently for the
    current queue size. Gauges can be useful, but they are susceptible to
    sample errors because they only represent the point in time as the time
    of the scrape.

    Exposing counters for the bytes (en|de)queued to/from the cache will be
    more useful because they can be aggregated.

    Signed-off-by: Danny Kopping <danny.kopping@grafana.com>

commit 609bc22
Author: Dylan Guedes <djmgguedes@gmail.com>
Date:   Thu Jun 8 09:04:45 2023 -0300

    Distributor: Make  key configurable when logging failures (#9659)

    **What this PR does / why we need it**:
    Make appending `insight=true` key-value pair to log failures
    configurable.

    **Which issue(s) this PR fixes**:
    N/A

commit d581258
Author: Nils Griebner <nils@nils-griebner.de>
Date:   Thu Jun 8 11:30:52 2023 +0200

    Make table manager retention options configurable in helm chart values (#9647)

    **What this PR does / why we need it**:

    Configuration options for table manager retention are hard-coded in the
    helm chart values at the moment so that it's not possible to enable
    retention deletes.

    **Which issue(s) this PR fixes**:
    Fixes #8676

    **Special notes for your reviewer**:
    -

    **Checklist**
    - [x] Reviewed the
    [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
    guide (**required**)
    - [x] Documentation added
    - [x] Tests updated
    - [x] `CHANGELOG.md` updated
    - [x] Changes that require user attention or interaction to upgrade are
    documented in `docs/sources/upgrading/_index.md`
    - [x] For Helm chart changes bump the Helm chart version in
    `production/helm/loki/Chart.yaml` and update
    `production/helm/loki/CHANGELOG.md` and
    `production/helm/loki/README.md`. [Example
    PR](d10549e)

    Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
@github-actions github-actions bot removed type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories area/helm labels Jun 12, 2023
@trevorwhitney trevorwhitney merged commit 4a56445 into main Jun 12, 2023
@trevorwhitney trevorwhitney deleted the trevorwhitney/fix-linting branch June 12, 2023 18:24
salvacorts pushed a commit that referenced this pull request Jun 13, 2023
**What this PR does / why we need it**:

Upgrade  `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.
salvacorts pushed a commit that referenced this pull request Jun 16, 2023
**What this PR does / why we need it**:

Upgrade  `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.
salvacorts added a commit that referenced this pull request Jun 16, 2023
This reverts commit
af287ac.

Note that I had to cherrypick #9601
so the CI would pass. That made the diff for this PR huge.

---------

Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants