Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a circuit-breaker to memcached dial calls #3051

Merged
merged 2 commits into from
Aug 25, 2020
Merged

Add a circuit-breaker to memcached dial calls #3051

merged 2 commits into from
Aug 25, 2020

Conversation

bboreham
Copy link
Contributor

If the server is unavailable, this will avoid Cortex hammering it with new connection requests and flooding the log.

We use a 3rd-party fork of the gomemcached library so we can set a custom dialer function - see bradfitz/gomemcache#86 (this branch is slightly behind the version we used before, but only in code we don't call)

Fixes #1821

This is an alternative to #3045

Checklist

  • NA Tests updated
  • NA Documentation added
  • CHANGELOG.md updated

@bboreham bboreham force-pushed the memcached-circuit-breaker-dial branch from fff5642 to fadcd89 Compare August 17, 2020 17:24
@bboreham
Copy link
Contributor Author

To test this I changed the querier integration test to attempt to dial an address, and to sleep a random time up to 12 seconds before each query. All tests still pass:

=== RUN   TestQuerierWithChunksStorage
15:43:26 Starting consul
15:43:26 consul: ==> Starting Consul agent...
15:43:26 consul: ==> Consul agent running!
15:43:26 consul: Version: 'v0.9.4'
15:43:26 consul: Node ID: '59a0d12f-d65e-c9a7-5f40-dac83a64dcad'
15:43:26 consul: Node name: 'consul'
15:43:26 consul: Datacenter: 'dc1' (Segment: '<all>')
15:43:26 consul: Server: true (Bootstrap: false)
15:43:26 consul: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
15:43:26 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
15:43:26 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
15:43:26 consul: ==> Log data will now stream in as it occurs:
15:43:27 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32918]
15:43:27 Starting dynamodb
15:43:27 Ports for container: e2e-cortex-test-dynamodb Mapping: map[8000:32919]
15:43:27 dynamodb: Initializing DynamoDB Local with the following configuration:
15:43:27 dynamodb: Port:        8000
15:43:27 dynamodb: InMemory:    true
15:43:27 dynamodb: DbPath:      null
15:43:27 dynamodb: SharedDb:    true
15:43:27 dynamodb: shouldDelayTransientStatuses:        false
15:43:27 dynamodb: CorsParams:  *
15:43:29 Starting table-manager
15:43:29 consul: ==> Newer Consul version available: 1.8.2 (currently running: 0.9.4)
15:43:29 Ports for container: e2e-cortex-test-table-manager Mapping: map[80:32921 9095:32920]
15:43:31 Starting distributor
15:43:32 Ports for container: e2e-cortex-test-distributor Mapping: map[80:32923 9095:32922]
15:43:32 Starting ingester
15:43:33 Ports for container: e2e-cortex-test-ingester Mapping: map[80:32925 9095:32924]
15:43:33 Starting querier
15:43:34 Ports for container: e2e-cortex-test-querier Mapping: map[80:32927 9095:32926]
15:43:34 querier: level=info ts=2020-08-18T15:43:34.652723748Z caller=main.go:162 msg="Starting Cortex" version="(version=1.2.0, branch=memcached-circuit-breaker-dial, revision=368f8b653)"
15:43:34 querier: level=info ts=2020-08-18T15:43:34.653105705Z caller=server.go:194 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
15:43:34 querier: level=warn ts=2020-08-18T15:43:34.653488193Z caller=experimental.go:19 msg="experimental feature in use" feature="DNS-based memcached service discovery"
15:43:34 querier: level=warn ts=2020-08-18T15:43:34.653840011Z caller=experimental.go:19 msg="experimental feature in use" feature="DNS-based memcached service discovery"
15:43:34 querier: level=info ts=2020-08-18T15:43:34.660037546Z caller=worker.go:62 msg="no address specified, not starting worker"
15:43:34 querier: level=info ts=2020-08-18T15:43:34.660415343Z caller=module_service.go:58 msg=initialising module=memberlist-kv
15:43:34 querier: level=info ts=2020-08-18T15:43:34.660454785Z caller=module_service.go:58 msg=initialising module=server
15:43:34 querier: level=info ts=2020-08-18T15:43:34.660530785Z caller=module_service.go:58 msg=initialising module=ring
15:43:34 querier: level=info ts=2020-08-18T15:43:34.660658596Z caller=module_service.go:58 msg=initialising module=distributor
15:43:34 querier: level=info ts=2020-08-18T15:43:34.660637669Z caller=module_service.go:58 msg=initialising module=store
15:43:34 querier: level=info ts=2020-08-18T15:43:34.660747855Z caller=cortex.go:307 msg="Cortex started"
15:43:35 querier: ts=2020-08-18T15:43:35.420861139Z caller=spanlogger.go:53 org_id=user-7 traceID=3b36a95b5779c057 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="dial tcp 127.0.0.1:11211: connect: connection refused"
15:43:35 querier: ts=2020-08-18T15:43:35.469690504Z caller=spanlogger.go:53 org_id=user-0 traceID=93b8be7030ee5c2 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="dial tcp 127.0.0.1:11211: connect: connection refused"
15:43:35 querier: level=error ts=2020-08-18T15:43:35.496543724Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: dial tcp 127.0.0.1:11211: connect: connection refused"
15:43:35 querier: level=error ts=2020-08-18T15:43:35.497988774Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: dial tcp 127.0.0.1:11211: connect: connection refused"
15:43:35 querier: ts=2020-08-18T15:43:35.735056026Z caller=spanlogger.go:53 org_id=user-0 traceID=7460d91580369023 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="dial tcp 127.0.0.1:11211: connect: connection refused"
15:43:35 querier: level=info ts=2020-08-18T15:43:35.746198227Z caller=memcached_client.go:135 msg="circuit-breaker state change" name=store.index-cache-read.:127.0.0.1:11211 from=closed to=open
15:43:35 querier: level=error ts=2020-08-18T15:43:35.746262754Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: dial tcp 127.0.0.1:11211: connect: connection refused"
15:43:36 querier: ts=2020-08-18T15:43:36.003812346Z caller=spanlogger.go:53 org_id=user-3 traceID=76c3a09c3636933a method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.014162992Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.048954602Z caller=spanlogger.go:53 org_id=user-1 traceID=537a1aff0c0191c5 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.063886244Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.114795838Z caller=spanlogger.go:53 org_id=user-7 traceID=789025579c2659c0 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.123607958Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.24398475Z caller=spanlogger.go:53 org_id=user-6 traceID=4c6e54e0e17bf9ab method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.269101112Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.403100581Z caller=spanlogger.go:53 org_id=user-5 traceID=1afd252f1f5e3420 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.432256589Z caller=spanlogger.go:53 org_id=user-8 traceID=1b01e38a69c12539 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.44284185Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.446742081Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.479075241Z caller=spanlogger.go:53 org_id=user-2 traceID=39d104a04ae8e0be method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.487794685Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.495772128Z caller=spanlogger.go:53 org_id=user-4 traceID=20e53543f05413ab method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.505245443Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.547384936Z caller=spanlogger.go:53 org_id=user-0 traceID=5c2897bac652ee6a method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.55549085Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.893277345Z caller=spanlogger.go:53 org_id=user-7 traceID=2c5787fbb3a49fea method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.904402389Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:36 querier: ts=2020-08-18T15:43:36.981334979Z caller=spanlogger.go:53 org_id=user-3 traceID=182b1d6c1b21b1b8 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:36 querier: level=error ts=2020-08-18T15:43:36.989636601Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.071311573Z caller=spanlogger.go:53 org_id=user-6 traceID=361247b0bd8a7ac3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.078614942Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.12933841Z caller=spanlogger.go:53 org_id=user-2 traceID=6affbda0c1a5c6b6 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.140293223Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.221341481Z caller=spanlogger.go:53 org_id=user-0 traceID=7d4d80ff49a2de3b method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.231487Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.26709128Z caller=spanlogger.go:53 org_id=user-7 traceID=66a4c6d35cd0aa87 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.274936236Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.275522299Z caller=spanlogger.go:53 org_id=user-8 traceID=695ce340fa19cc7f method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.283918371Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.409362483Z caller=spanlogger.go:53 org_id=user-6 traceID=7faea3380f41c4cb method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.421996619Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.550029306Z caller=spanlogger.go:53 org_id=user-3 traceID=eafd70f8e13195b method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.558210224Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.590334864Z caller=spanlogger.go:53 org_id=user-1 traceID=7c985408ff56636f method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.598036142Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.662565039Z caller=spanlogger.go:53 org_id=user-2 traceID=8c026e456d35e61 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.670255741Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:37 querier: ts=2020-08-18T15:43:37.982404898Z caller=spanlogger.go:53 org_id=user-8 traceID=375bb93fd4eb1ac3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:37 querier: level=error ts=2020-08-18T15:43:37.991418955Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.004206018Z caller=spanlogger.go:53 org_id=user-2 traceID=4dfd89eba95382ac method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.012055419Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.134528717Z caller=spanlogger.go:53 org_id=user-2 traceID=aca71d7e6aad56 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.142685149Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.198553111Z caller=spanlogger.go:53 org_id=user-0 traceID=2c77c49dac105a57 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.206396885Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.420330804Z caller=spanlogger.go:53 org_id=user-5 traceID=3276adfdc0fdb627 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.430651286Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.431849911Z caller=spanlogger.go:53 org_id=user-5 traceID=6f7313233ce585a3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.439413321Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.565567149Z caller=spanlogger.go:53 org_id=user-9 traceID=356bc5cd94f04d3e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.575279162Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.615904875Z caller=spanlogger.go:53 org_id=user-8 traceID=51faa1cc9f5fcf7 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.61590344Z caller=spanlogger.go:53 org_id=user-7 traceID=5f5cecebe3a67c22 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.624436425Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.627673827Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.794773508Z caller=spanlogger.go:53 org_id=user-8 traceID=710c815f33676808 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.80216715Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.889870948Z caller=spanlogger.go:53 org_id=user-1 traceID=3f0ac6b6ff8c640f method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.897710849Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:38 querier: ts=2020-08-18T15:43:38.900504692Z caller=spanlogger.go:53 org_id=user-7 traceID=441d56821bb12cc9 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:38 querier: level=error ts=2020-08-18T15:43:38.920274125Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:39 querier: ts=2020-08-18T15:43:39.100308767Z caller=spanlogger.go:53 org_id=user-1 traceID=57292e698f08e61 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:39 querier: level=error ts=2020-08-18T15:43:39.108084647Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:39 querier: ts=2020-08-18T15:43:39.227527917Z caller=spanlogger.go:53 org_id=user-4 traceID=d14898731028b0c method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:39 querier: level=error ts=2020-08-18T15:43:39.240894153Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:39 querier: ts=2020-08-18T15:43:39.391297348Z caller=spanlogger.go:53 org_id=user-7 traceID=48dd204269c5fd8 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:39 querier: level=error ts=2020-08-18T15:43:39.412743136Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:39 querier: ts=2020-08-18T15:43:39.420533997Z caller=spanlogger.go:53 org_id=user-2 traceID=5d83fbabef2bd2a0 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:39 querier: level=error ts=2020-08-18T15:43:39.428857464Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:39 querier: ts=2020-08-18T15:43:39.519243066Z caller=spanlogger.go:53 org_id=user-7 traceID=14c4be3a2cbdc767 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:39 querier: level=error ts=2020-08-18T15:43:39.526576881Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:39 querier: ts=2020-08-18T15:43:39.561645048Z caller=spanlogger.go:53 org_id=user-1 traceID=63e35ba7d8d33522 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:39 querier: level=error ts=2020-08-18T15:43:39.570450623Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:40 querier: ts=2020-08-18T15:43:40.287625594Z caller=spanlogger.go:53 org_id=user-9 traceID=4ac42b3f4a240a49 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:40 querier: level=error ts=2020-08-18T15:43:40.304413935Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:40 querier: ts=2020-08-18T15:43:40.330009502Z caller=spanlogger.go:53 org_id=user-5 traceID=32f50537bffab32e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:40 querier: level=error ts=2020-08-18T15:43:40.337358305Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:40 querier: ts=2020-08-18T15:43:40.622780684Z caller=spanlogger.go:53 org_id=user-9 traceID=3cfc3fed21f93e0d method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:40 querier: level=error ts=2020-08-18T15:43:40.630158788Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:40 querier: ts=2020-08-18T15:43:40.660740203Z caller=spanlogger.go:53 org_id=user-1 traceID=5c02baa632638750 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:40 querier: level=error ts=2020-08-18T15:43:40.667888951Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:40 querier: ts=2020-08-18T15:43:40.734746943Z caller=spanlogger.go:53 org_id=user-6 traceID=711ad942ea8af61e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:40 querier: level=error ts=2020-08-18T15:43:40.741946363Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:40 querier: ts=2020-08-18T15:43:40.97112782Z caller=spanlogger.go:53 org_id=user-9 traceID=7a706fa640d3077e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:40 querier: level=error ts=2020-08-18T15:43:40.979148163Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.009898671Z caller=spanlogger.go:53 org_id=user-8 traceID=17166aaa8276a8a method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.016992707Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.100088102Z caller=spanlogger.go:53 org_id=user-3 traceID=7458413537c2d81b method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.10725629Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.522108893Z caller=spanlogger.go:53 org_id=user-2 traceID=74650a8a2598bc75 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.54438001Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.561720474Z caller=spanlogger.go:53 org_id=user-0 traceID=41fd05b23d62642c method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.575822158Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.588562213Z caller=spanlogger.go:53 org_id=user-6 traceID=45f14facf4857c9d method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.597175857Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.62037154Z caller=spanlogger.go:53 org_id=user-8 traceID=4a13ffdf390773d3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.629802737Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.652569969Z caller=spanlogger.go:53 org_id=user-4 traceID=11d93d92408e1bf7 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.660115981Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.790400624Z caller=spanlogger.go:53 org_id=user-0 traceID=6d83b26d282d4699 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.79934693Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.83121053Z caller=spanlogger.go:53 org_id=user-4 traceID=644678d994972478 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: ts=2020-08-18T15:43:41.841013028Z caller=spanlogger.go:53 org_id=user-4 traceID=3f8777fec3e84a62 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.849491888Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:41 querier: level=error ts=2020-08-18T15:43:41.857313148Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:42 querier: ts=2020-08-18T15:43:42.482542823Z caller=spanlogger.go:53 org_id=user-0 traceID=da10386fce0a7e3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:42 querier: level=error ts=2020-08-18T15:43:42.494135526Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:42 querier: ts=2020-08-18T15:43:42.514528357Z caller=spanlogger.go:53 org_id=user-9 traceID=169ec9cd5d22fb65 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:42 querier: level=error ts=2020-08-18T15:43:42.522707427Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:42 querier: ts=2020-08-18T15:43:42.581907088Z caller=spanlogger.go:53 org_id=user-7 traceID=5c2f8580cdea66b method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:42 querier: level=error ts=2020-08-18T15:43:42.589092389Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:42 querier: ts=2020-08-18T15:43:42.661356151Z caller=spanlogger.go:53 org_id=user-1 traceID=144c3a32449db5b8 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:42 querier: level=error ts=2020-08-18T15:43:42.668746963Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:42 querier: ts=2020-08-18T15:43:42.863314385Z caller=spanlogger.go:53 org_id=user-4 traceID=493568759c28a0a9 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:42 querier: level=error ts=2020-08-18T15:43:42.887375404Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:42 querier: ts=2020-08-18T15:43:42.949542001Z caller=spanlogger.go:53 org_id=user-3 traceID=49003a651507cb3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:42 querier: level=error ts=2020-08-18T15:43:42.968872634Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:42 querier: ts=2020-08-18T15:43:42.99256687Z caller=spanlogger.go:53 org_id=user-4 traceID=917dd69d3a3ca22 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:42.999759735Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.04364058Z caller=spanlogger.go:53 org_id=user-1 traceID=1d5b34f28d3bb8ee method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.050670404Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.057254527Z caller=spanlogger.go:53 org_id=user-6 traceID=4c2f76823a017373 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.064088323Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.080742539Z caller=spanlogger.go:53 org_id=user-8 traceID=b4ecba20db65f9b method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.088138544Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.309496114Z caller=spanlogger.go:53 org_id=user-4 traceID=3ee5d2508cc30c02 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.309507854Z caller=spanlogger.go:53 org_id=user-7 traceID=58d7e12e1e944f62 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.318502896Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.318960964Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.494975182Z caller=spanlogger.go:53 org_id=user-9 traceID=46df2713e107cebd method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.49962137Z caller=spanlogger.go:53 org_id=user-4 traceID=4bcd0f1204607f64 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.502841232Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.507728093Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:43 querier: ts=2020-08-18T15:43:43.979085765Z caller=spanlogger.go:53 org_id=user-1 traceID=603127d375308891 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:43 querier: level=error ts=2020-08-18T15:43:43.992484703Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:44 querier: ts=2020-08-18T15:43:44.044048165Z caller=spanlogger.go:53 org_id=user-6 traceID=7bb20a5d5fc2356c method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:44 querier: level=error ts=2020-08-18T15:43:44.056275105Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:44 querier: ts=2020-08-18T15:43:44.057146502Z caller=spanlogger.go:53 org_id=user-4 traceID=27c7188beedca7d3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:44 querier: level=error ts=2020-08-18T15:43:44.069779855Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:44 querier: ts=2020-08-18T15:43:44.129255816Z caller=spanlogger.go:53 org_id=user-3 traceID=3edddd34c52a17ab method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:44 querier: level=error ts=2020-08-18T15:43:44.133007466Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:44 querier: ts=2020-08-18T15:43:44.574186964Z caller=spanlogger.go:53 org_id=user-5 traceID=368c52eec9f5f08 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:44 querier: level=error ts=2020-08-18T15:43:44.582424623Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:44 querier: ts=2020-08-18T15:43:44.96072615Z caller=spanlogger.go:53 org_id=user-2 traceID=ed26eac0417cd5a method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:44 querier: ts=2020-08-18T15:43:44.964083307Z caller=spanlogger.go:53 org_id=user-5 traceID=729ee5e0131c274e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:44 querier: ts=2020-08-18T15:43:44.972200844Z caller=spanlogger.go:53 org_id=user-6 traceID=7563a75f4dbae9ec method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.024510258Z caller=spanlogger.go:53 org_id=user-3 traceID=7e629c6c0a9d7772 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.058529425Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.060071564Z caller=spanlogger.go:53 org_id=user-1 traceID=566eec81c44bcc6e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.064890484Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.064935724Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.068002126Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.069495046Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.248426183Z caller=spanlogger.go:53 org_id=user-5 traceID=6d231c494a883926 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.252115501Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.404004757Z caller=spanlogger.go:53 org_id=user-3 traceID=8f57e0a7ed8bc7c method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.427298527Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.448071574Z caller=spanlogger.go:53 org_id=user-5 traceID=7fca54ce89484357 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.456655196Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.669378135Z caller=spanlogger.go:53 org_id=user-3 traceID=57093c23b7423739 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.673088916Z caller=spanlogger.go:53 org_id=user-0 traceID=10e399131f8e9a38 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.680068207Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.68150234Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.699435437Z caller=spanlogger.go:53 org_id=user-9 traceID=28a9e97f1f138193 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.703175805Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.74241035Z caller=spanlogger.go:53 org_id=user-0 traceID=6fb7937595df9607 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.745918566Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: level=info ts=2020-08-18T15:43:45.767117283Z caller=memcached_client.go:135 msg="circuit-breaker state change" name=store.index-cache-read.:127.0.0.1:11211 from=open to=half-open
15:43:45 querier: level=info ts=2020-08-18T15:43:45.767552157Z caller=memcached_client.go:135 msg="circuit-breaker state change" name=store.index-cache-read.:127.0.0.1:11211 from=half-open to=open
15:43:45 querier: ts=2020-08-18T15:43:45.767570241Z caller=spanlogger.go:53 org_id=user-2 traceID=3123795fa6ea5f51 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="dial tcp 127.0.0.1:11211: connect: connection refused"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.771032584Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.846899788Z caller=spanlogger.go:53 org_id=user-6 traceID=48ef38c1ee37bfba method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.850150613Z caller=spanlogger.go:53 org_id=user-8 traceID=4d9a539d791900b4 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.850718048Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.85755053Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:45 querier: ts=2020-08-18T15:43:45.918457704Z caller=spanlogger.go:53 org_id=user-9 traceID=7102f6cd0cd5d86e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:45 querier: level=error ts=2020-08-18T15:43:45.924277833Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.044515454Z caller=spanlogger.go:53 org_id=user-9 traceID=42a17cc0649e596a method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.057108404Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.063995038Z caller=spanlogger.go:53 org_id=user-2 traceID=2ce5f8d3e1340322 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.072469214Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.424427864Z caller=spanlogger.go:53 org_id=user-6 traceID=585c7afe139bb61e method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.439691087Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.538812471Z caller=spanlogger.go:53 org_id=user-5 traceID=39d3e9d5493a4ff9 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.542249316Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.548606281Z caller=spanlogger.go:53 org_id=user-3 traceID=390090c5be8c9bf3 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.552620378Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.627137132Z caller=spanlogger.go:53 org_id=user-5 traceID=46278f212671eb81 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.634254061Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.756578122Z caller=spanlogger.go:53 org_id=user-8 traceID=1cd6def166429edb method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.768381063Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:46 querier: ts=2020-08-18T15:43:46.807780996Z caller=spanlogger.go:53 org_id=user-9 traceID=21c1653aeda61f54 method=Memcache.GetMulti level=error msg="Failed to get keys from memcached" err="circuit breaker is open"
15:43:46 querier: level=error ts=2020-08-18T15:43:46.815449538Z caller=memcached.go:236 msg="failed to put to memcached" name=store.index-cache-read. err="server=127.0.0.1:11211: circuit breaker is open"
15:43:47 Killing querier
15:43:47 querier: level=info ts=2020-08-18T15:43:47.090465979Z caller=signals.go:55 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
15:43:47 querier: level=info ts=2020-08-18T15:43:47.090908522Z caller=module_service.go:90 msg="module stopped" module=distributor
15:43:47 querier: level=error ts=2020-08-18T15:43:47.091099104Z caller=client.go:215 msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=10&stale=&wait=10000ms\": context canceled"
15:43:47 querier: level=info ts=2020-08-18T15:43:47.091265358Z caller=module_service.go:90 msg="module stopped" module=ring
15:43:47 querier: level=info ts=2020-08-18T15:43:47.09135188Z caller=module_service.go:90 msg="module stopped" module=store
15:43:47 querier: level=info ts=2020-08-18T15:43:47.091375418Z caller=module_service.go:90 msg="module stopped" module=memberlist-kv
15:43:47 querier: level=info ts=2020-08-18T15:43:47.091533612Z caller=server_service.go:50 msg="server stopped"
15:43:47 querier: level=info ts=2020-08-18T15:43:47.091550097Z caller=module_service.go:90 msg="module stopped" module=server
15:43:47 querier: level=info ts=2020-08-18T15:43:47.091561253Z caller=cortex.go:308 msg="Cortex stopped"
15:43:47 Killing ingester
15:43:47 ingester: level=warn ts=2020-08-18T15:43:47.359727109Z caller=transfer.go:294 msg="transfer attempt failed" err="cannot find ingester to transfer chunks to: no pending ingesters" attempt=1 max_retries=10
15:43:47 Killing distributor
15:43:47 distributor: level=error ts=2020-08-18T15:43:47.66457642Z caller=client.go:215 msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=11&stale=&wait=10000ms\": context canceled"
15:43:47 Killing table-manager
15:43:48 Killing dynamodb
15:43:48 Killing consul
--- PASS: TestQuerierWithChunksStorage (22.80s)
PASS

jtlisi
jtlisi approved these changes Aug 19, 2020
View reviewed changes
Copy link
Contributor

@jtlisi jtlisi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Suggestion(non-blocking): I don't feel strongly about this, but it may be worth hiding this functionality behind a flag.

@bboreham
Add a circuit-breaker to memcached dial calls
5927a95 
If the server is unavailable, this will avoid Cortex hammering it with
new connection requests and flooding the log.

We use a 3rd-party fork of the gomemcached library so we can set a
custom dialer function - see bradfitz/gomemcache#86
(this branch is slightly behind the version we used before, but only
in code we don't call)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
@bboreham bboreham force-pushed the memcached-circuit-breaker-dial branch 2 times, most recently from aabfc25 to cfc1300 Compare August 25, 2020 11:17
@bboreham
Add config and flags to turn on memcached circuit-breaker
cdf2d4a 
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
@bboreham bboreham force-pushed the memcached-circuit-breaker-dial branch from cfc1300 to cdf2d4a Compare August 25, 2020 11:18
@bboreham
Copy link
Contributor Author

@jtlisi I added a flag, and made the defaulted that the circuit-breaker is disabled.

@bboreham bboreham merged commit ae80670 into master Aug 25, 2020
@bboreham bboreham deleted the memcached-circuit-breaker-dial branch August 25, 2020 13:19
@bboreham bboreham mentioned this pull request Aug 25, 2020
Closed
1 task
@pracucci pracucci mentioned this pull request Aug 25, 2020
Merged
3 tasks
@yeya24 yeya24 mentioned this pull request Sep 1, 2020
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jtlisi jtlisi jtlisi approved these changes

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cortex ingester can overload memcached connection backlog
2 participants
@bboreham @jtlisi