stats: Remember recent lookups and display them in an admin endpoint #8116

jmarantz · 2019-09-02T11:26:37Z

Description: Adds an admin endpoint to show recent lookups. This blocks #4980 .
Risk Level: medium -- this does add overhead when creating new StatNames.
Testing: //test/...
Docs Changes: yes, in the PR.
Release Notes: in the PR.
Fixes: #8035

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz · 2019-09-06T20:30:20Z

This is ready for a round of review, though I realize I may need to work a little more on the stats.md doc.

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz · 2019-09-16T16:53:06Z

I have an un-pushed change on top of this that adds logging. The extra code is small (~ 300 lines) but the implication is risk of spamming logs.

Currently I have this set to dump the remembered (10) recent lookups for the last 5 minutes, at most once every 5 minutes. I am wondering if that should be flag-controlled and if therefore we should review this as is, and then let the logging addition be a follow-up.

I'm also thinking that a config API to add builtin stat-names would make any reports in those logs actionable by Envoy users without having to make code changes. Maybe that should all be integrated into this one PR? WDTY?

stale · 2019-10-11T01:07:16Z

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

Signed-off-by: Joshua Marantz <jmarantz@google.com>

…r to gauge. Signed-off-by: Joshua Marantz <jmarantz@google.com>

Signed-off-by: Joshua Marantz <jmarantz@google.com>

…k for this! Signed-off-by: Joshua Marantz <jmarantz@google.com>

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz · 2019-10-14T20:13:27Z

/retest

repokitteh-read-only · 2019-10-14T20:13:33Z

🔨 rebuilding ci/circleci: docs (failed build)

🐱

Caused by: a #8116 (comment) was created by @jmarantz.

see: more, trace.

mattklein123

Thanks this looks great, just some small nits.

/wait

mattklein123 · 2019-10-16T19:48:42Z

docs/root/operations/admin.rst

+  but in response to user requests on high core-count machines, this
+  can cause performance issues due to mutex contention.
+
+  This option requires Envoy to be started with `use-fake-symbol-table 0`.


nit: I think you can use an :option ref link here.

done; thanks.

Actually I backed this out, as use-fake-symbol-table is not docced, as it was meant to be temporary.

mattklein123 · 2019-10-16T19:49:13Z

docs/root/operations/admin.rst

+  .. http:post:: /stats/recentlookups/clear
+
+  Clears all outstanding lookups and counts. If called when recent lookup
+  collection is enabled, this clears all the, but collection


fixed, and I simplified this, as I don't think it adds anything to the paragraph to discuss the two different cases.

commit coming.

Signed-off-by: Joshua Marantz <jmarantz@google.com>

mattklein123

Thanks!

…nvoyproxy#8116) * stats: Remember recent lookups and display them in an admin endpoint Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz added 8 commits September 1, 2019 17:34

initial impl.

f871f2b

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Add optional recent-lookups tracking to SymbolTable.

be02163

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Hook up recent_lookups_ maps into SymbolTable.

7f404a0

Signed-off-by: Joshua Marantz <jmarantz@google.com>

full stack of recent-lookups admin access compiles, not tested or tried.

4eeb7eb

Signed-off-by: Joshua Marantz <jmarantz@google.com>

cleanup

c1a1a32

Signed-off-by: Joshua Marantz <jmarantz@google.com>

format

e2e384e

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Hold fewer locks for less time.

33c0f88

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Add test for SymbolTableImpl::getRecentLookups().

8808258

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz mentioned this pull request Sep 2, 2019

stats: Shared cluster isolated stats #8118

Merged

jmarantz changed the title ~~WiP stats: Remember recent lookups~~ WiP stats: Remember recent lookups and display them in an admin endpoint Sep 2, 2019

jmarantz added 12 commits September 2, 2019 19:32

simplified; admin flow almost working

8bc22f6

Signed-off-by: Joshua Marantz <jmarantz@google.com>

admin interface kind of works.

86e9204

Signed-off-by: Joshua Marantz <jmarantz@google.com>

collate fix.

cd83c62

Signed-off-by: Joshua Marantz <jmarantz@google.com>

report the total lookup count as well.

fd3bbbd

Signed-off-by: Joshua Marantz <jmarantz@google.com>

format and add admin tests.

cfa1634

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Merge branch 'master' into remember-recent-lookups

49dfcb4

Signed-off-by: Joshua Marantz <jmarantz@google.com>

add function doc to new functions.

e8498a2

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Merge branch 'master' into remember-recent-lookups

312d389

Signed-off-by: Joshua Marantz <jmarantz@google.com>

cleanup

491ab32

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Privatize some functions.

45a4feb

Signed-off-by: Joshua Marantz <jmarantz@google.com>

add doc.

90327f7

Signed-off-by: Joshua Marantz <jmarantz@google.com>

format

7b867b3

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz changed the title ~~WiP stats: Remember recent lookups and display them in an admin endpoint~~ stats: Remember recent lookups and display them in an admin endpoint Sep 6, 2019

jmarantz marked this pull request as ready for review September 6, 2019 20:29

jmarantz assigned mattklein123 Sep 6, 2019

jmarantz added 3 commits September 7, 2019 20:28

Merge branch 'master' into remember-recent-lookups

52793db

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Merge branch 'master' into remember-recent-lookups

3e9e3a5

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Merge branch 'master' into remember-recent-lookups

e9662a0

Signed-off-by: Joshua Marantz <jmarantz@google.com>

repokitteh-read-only bot added the waiting label Oct 4, 2019

stale bot added the stale stalebot believes this issue/PR has not been touched recently label Oct 11, 2019

jmarantz added 3 commits October 13, 2019 23:17

Merge branch 'master' into remember-recent-lookups

6cdf862

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Merge branch 'master' into remember-recent-lookups

9f0b754

Signed-off-by: Joshua Marantz <jmarantz@google.com>

fix builda

6bca221

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz added no stalebot Disables stalebot from closing an issue and removed stale stalebot believes this issue/PR has not been touched recently labels Oct 14, 2019

jmarantz added 2 commits October 14, 2019 10:04

Record the total recent lookups as a counter.

14cbbcf

Signed-off-by: Joshua Marantz <jmarantz@google.com>

simplify flush-stats impl for recent lookups by switching from counte…

d82b1df

…r to gauge. Signed-off-by: Joshua Marantz <jmarantz@google.com>

repokitteh-read-only bot removed the waiting label Oct 14, 2019

update doc

ff84c2c

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz added the waiting label Oct 14, 2019

language cleanup.

e8cb00c

Signed-off-by: Joshua Marantz <jmarantz@google.com>

repokitteh-read-only bot removed the waiting label Oct 14, 2019

jmarantz added 3 commits October 14, 2019 15:46

add release note.

faf1cfa

Signed-off-by: Joshua Marantz <jmarantz@google.com>

use correct method name. Awesome that we have a docs consistency chec…

a70b2de

…k for this! Signed-off-by: Joshua Marantz <jmarantz@google.com>

quotes vs backquotes

c18f184

Signed-off-by: Joshua Marantz <jmarantz@google.com>

mattklein123 requested changes Oct 16, 2019

View reviewed changes

repokitteh-read-only bot added the waiting label Oct 16, 2019

jmarantz added 2 commits October 16, 2019 16:31

Merge branch 'master' into remember-recent-lookups

a7bb051

Signed-off-by: Joshua Marantz <jmarantz@google.com>

fix typos and add explicit option syntax.

f17bcdb

Signed-off-by: Joshua Marantz <jmarantz@google.com>

repokitteh-read-only bot removed the waiting label Oct 16, 2019

un-option the reference to use-fake-symbol-table, which is not doc'd.

298210c

Signed-off-by: Joshua Marantz <jmarantz@google.com>

mattklein123 approved these changes Oct 16, 2019

View reviewed changes

jmarantz merged commit 39c323d into envoyproxy:master Oct 17, 2019

jmarantz deleted the remember-recent-lookups branch October 17, 2019 01:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stats: Remember recent lookups and display them in an admin endpoint #8116

stats: Remember recent lookups and display them in an admin endpoint #8116

jmarantz commented Sep 2, 2019 •

edited

Loading

jmarantz commented Sep 6, 2019 •

edited

Loading

jmarantz commented Sep 16, 2019

stale bot commented Oct 11, 2019

jmarantz commented Oct 14, 2019

repokitteh-read-only bot commented Oct 14, 2019

mattklein123 left a comment

mattklein123 Oct 16, 2019

jmarantz Oct 16, 2019

jmarantz Oct 16, 2019

mattklein123 Oct 16, 2019

jmarantz Oct 16, 2019

mattklein123 left a comment

stats: Remember recent lookups and display them in an admin endpoint #8116

stats: Remember recent lookups and display them in an admin endpoint #8116

Conversation

jmarantz commented Sep 2, 2019 • edited Loading

jmarantz commented Sep 6, 2019 • edited Loading

jmarantz commented Sep 16, 2019

stale bot commented Oct 11, 2019

jmarantz commented Oct 14, 2019

repokitteh-read-only bot commented Oct 14, 2019

mattklein123 left a comment

Choose a reason for hiding this comment

mattklein123 Oct 16, 2019

Choose a reason for hiding this comment

jmarantz Oct 16, 2019

Choose a reason for hiding this comment

jmarantz Oct 16, 2019

Choose a reason for hiding this comment

mattklein123 Oct 16, 2019

Choose a reason for hiding this comment

jmarantz Oct 16, 2019

Choose a reason for hiding this comment

mattklein123 left a comment

Choose a reason for hiding this comment

jmarantz commented Sep 2, 2019 •

edited

Loading

jmarantz commented Sep 6, 2019 •

edited

Loading