Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove time logging on the prom/remote/write endpoint #1164

Merged
merged 1 commit into from
Nov 17, 2018

Conversation

BertHartm
Copy link
Contributor

The logging currently kicks in with force if there's any issues with m3, but we can get all the information from prometheus_remote_storage_sent_batch_duration_seconds_bucket in prometheus, so removing the loggging cuts down on noise and potential side effects in case of issues.

…han 1s because we can rely on prometheus metrics to inform us
@BertHartm
Copy link
Contributor Author

I was hoping this would improve performance under load too, but it seems to have no immediate effect on that.

@arnikola
Copy link
Collaborator

I think this may be slightly better served by making response time logging a noop unless a debug flag is supplied, possibly with a param to determine the logging threshold.

I'm not sure we currently really do anything with these logs but will try and double check if that would be a decent approach in the morning

Copy link
Collaborator

@arnikola arnikola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talking with @robskillington decided to drop the logging from the write path entirely, and maybe consider taking a flag to turn it on for the read side only, so this should be good 👍

@robskillington robskillington merged commit 873e980 into m3db:master Nov 17, 2018
@nikunjgit
Copy link
Contributor

One thing to wonder here though is that when write path is not working well, we won't really have a way anymore to figure that out. Earlier, once you started seeing those logs, you can get an early warning.

@BertHartm
Copy link
Contributor Author

I've been graphing prometheus_remote_storage_sent_batch_duration_seconds_bucket which gives me a much more detailed signal. I've been keeping a separate dashboard from the m3db node dashboard for prometheus remote write stats, but I could add a row to that dashboard in a PR if it would be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants