-
Notifications
You must be signed in to change notification settings - Fork 183
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
filter fluentd container logs #402
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's talk about this -- it's not what I had in mind.
@@ -8,51 +8,64 @@ | |||
time ${[record["log"].split(/[\n\t]+/)[0]].map! {|item| JSON.parse(item)["time"]}.join("")} | |||
</record> | |||
</filter> | |||
# only match fluentd logs based on fluentd container name and label them @FLUENT |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should try to see if these logs are already tagged by Fluentd itself: https://docs.fluentd.org/deployment/logging#capture-fluentd-logs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The label @FLUENT_LOG
uses <match fluent.**>
which will not stop fluentd from outputting to stdout. We will then have duplicate fluentd logs.
Ref: fluent/fluentd#641 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did it not work starting fluentd with -o /dev/null
to output logs to file /dev/null instead of stdout? https://docs.fluentd.org/deployment/logging#output-to-log-file
…ub.com/SumoLogic/sumologic-kubernetes-collection into vsinghal-filter-fluentd-container-logs
@rvmiller89 have modified the existing pipeline @samjsong Still kept the filter plugin confs in separate files to keep the fluentd pipeline dry and easy to read. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM. I have mixed feelings about pulling out each of logs.*.filter.conf
filter configs, mostly because it creates another layer of indirection, though I agree it may make the pipeline itself easier to read. I am okay either way.
@type grep | ||
<regexp> | ||
key log | ||
pattern /\[info\]|\[fatal\]|drop_oldest_chunk/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait, we want to ingest info logs here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch, I had changed that to test locally. Will fix it to be error
.
…ub.com/SumoLogic/sumologic-kubernetes-collection into vsinghal-filter-fluentd-container-logs # Conflicts: # deploy/kubernetes/fluentd-sumologic.yaml.tmpl
# only ingest fluentd logs of levels: {error, fatal} and warning messages if buffer is full | ||
@type grep | ||
<regexp> | ||
key log |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to indent this line and the next?
…ub.com/SumoLogic/sumologic-kubernetes-collection into vsinghal-filter-fluentd-container-logs
<filter **> | ||
# sumologic kubernetes metadata enrichment filter plugin | ||
<filter containers.**> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@samjsong @rvmiller89 I suspect it is because of this change.
Description
This PR changes the default fluentd log level to be
info
and modifies the fluentd pipeline to ingest only specific fluentd logs into Sumo.Testing performed