From 4c54a47f4d6093aa1d2b446547e552f6c99b16df Mon Sep 17 00:00:00 2001 From: nic Date: Wed, 13 Oct 2021 19:35:10 -0500 Subject: [PATCH] doco markdown --- README.md | 2 +- USAGE.md | 82 +++++++++++++++++++++++++++++++------------------------ 2 files changed, 48 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index bae8a4e..9c7733c 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ I wrote this to dabble with TclOO while improving my solution for throttling ema * Configurable sub exclusions to filter out false alarm noise. * Selectively alert different support groups, including shorter messages to pagers or mobile devices. -## concept +## Concept * Reads standard input from syslog-ng OSE by using the program() driver. * Alerted logs are tracked within SQLite so recent occurrences can be discarded * Sendmail recipients are compiled from group memberships within SQLite diff --git a/USAGE.md b/USAGE.md index 9bda9de..ea6425b 100644 --- a/USAGE.md +++ b/USAGE.md @@ -23,7 +23,7 @@ As a result of using only an in memory database, if this program crashes or sysl you may see alerts which notify again within X threshold. syslog-ng expects said program not to exit and continually accept stdio -- you will see the pid change if this script happens to crash as syslog-ng will restart it -* syslog.info syslog-ng[]: Child program exited, restarting; cmdline='/usr/local/sbin/syslog-alert.tcl', status='256' * + *syslog.info syslog-ng[]: Child program exited, restarting; cmdline='/usr/local/sbin/syslog-alert.tcl', status='256'* This usually indicates a syntax issue introduced into the code base. Since the switch condition is now generated from the user configuration file, errors have been reduced from manually editing conditions within code. @@ -59,11 +59,11 @@ syntax tcl list {name} {group} {one@example.com, two@example.com} {pager1@example.com, pager2@example.com} ``` -NAME is unused, exists for your config doco purposes -GROUP is a label for a team or SME for who should receive a particular type of event -EMAIL and PAGE/mobile both expect valid email addresses. -multiple email address can be added if csv defined (ie ", " separator) -We aren't doing any sms integration so Lookup their carriers phonenumber@domain online +* NAME is unused, exists for your config doco purposes +* GROUP is a label for a team or SME for who should receive a particular type of event +* EMAIL and PAGE/mobile both expect valid email addresses. +** multiple email address can be added if csv defined (ie ", " separator) +** We aren't doing any sms integration so Lookup their carriers phonenumber@domain online Both are optional. pages get a smaller formatted message, not the entire raw log like email If someone only has one type of contact method then just leave it blank (ie "{}") @@ -79,17 +79,20 @@ If someone only has one type of contact method then just leave it blank (ie "{}" ## alert.conf copy into /etc/syslog-ng/ + This file defines which events to alert on by dynamically generating the tcl switch conditions First matched so order matters syntax tcl list, some elements are lists themselves +``` {{pattern1} {pattern2}} {{exclude1} {exclude2}} {hash} {delay} {email} {page} {ignore} {custom tcl code} +``` PATTERN nocase glob matches against $log(all) - all=is the complete reassembled log message - Must escape with double quotes to match tcl switch; unless it's the default (you must define that here too) - These are a list of lists to allow the same throttling and action to occur with minimizing config lines - and minimizing duplicate switch bodies in memory. +* all=is the complete reassembled log message +** Must escape with double quotes to match tcl switch; unless it's the default (you must define that here too) +** These are a list of lists to allow the same throttling and action to occur with minimizing config lines and minimizing duplicate switch bodies in memory. + ``` {{""}} {{"*mdadm*}} @@ -98,9 +101,10 @@ PATTERN nocase glob matches against $log(all) ``` EXCLUDE pattern sub negates what matched at PATTERN - Also nocase glob matches but this filters to a specific section of the log message - Multiple conditions are treated as OR - if you need AND then use all= and string together the template order with glob wildcards +* Also nocase glob matches but this filters to a specific section of the log message +** Multiple conditions are treated as OR +** if you need AND then use all= and string together the template order with glob wildcards + ``` {} {{}} @@ -109,48 +113,57 @@ EXCLUDE pattern sub negates what matched at PATTERN {{msg="*some other event all daemons*"}} ``` HASH controls how we throttle an alert - This is also the default subject for email and pages - Usually I include the host which allows similar events from other nodes to still alert - This can be anything, as it does not pattern match against the real message, - although they are linked so this needs to be unique for the log event and if too generic - or matched too soon, it could suppress other events +* This is also the default subject for email and pages +* Usually I include the host which allows similar events from other nodes to still alert + +> This can be anything, as it does not pattern match against the real message, although they are linked so this needs to be unique for the log +event and if too generic or matched too soon, it could suppress other events + ``` {"$log(host) label"} {"common message from all sources"} ``` + DELAY throttles how long between getting another alert - Defined as an integer in seconds +* Defined as an integer in seconds + ``` {600} {3600} {86400} ``` + EMAIL these groups - multiple can be listed separated with a space - can be omitted, then no action is taken - (ie maybe use with IGNORE or CUSTOM if no EMAIL action is desired) +* multiple can be listed separated with a space +* can be omitted, then no action is taken +** (ie maybe use with IGNORE or CUSTOM if no EMAIL action is desired) + ``` {} {admin disk} {oncall} ``` + PAGE these groups - same as EMAIL +* same as EMAIL IGNORE is a boolean true false - This does not process the pattern for alerting - Consider filtering heavy noise within syslog-ng itself +* This does not process the pattern for alerting + +> Consider filtering heavy noise within syslog-ng itself + ``` {} {0} {1} ``` + CUSTOM eval as tcl code - This extends further flexibility of the program by tclsh code injection - customize the action behavior, positioned before email/page actions +* This extends further flexibility of the program by tclsh code injection +* customize the action behavior, positioned before email/page actions + +> A good reminder to limit modification to the config files and tcl script this will exec as the UID of syslog-ng - A good reminder to limit modification to the config files and tcl script - this will exec as the UID of syslog-ng ``` {} {set subject "$log(host) event"} @@ -185,7 +198,6 @@ This program depends on adjusting your /etc/syslog-ng/syslog-ng.conf Below describes how to integrate this into your config as an external program call - ### Template In order to parse log events into variables, a predictable and parsable structure needs to be established. We escape each message section with {} to create a tcl list. @@ -198,7 +210,7 @@ template t_alert { ``` ### Destination -define where you placed this script so syslog-ng can pipe events to it +Defines where you placed this script so syslog-ng can pipe events to it Note that this is where you bind the template formatting ``` @@ -210,10 +222,10 @@ destination d_alert { program("/usr/local/sbin/syslog-alert.tcl" template(t_aler So. This script was written with the mindset of pre-filtering events within syslog-ng The expectation is, if you pipe messages to this alert script then you intend send emails. -Think of syslog-ng as the course comb and this script as the fine side of the comb. -This results in some duplication of config mgmt; in syslog-ng.conf and within this script -While it appears capable to handle forward all events, I haven't extensively load tested this -inversed behavior. +> Think of syslog-ng as the course comb and this script as the fine side of the comb. +> This results in some duplication of config mgmt; in syslog-ng.conf and within this script +> While it appears capable to handle forward all events, I haven't extensively load tested this +> inversed behavior. filter f_level3 { level (err..emerg); };