Skip to content

Commit

Permalink
doco markdown
Browse files Browse the repository at this point in the history
  • Loading branch information
nabbi committed Oct 14, 2021
1 parent 76c613f commit 4c54a47
Show file tree
Hide file tree
Showing 2 changed files with 48 additions and 36 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ I wrote this to dabble with TclOO while improving my solution for throttling ema
* Configurable sub exclusions to filter out false alarm noise.
* Selectively alert different support groups, including shorter messages to pagers or mobile devices.

## concept
## Concept
* Reads standard input from syslog-ng OSE by using the program() driver.
* Alerted logs are tracked within SQLite so recent occurrences can be discarded
* Sendmail recipients are compiled from group memberships within SQLite
Expand Down
82 changes: 47 additions & 35 deletions USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ As a result of using only an in memory database, if this program crashes or sysl
you may see alerts which notify again within X threshold. syslog-ng expects said program not to exit and
continually accept stdio -- you will see the pid change if this script happens to crash as syslog-ng will restart it

* syslog.info syslog-ng[]: Child program exited, restarting; cmdline='/usr/local/sbin/syslog-alert.tcl', status='256' *
*syslog.info syslog-ng[]: Child program exited, restarting; cmdline='/usr/local/sbin/syslog-alert.tcl', status='256'*

This usually indicates a syntax issue introduced into the code base. Since the switch condition is now generated
from the user configuration file, errors have been reduced from manually editing conditions within code.
Expand Down Expand Up @@ -59,11 +59,11 @@ syntax tcl list
{name} {group} {one@example.com, two@example.com} {pager1@example.com, pager2@example.com}
```

NAME is unused, exists for your config doco purposes
GROUP is a label for a team or SME for who should receive a particular type of event
EMAIL and PAGE/mobile both expect valid email addresses.
multiple email address can be added if csv defined (ie ", " separator)
We aren't doing any sms integration so Lookup their carriers phonenumber@domain online
* NAME is unused, exists for your config doco purposes
* GROUP is a label for a team or SME for who should receive a particular type of event
* EMAIL and PAGE/mobile both expect valid email addresses.
** multiple email address can be added if csv defined (ie ", " separator)
** We aren't doing any sms integration so Lookup their carriers phonenumber@domain online

Both are optional. pages get a smaller formatted message, not the entire raw log like email
If someone only has one type of contact method then just leave it blank (ie "{}")
Expand All @@ -79,17 +79,20 @@ If someone only has one type of contact method then just leave it blank (ie "{}"
## alert.conf

copy into /etc/syslog-ng/

This file defines which events to alert on by dynamically generating the tcl switch conditions
First matched so order matters

syntax tcl list, some elements are lists themselves
```
{{pattern1} {pattern2}} {{exclude1} {exclude2}} {hash} {delay} {email} {page} {ignore} {custom tcl code}
```

PATTERN nocase glob matches against $log(all)
all=is the complete reassembled log message
Must escape with double quotes to match tcl switch; unless it's the default (you must define that here too)
These are a list of lists to allow the same throttling and action to occur with minimizing config lines
and minimizing duplicate switch bodies in memory.
* all=is the complete reassembled log message
** Must escape with double quotes to match tcl switch; unless it's the default (you must define that here too)
** These are a list of lists to allow the same throttling and action to occur with minimizing config lines and minimizing duplicate switch bodies in memory.

```
{{""}}
{{"*mdadm*}}
Expand All @@ -98,9 +101,10 @@ PATTERN nocase glob matches against $log(all)
```

EXCLUDE pattern sub negates what matched at PATTERN
Also nocase glob matches but this filters to a specific section of the log message
Multiple conditions are treated as OR
if you need AND then use all= and string together the template order with glob wildcards
* Also nocase glob matches but this filters to a specific section of the log message
** Multiple conditions are treated as OR
** if you need AND then use all= and string together the template order with glob wildcards

```
{}
{{}}
Expand All @@ -109,48 +113,57 @@ EXCLUDE pattern sub negates what matched at PATTERN
{{msg="*some other event all daemons*"}}
```
HASH controls how we throttle an alert
This is also the default subject for email and pages
Usually I include the host which allows similar events from other nodes to still alert
This can be anything, as it does not pattern match against the real message,
although they are linked so this needs to be unique for the log event and if too generic
or matched too soon, it could suppress other events
* This is also the default subject for email and pages
* Usually I include the host which allows similar events from other nodes to still alert

> This can be anything, as it does not pattern match against the real message, although they are linked so this needs to be unique for the log
event and if too generic or matched too soon, it could suppress other events

```
{"$log(host) label"}
{"common message from all sources"}
```

DELAY throttles how long between getting another alert
Defined as an integer in seconds
* Defined as an integer in seconds

```
{600}
{3600}
{86400}
```

EMAIL these groups
multiple can be listed separated with a space
can be omitted, then no action is taken
(ie maybe use with IGNORE or CUSTOM if no EMAIL action is desired)
* multiple can be listed separated with a space
* can be omitted, then no action is taken
** (ie maybe use with IGNORE or CUSTOM if no EMAIL action is desired)

```
{}
{admin disk}
{oncall}
```

PAGE these groups
same as EMAIL
* same as EMAIL

IGNORE is a boolean true false
This does not process the pattern for alerting
Consider filtering heavy noise within syslog-ng itself
* This does not process the pattern for alerting

> Consider filtering heavy noise within syslog-ng itself
```
{}
{0}
{1}
```

CUSTOM eval as tcl code
This extends further flexibility of the program by tclsh code injection
customize the action behavior, positioned before email/page actions
* This extends further flexibility of the program by tclsh code injection
* customize the action behavior, positioned before email/page actions

> A good reminder to limit modification to the config files and tcl script this will exec as the UID of syslog-ng
A good reminder to limit modification to the config files and tcl script
this will exec as the UID of syslog-ng
```
{}
{set subject "$log(host) event"}
Expand Down Expand Up @@ -185,7 +198,6 @@ This program depends on adjusting your /etc/syslog-ng/syslog-ng.conf
Below describes how to integrate this into your config as an external program call



### Template
In order to parse log events into variables, a predictable and parsable structure
needs to be established. We escape each message section with {} to create a tcl list.
Expand All @@ -198,7 +210,7 @@ template t_alert {
```

### Destination
define where you placed this script so syslog-ng can pipe events to it
Defines where you placed this script so syslog-ng can pipe events to it
Note that this is where you bind the template formatting

```
Expand All @@ -210,10 +222,10 @@ destination d_alert { program("/usr/local/sbin/syslog-alert.tcl" template(t_aler
So. This script was written with the mindset of pre-filtering events within syslog-ng
The expectation is, if you pipe messages to this alert script then you intend send emails.

Think of syslog-ng as the course comb and this script as the fine side of the comb.
This results in some duplication of config mgmt; in syslog-ng.conf and within this script
While it appears capable to handle forward all events, I haven't extensively load tested this
inversed behavior.
> Think of syslog-ng as the course comb and this script as the fine side of the comb.
> This results in some duplication of config mgmt; in syslog-ng.conf and within this script
> While it appears capable to handle forward all events, I haven't extensively load tested this
> inversed behavior.
filter f_level3 { level (err..emerg); };

Expand Down

0 comments on commit 4c54a47

Please sign in to comment.