Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SOC-3529 add support for pd merging #10

Merged
merged 14 commits into from
Jul 28, 2023
Merged

SOC-3529 add support for pd merging #10

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7bb15df
Add comment explaining how the different pieces fit together, and sta…
eric-eisenhart Jul 24, 2023
ec2d3c2
Rename `zabbix_get_event_id` to `zabbix_get_Event_id_from_pd_object` and
eric-eisenhart Jul 24, 2023
8cb8a22
Minimum viable code (it ACK'd two zabbix events that were part of a s…
eric-eisenhart Jul 25, 2023
f6552e8
Attempt to fallback to old mechanism if /alerts end-point didn't work...
eric-eisenhart Jul 25, 2023
dbfa678
It actually works now. Merges the events properly in Zabbix.
eric-eisenhart Jul 25, 2023
f756d22
Style fixups with `perltidy`
eric-eisenhart Jul 26, 2023
a860412
Another pass with (newer version of) `perltidy`.
eric-eisenhart Jul 26, 2023
4ea04a4
Add a dependency blocking LWP from installing on perl 5.10 pipeline
eric-eisenhart Jul 26, 2023
ddc30d6
Try to get a lower version of a dependency module when compile testin…
eric-eisenhart Jul 26, 2023
7d0732b
Added configurable option for how to handle incoming merge events
eric-eisenhart Jul 28, 2023
7fd1c0a
Update documentation and example config to reflect feature changes.
eric-eisenhart Jul 28, 2023
cabce59
Minor `perltidy` fixes.
eric-eisenhart Jul 28, 2023
6a54712
Remove the attempt to validate the pdmergeaction option; it wasn't wo…
eric-eisenhart Jul 28, 2023
0603ea2
Minor formatting fixup via `perltidy`
eric-eisenhart Jul 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Minimum viable code (it ACK'd two zabbix events that were part of a s… 
…ingle merged PD incident).
  • Loading branch information
@eric-eisenhart
eric-eisenhart committed Jul 25, 2023
commit 8cb8a22310956aa0e7b629317a235f11e905c50b
132 changes: 73 additions & 59 deletions pd2zabbix.cgi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,77 +226,86 @@ sub pagerduty_handle_webhook {
my $html_url = ( $event->{'data'}{'html_url'} || $event->{'data'}{'incident'}{'html_url'} );
warn("html_url: $html_url\n") if $DEBUG >= 2;

# Fetch more details from PagerDuty
my $event_details = pagerduty_get_incident_details($self_url);

# Those more details from PD should include info to work out the zabbix
# event id for use with the Zabbix API
my $zabbix_event_id = zabbix_get_event_id_from_pd_object($event_details);

# If we couldn't work out a zabbix event id, we can't update zabbix.
# In case this was due to transient PagerDuty API issues, return "429"
# to tell PD to try the webhook again a bit later.
# Check PD WebHook docs for retry limits.
unless ($zabbix_event_id) {
print $cgi->header( -status => "429 Can't determine zabbix event id; retry" );
die "Unable to determine zabbix event id";
}
my @pagerduty_alerts = @{pagerduty_get_incident_alerts($self_url)};

warn ("pagerduty_alerts count: ".$#pagerduty_alerts, "\n") if $DEBUG >= 2;

foreach my $pagerduty_alert (@pagerduty_alerts) {
warn( "alert: " . to_json($pagerduty_alert) . "\n" ) if $DEBUG >= 2;
my $zabbix_event_id = zabbix_get_event_id_from_pd_object($pagerduty_alert);
local $@; # exception-handling

# If we couldn't work out a zabbix event id, we can't update zabbix.
# In case this was due to transient PagerDuty API issues, return "429"
# to tell PD to try the webhook again a bit later.
# Check PD WebHook docs for retry limits.
unless ($zabbix_event_id) {
print $cgi->header( -status => "429 Can't determine zabbix event id; retry" );
warn "Unable to determine zabbix event id";
next;
}

# Do appropriate actions on incident event types

# triggered==created (or maybe also end of silencing)
if ( $event_type eq 'incident.triggered' && $config->get('triggeredupdate') ) {
# Do appropriate actions on incident event types

# Add PD incident URL as comment on Zabbix event:
zabbix_event_annotate( $zabbix_event_id, $html_url );
}
# triggered==created (or maybe also end of silencing)
if ( $event_type eq 'incident.triggered' && $config->get('triggeredupdate') ) {

# The original main reason for this: PD ACK to Zabbix ACK
elsif ( $event_type eq 'incident.acknowledged' ) {
# Add PD incident URL as comment on Zabbix event:
eval { zabbix_event_annotate( $zabbix_event_id, $html_url ) };
}

# Update Zabbix event acknowledgement
zabbix_event_acknowledge( $zabbix_event_id, $event, $event_details );
}
# The original main reason for this: PD ACK to Zabbix ACK
elsif ( $event_type eq 'incident.acknowledged' ) {

# And UNACK
elsif ( $event_type eq 'incident.unacknowledged' ) {
# Update Zabbix event acknowledgement
eval { zabbix_event_acknowledge( $zabbix_event_id, $event, $pagerduty_alert ) };
}

# Clear acknowledgement from zabbix event
zabbix_event_unacknowledge( $zabbix_event_id, $event, $event_details );
}
# And UNACK
elsif ( $event_type eq 'incident.unacknowledged' ) {

# If a note is added in PD (if note added when doing another action,
# sends webhook for both that action and the note)
elsif ( $event_type eq 'incident.annotated' ) {
# Clear acknowledgement from zabbix event
eval { zabbix_event_unacknowledge( $zabbix_event_id, $event, $pagerduty_alert ) };
}

# Add comment to zabbix event when PD event gets a note
my $who = $event->{'agent'}{'summary'};
my $content = $event->{'data'}{'content'};
$who ||= "PD";
my $message = "$content -$who";
# If a note is added in PD (if note added when doing another action,
# sends webhook for both that action and the note)
elsif ( $event_type eq 'incident.annotated' ) {

zabbix_event_annotate( $zabbix_event_id, $message );
}
# Add comment to zabbix event when PD event gets a note
my $who = $event->{'agent'}{'summary'};
my $content = $event->{'data'}{'content'};
$who ||= "PD";
my $message = "$content -$who";

# If someone clicks "resolve" in PD, try to close the Zabbix event
elsif ( $event_type eq 'incident.resolved' && $config->get('resolvedupdate') ) {
eval { zabbix_event_annotate( $zabbix_event_id, $message ) };
}

# Send event close attempt to Zabbix
zabbix_event_close( $zabbix_event_id, $event, $event_details );
}
# If someone clicks "resolve" in PD, try to close the Zabbix event
elsif ( $event_type eq 'incident.resolved' && $config->get('resolvedupdate') ) {

# If priority changed in PD, update zabbix event severity to match
# TODO: make this configurable?
elsif ( $event_type eq 'incident.priority_updated' ) {
# Send event close attempt to Zabbix
eval { zabbix_event_close( $zabbix_event_id, $event, $pagerduty_alert ) };
}

# Update Zabbix event severity if PD incident priority changed
zabbix_event_update_priority( $zabbix_event_id, $event, $event_details );
}
# If priority changed in PD, update zabbix event severity to match
# TODO: make this configurable?
elsif ( $event_type eq 'incident.priority_updated' ) {

# If don't know what to do, log it. Not an error, since could have
# simply accepted the default of WebHook sending all event types.
else {
warn("Don't know how to handle event type $event_type\n");
# Update Zabbix event severity if PD incident priority changed
eval { zabbix_event_update_priority( $zabbix_event_id, $event, $pagerduty_alert ) };
}

# If don't know what to do, log it. Not an error, since could have
# simply accepted the default of WebHook sending all event types.
else {
warn("Don't know how to handle event type $event_type\n");
}

if (my $exception = $@) {
warn $exception;
}
}

# Other PD event types that we don't currently do anything with:
Expand Down Expand Up @@ -328,19 +337,24 @@ sub pagerduty_get_incident_details {

}

# Use PD event/incident API to get a list of alerts on the incident
sub pagerduty_get_incident_alerts {
my ($self_url) = @_;
my $pdtoken = $config->get('pdtoken');

my $pd_response = $ua->get( "${self_url}/aerts", 'Authorization' => "Token token=${pdtoken}", );
warn("pagerduty_get_incident_alerts: self_url: $self_url\n") if $DEBUG >= 2;

my $pd_response = $ua->get( "${self_url}/alerts", 'Authorization' => "Token token=${pdtoken}", );
warn( to_json( $pd_response, { allow_blessed => 1 } ) ) if $DEBUG >= 4;
if ( $pd_response->is_success ) {
my $pd_json_content = $pd_response->content();
my $content = decode_json($pd_json_content);
return @{$content->{'alerts'}};
return $content->{'alerts'};
}
else {
die "Unable to fetch details from PagerDuty\n";
warn "Unable to fetch alerts from PagerDuty\n";
warn $pd_response->as_string(). "\n";
die;
}

}
Expand Down
Loading