Alerting

How to chain custom alert actions?

emc2family
New Member

Hi,

I am looking to chain a couple of custom alert actions. A use case is monitoring a node that is down.

When down, alert action #1 is triggered and pings the node as verification.

If the node is indeed down, then alert action #2 is triggered that will cut a ticket.

Action #2 is already existing so I'm just looking to re-use it.

I'm thinking of calling the 2nd alert action using REST (assuming there's an endpoint to fire an alert).

I appreciate any info.

Thanks

0 Karma
1 Solution

starcher
SplunkTrust
SplunkTrust

Within splunk natively that is not possible. alert actions attached to a search are triggered in a bucket of asynchronous actions with no interdependence.

Writing your own chaining might be possible but complicated.

View solution in original post

0 Karma

Claw
Splunk Employee
Splunk Employee

This is exactly the use case for Phantom and using Phantom in the non-security role. Send the alert to Phantom and handle the rest of the logic there.

However, I have to disagree with the assertion above. there are many ways to address this in Splunk directly. To do this you need to create a state table of some sort in Splunk and look for results.

There are many ways you can do this directly from an alert. Some examples are...
- Update a lookup table to run in your search
- Create an entry in the triggered alert index and monitor that.
- Add a Splunk index entry directly and monitor that

If we use an index, the process could look like this...

  1. Write a Splunk Search that results in triggering an alert action, well call it "Node Suspected Down" which tests the node to see if it responds.
  2. When the test for "Node Suspected Down" completes, write a record to "SplunkDex" with condition "Node Confirmed Up" or "Node Confirmed Down"
  3. In the same alert also write an event to a Splunk index. Let's call this index "SplunkDex" and add an event with the condition of "Node Suspected Down"
  4. On a timed basis, execute a search looking for active records in "SplunkDex" that haven’t been closed. That is their most current state is "Node Confirmed Down"
  5. If the condition is "Node Confirmed Down" set Alert and "Open a Ticket"
  6. In the same alert also write an event to "SplunkDex" setting the condition to "Node Ticket Created"

starcher
SplunkTrust
SplunkTrust

Within splunk natively that is not possible. alert actions attached to a search are triggered in a bucket of asynchronous actions with no interdependence.

Writing your own chaining might be possible but complicated.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...