topic How to chain custom alert actions? in Alerting

How to chain custom alert actions?

emc2family — Mon, 30 Sep 2019 18:33:55 GMT

Hi,

I am looking to chain a couple of custom alert actions. A use case is monitoring a node that is down.

When down, alert action #1 is triggered and pings the node as verification.

If the node is indeed down, then alert action #2 is triggered that will cut a ticket.

Action #2 is already existing so I'm just looking to re-use it.

I'm thinking of calling the 2nd alert action using REST (assuming there's an endpoint to fire an alert).

I appreciate any info.

Thanks

Re: How to chain custom alert actions?

starcher — Mon, 30 Sep 2019 18:49:17 GMT

Within splunk natively that is not possible. alert actions attached to a search are triggered in a bucket of asynchronous actions with no interdependence.

Writing your own chaining might be possible but complicated.

Re: How to chain custom alert actions?

Claw — Tue, 01 Oct 2019 19:21:10 GMT

This is exactly the use case for Phantom and using Phantom in the non-security role. Send the alert to Phantom and handle the rest of the logic there.

However, I have to disagree with the assertion above. there are many ways to address this in Splunk directly. To do this you need to create a state table of some sort in Splunk and look for results.

There are many ways you can do this directly from an alert. Some examples are...
- Update a lookup table to run in your search
- Create an entry in the triggered alert index and monitor that.
- Add a Splunk index entry directly and monitor that

If we use an index, the process could look like this...

Write a Splunk Search that results in triggering an alert action, well call it "Node Suspected Down" which tests the node to see if it responds.
When the test for "Node Suspected Down" completes, write a record to "SplunkDex" with condition "Node Confirmed Up" or "Node Confirmed Down"
In the same alert also write an event to a Splunk index. Let's call this index "SplunkDex" and add an event with the condition of "Node Suspected Down"
On a timed basis, execute a search looking for active records in "SplunkDex" that haven’t been closed. That is their most current state is "Node Confirmed Down"
If the condition is "Node Confirmed Down" set Alert and "Open a Ticket"
In the same alert also write an event to "SplunkDex" setting the condition to "Node Ticket Created"