Alerting

How to chain custom alert actions?

emc2family
New Member

Hi,

I am looking to chain a couple of custom alert actions. A use case is monitoring a node that is down.

When down, alert action #1 is triggered and pings the node as verification.

If the node is indeed down, then alert action #2 is triggered that will cut a ticket.

Action #2 is already existing so I'm just looking to re-use it.

I'm thinking of calling the 2nd alert action using REST (assuming there's an endpoint to fire an alert).

I appreciate any info.

Thanks

0 Karma
1 Solution

starcher
SplunkTrust
SplunkTrust

Within splunk natively that is not possible. alert actions attached to a search are triggered in a bucket of asynchronous actions with no interdependence.

Writing your own chaining might be possible but complicated.

View solution in original post

0 Karma

Claw
Splunk Employee
Splunk Employee

This is exactly the use case for Phantom and using Phantom in the non-security role. Send the alert to Phantom and handle the rest of the logic there.

However, I have to disagree with the assertion above. there are many ways to address this in Splunk directly. To do this you need to create a state table of some sort in Splunk and look for results.

There are many ways you can do this directly from an alert. Some examples are...
- Update a lookup table to run in your search
- Create an entry in the triggered alert index and monitor that.
- Add a Splunk index entry directly and monitor that

If we use an index, the process could look like this...

  1. Write a Splunk Search that results in triggering an alert action, well call it "Node Suspected Down" which tests the node to see if it responds.
  2. When the test for "Node Suspected Down" completes, write a record to "SplunkDex" with condition "Node Confirmed Up" or "Node Confirmed Down"
  3. In the same alert also write an event to a Splunk index. Let's call this index "SplunkDex" and add an event with the condition of "Node Suspected Down"
  4. On a timed basis, execute a search looking for active records in "SplunkDex" that haven’t been closed. That is their most current state is "Node Confirmed Down"
  5. If the condition is "Node Confirmed Down" set Alert and "Open a Ticket"
  6. In the same alert also write an event to "SplunkDex" setting the condition to "Node Ticket Created"

starcher
SplunkTrust
SplunkTrust

Within splunk natively that is not possible. alert actions attached to a search are triggered in a bucket of asynchronous actions with no interdependence.

Writing your own chaining might be possible but complicated.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...