Alerting

How to chain custom alert actions?

emc2family
New Member

Hi,

I am looking to chain a couple of custom alert actions. A use case is monitoring a node that is down.

When down, alert action #1 is triggered and pings the node as verification.

If the node is indeed down, then alert action #2 is triggered that will cut a ticket.

Action #2 is already existing so I'm just looking to re-use it.

I'm thinking of calling the 2nd alert action using REST (assuming there's an endpoint to fire an alert).

I appreciate any info.

Thanks

0 Karma
1 Solution

starcher
SplunkTrust
SplunkTrust

Within splunk natively that is not possible. alert actions attached to a search are triggered in a bucket of asynchronous actions with no interdependence.

Writing your own chaining might be possible but complicated.

View solution in original post

0 Karma

Claw
Splunk Employee
Splunk Employee

This is exactly the use case for Phantom and using Phantom in the non-security role. Send the alert to Phantom and handle the rest of the logic there.

However, I have to disagree with the assertion above. there are many ways to address this in Splunk directly. To do this you need to create a state table of some sort in Splunk and look for results.

There are many ways you can do this directly from an alert. Some examples are...
- Update a lookup table to run in your search
- Create an entry in the triggered alert index and monitor that.
- Add a Splunk index entry directly and monitor that

If we use an index, the process could look like this...

  1. Write a Splunk Search that results in triggering an alert action, well call it "Node Suspected Down" which tests the node to see if it responds.
  2. When the test for "Node Suspected Down" completes, write a record to "SplunkDex" with condition "Node Confirmed Up" or "Node Confirmed Down"
  3. In the same alert also write an event to a Splunk index. Let's call this index "SplunkDex" and add an event with the condition of "Node Suspected Down"
  4. On a timed basis, execute a search looking for active records in "SplunkDex" that haven’t been closed. That is their most current state is "Node Confirmed Down"
  5. If the condition is "Node Confirmed Down" set Alert and "Open a Ticket"
  6. In the same alert also write an event to "SplunkDex" setting the condition to "Node Ticket Created"

starcher
SplunkTrust
SplunkTrust

Within splunk natively that is not possible. alert actions attached to a search are triggered in a bucket of asynchronous actions with no interdependence.

Writing your own chaining might be possible but complicated.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...