This is exactly the use case for Phantom and using Phantom in the non-security role. Send the alert to Phantom and handle the rest of the logic there.
However, I have to disagree with the assertion above. there are many ways to address this in Splunk directly. To do this you need to create a state table of some sort in Splunk and look for results.
There are many ways you can do this directly from an alert. Some examples are...
- Update a lookup table to run in your search
- Create an entry in the triggered alert index and monitor that.
- Add a Splunk index entry directly and monitor that
If we use an index, the process could look like this...
Write a Splunk Search that results in triggering an alert action, well call it "Node Suspected Down" which tests the node to see if it responds.
When the test for "Node Suspected Down" completes, write a record to "SplunkDex" with condition "Node Confirmed Up" or "Node Confirmed Down"
In the same alert also write an event to a Splunk index. Let's call this index "SplunkDex" and add an event with the condition of "Node Suspected Down"
On a timed basis, execute a search looking for active records in "SplunkDex" that haven’t been closed. That is their most current state is "Node Confirmed Down"
If the condition is "Node Confirmed Down" set Alert and "Open a Ticket"
In the same alert also write an event to "SplunkDex" setting the condition to "Node Ticket Created"