I have been using Splunk Enterprise 7.0.3 to do real-time search alert trigger without any issues previously. Recently, I attempt to upgrade Splunk Enterprise to 7.1.0 and found some weird problem with the alert trigger.
This is the setup I have:
1. Using amqp-ta plugin to consume messages from RabbitMQ into IndexA.
2. An alert trigger running on All Time (real-time) search on IndexA to find newly indexed events.
3. The alert is trigger per result.
4. Each alert has 2 actions: Custom Action to write a result to another RabbitMQ Exchange and also log the event to another index.
Problem:
Whenever a new event is being added to IndexA, it will trigger repeatedly trigger the alert action. All the alert action search result is showing the same event that was added. This trigger will continue infinitely until I disable the Alert.
I'm not sure if there are any changes to the architecture of Real-time search alert trigger from 7.0.3 to 7.1.0. Any help would be greatly appreciated!
Hi agamemnon23, I have met up with Splunk support team in live troubleshooting session and the conclusion is that the complex search query is causing the issue we are facing. And this is only happening on Splunk 7.1.0.
To illustrate more...for search query, (index="test_index"), this will only trigger one alert per result. But for search query, (index="test_index" | table _raw), the repeating alert trigger problem will reappear.
I will keep you posted if I got further updates from Splunk regarding this issue.
I've no ready solution, but did you try changing it to scheduled (every minute) search and see if it still happens?
Hi xpac, thanks for your suggestion. I did tried that and scheduled search is working fine. But my use-case would need to real-time search as I need alerts to be send out immediately when an event is detected.