I have been using Splunk Enterprise 7.0.3 to do real-time search alert trigger without any issues previously. Recently, I attempt to upgrade Splunk Enterprise to 7.1.0 and found some weird problem with the alert trigger.
This is the setup I have:
1. Using amqp-ta plugin to consume messages from RabbitMQ into IndexA.
2. An alert trigger running on All Time (real-time) search on IndexA to find newly indexed events.
3. The alert is trigger per result.
4. Each alert has 2 actions: Custom Action to write a result to another RabbitMQ Exchange and also log the event to another index.
Whenever a new event is being added to IndexA, it will trigger repeatedly trigger the alert action. All the alert action search result is showing the same event that was added. This trigger will continue infinitely until I disable the Alert.
I'm not sure if there are any changes to the architecture of Real-time search alert trigger from 7.0.3 to 7.1.0. Any help would be greatly appreciated!
I've no ready solution, but did you try changing it to scheduled (every minute) search and see if it still happens?
Hi xpac, thanks for your suggestion. I did tried that and scheduled search is working fine. But my use-case would need to real-time search as I need alerts to be send out immediately when an event is detected.
Are you, by any chance, running on the free license? There is a known bug with realtime and the free license on 7.1.0.
I have tried on both free license and paid license both are having the same problem.
Okay, then this seems to be a different issue. I was just guessing as I saw a few issues regarding realtime search with 7.1 recently.
You said that your alert is set at All Time. So,
1. isn't this the reason that your search is meeting the criteria for the same event every time and hence you are seeing it repeatedly ?
2. Could you add another action on your alert - "Add in Triggered Alerts". And then after your alert triggers, use the dispatched search from "Activity - > Triggered Alerts" from the navigation bar and analyse a few of the results that your alert is generating.
From my understanding, All Time (real-time) search alert only searches new events coming into the indexer. Once the event is indexed, it should not show up in the search result of the all time real-time search query. This is working well in previous version of Splunk before I tried the new release of Splunk Version 7.1.0.
I have tried to remove all my alert action and just adding the "Add in Triggered Alerts" action. Once an event trigger the alert, I keep receive constant stream of triggered alerts result under the triggered alert page. Each triggered result is showing the same set of results.
The default backfill for real time searches is set to true. You might want to check that in your limits.conf and set it to false. You will find this setting in "realtime" stanza. Something like:
* Specifies if windowed real-time searches should backfill events
* Defaults to true
are you saying that you used all-time realtime in 7.0.3 and didn’t recieve repeat alerts?