Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

[pweijian]

I have been using Splunk Enterprise 7.0.3 to do real-time search alert trigger without any issues previously. Recently, I attempt to upgrade Splunk Enterprise to 7.1.0 and found some weird problem with the alert trigger.

This is the setup I have:
1. Using amqp-ta plugin to consume messages from RabbitMQ into IndexA.
2. An alert trigger running on All Time (real-time) search on IndexA to find newly indexed events.
3. The alert is trigger per result.
4. Each alert has 2 actions: Custom Action to write a result to another RabbitMQ Exchange and also log the event to another index.

Problem:
Whenever a new event is being added to IndexA, it will trigger repeatedly trigger the alert action. All the alert action search result is showing the same event that was added. This trigger will continue infinitely until I disable the Alert.

I'm not sure if there are any changes to the architecture of Real-time search alert trigger from 7.0.3 to 7.1.0. Any help would be greatly appreciated!

[jd0323fhl]

One other way to fix this if you can get away with near real time is to schedule the alerts for a one minute interval using the cron scheduling and set the time frame for one minute as well.

[jitumanidas]

This issue i faced it for our customer, but once i restarted all the services and now its working fine.

[pweijian · Accepted solution]

Hi agamemnon23, I have met up with Splunk support team in live troubleshooting session and the conclusion is that the complex search query is causing the issue we are facing. And this is only happening on Splunk 7.1.0.

To illustrate more...for search query, (index="test_index"), this will only trigger one alert per result. But for search query, (index="test_index" | table _raw), the repeating alert trigger problem will reappear.

I will keep you posted if I got further updates from Splunk regarding this issue.

[agamemnon23]

Hi pweijian, thanks for posting this update - I really appreciate it!

I have just tried a similar search to the one you suggested, and I am getting the same results as you:

source="/path/to/my/syslog/files/*"
sourcetype="syslog"
| search "my_search_string"

This sends one alert ^^^

source="/path/to/my/syslog/files/*"
sourcetype="syslog"
| search "my_search_string"
| table my_field1, my_field2, my_field3

This sends multiple continuous alerts ^^^

My searches really are not that complex, so I am surprised at the difference.
I will also raise a support ticket with Splunk.

Thanks again, and likewise - I will let you know if I find out anything more.

[pweijian]

Hi agamemnon23, I have some updates from Splunk support. He is suspecting that this is a potential bug and he has filed an internal ticket for Splunk engineering team to look into this issue. Will keep you posted if there is more update from them.

[agamemnon23]

Thanks for the update peijian!

We are still experiencing the issue, but we've reconfigured most of our alerts to use simpler queries in the meantime. I haven't raised a support ticket as yet unfortunately due to other work.

I'll keep you posted when I do/when I find out any more.

[agamemnon23]

Hi all,

I'm also having this issue since upgrading from 7.0.3 to 7.1.0 (enterprise version) yesterday.

I have multiple alerts set up as "All time (real-time)", and on previous versions of Splunk I would receive one email when the alert triggered.

Now I receive a constant stream of the same email until I disable the alert in Splunk.

I tried setting default_backfull to false in my limits.conf as described above, but it hasn't helped unfortunately.

I'm still investigating, so I'll drop any other details I find in here...

[pweijian]

Hi agamemnon23, Splunk support has confirmed this issue as a bug and have added this issues into the known issue for Version 7.1.0. There is no workaround for this issue in Version 7.1.0. and they will fix this problem in Release 7.1.2. You can refer to bug number SPL-154136 in http://docs.splunk.com/Documentation/Splunk/7.1.0/ReleaseNotes/KnownIssues

[agamemnon23]

Hi pweijian,

Good news - Splunk 7.1.2 has been released and they have appeared to fix the problem!
SPL-154136 is mentioned as "fixed" in the release notes.

I've upgraded to 7.1.2 and tested most of my alerts, and they all seem to be working correctly now.

Thanks again for your help!

[agamemnon23]

Thanks pweijian!

We still have most of our alerts disabled and are patiently waiting for the release of 7.1.2

I upgraded to 7.1.1 recently, although the issue still persists. Strangely, SPL-154136 is neither mentioned in the "known issues" or the "fixed issues" for 7.1.1, so hopefully it hasn't disappeared into a black hole!

[mattymo]

are you saying that you used all-time realtime in 7.0.3 and didn’t recieve repeat alerts?

[pweijian]

yup, all works fine in 7.0.3.

[mattymo]

k please open a support case and provide the saved search configs. I have asked our search teams to take a look and advise.

[pweijian]

Hi mmodestino, thank you. I have filed a support case and will be meeting up with Splunk support team on WebEx for live troubleshooting.

[amitm05]

You said that your alert is set at All Time. So,
1. isn't this the reason that your search is meeting the criteria for the same event every time and hence you are seeing it repeatedly ?
2. Could you add another action on your alert - "Add in Triggered Alerts". And then after your alert triggers, use the dispatched search from "Activity - > Triggered Alerts" from the navigation bar and analyse a few of the results that your alert is generating.

[pweijian]

Hi amitm05,

From my understanding, All Time (real-time) search alert only searches new events coming into the indexer. Once the event is indexed, it should not show up in the search result of the all time real-time search query. This is working well in previous version of Splunk before I tried the new release of Splunk Version 7.1.0.

I have tried to remove all my alert action and just adding the "Add in Triggered Alerts" action. Once an event trigger the alert, I keep receive constant stream of triggered alerts result under the triggered alert page. Each triggered result is showing the same set of results.

[amitm05]

The default backfill for real time searches is set to true. You might want to check that in your limits.conf and set it to false. You will find this setting in "realtime" stanza. Something like:

[realtime]

default_backfill =
* Specifies if windowed real-time searches should backfill events
* Defaults to true

[xpac]

Are you, by any chance, running on the free license? There is a known bug with realtime and the free license on 7.1.0.

[pweijian]

I have tried on both free license and paid license both are having the same problem.

[xpac]

Okay, then this seems to be a different issue. I was just guessing as I saw a few issues regarding realtime search with 7.1 recently.