topic Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0? in Alerting

Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Mon, 30 Apr 2018 10:07:21 GMT

I have been using Splunk Enterprise 7.0.3 to do real-time search alert trigger without any issues previously. Recently, I attempt to upgrade Splunk Enterprise to 7.1.0 and found some weird problem with the alert trigger.

This is the setup I have:
1. Using amqp-ta plugin to consume messages from RabbitMQ into IndexA.
2. An alert trigger running on All Time (real-time) search on IndexA to find newly indexed events.
3. The alert is trigger per result.
4. Each alert has 2 actions: Custom Action to write a result to another RabbitMQ Exchange and also log the event to another index.

Problem:
Whenever a new event is being added to IndexA, it will trigger repeatedly trigger the alert action. All the alert action search result is showing the same event that was added. This trigger will continue infinitely until I disable the Alert.

I'm not sure if there are any changes to the architecture of Real-time search alert trigger from 7.0.3 to 7.1.0. Any help would be greatly appreciated!

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

xpac — Mon, 30 Apr 2018 12:24:33 GMT

I've no ready solution, but did you try changing it to scheduled (every minute) search and see if it still happens?

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

xpac — Mon, 30 Apr 2018 22:37:06 GMT

Are you, by any chance, running on the free license? There is a known bug with realtime and the free license on 7.1.0.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Tue, 01 May 2018 03:26:14 GMT

Hi xpac, thanks for your suggestion. I did tried that and scheduled search is working fine. But my use-case would need to real-time search as I need alerts to be send out immediately when an event is detected.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Tue, 01 May 2018 03:27:06 GMT

I have tried on both free license and paid license both are having the same problem.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

xpac — Tue, 01 May 2018 10:30:55 GMT

Okay, then this seems to be a different issue. I was just guessing as I saw a few issues regarding realtime search with 7.1 recently.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

amitm05 — Tue, 01 May 2018 11:06:07 GMT

You said that your alert is set at All Time. So,
1. isn't this the reason that your search is meeting the criteria for the same event every time and hence you are seeing it repeatedly ?
2. Could you add another action on your alert - "Add in Triggered Alerts". And then after your alert triggers, use the dispatched search from "Activity - > Triggered Alerts" from the navigation bar and analyse a few of the results that your alert is generating.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

mattymo — Tue, 01 May 2018 11:41:52 GMT

are you saying that you used all-time realtime in 7.0.3 and didn’t recieve repeat alerts?

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Tue, 01 May 2018 14:41:32 GMT

yup, all works fine in 7.0.3.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Tue, 01 May 2018 14:47:21 GMT

Hi amitm05,

From my understanding, All Time (real-time) search alert only searches new events coming into the indexer. Once the event is indexed, it should not show up in the search result of the all time real-time search query. This is working well in previous version of Splunk before I tried the new release of Splunk Version 7.1.0.

I have tried to remove all my alert action and just adding the "Add in Triggered Alerts" action. Once an event trigger the alert, I keep receive constant stream of triggered alerts result under the triggered alert page. Each triggered result is showing the same set of results.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

mattymo — Tue, 01 May 2018 16:05:07 GMT

k please open a support case and provide the saved search configs. I have asked our search teams to take a look and advise.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

amitm05 — Tue, 01 May 2018 19:35:32 GMT

The default backfill for real time searches is set to true. You might want to check that in your limits.conf and set it to false. You will find this setting in "realtime" stanza. Something like:

[realtime]

default_backfill =
* Specifies if windowed real-time searches should backfill events
* Defaults to true

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Wed, 02 May 2018 10:09:36 GMT

Hi mmodestino, thank you. I have filed a support case and will be meeting up with Splunk support team on WebEx for live troubleshooting.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

agamemnon23 — Wed, 02 May 2018 18:01:06 GMT

Hi all,

I'm also having this issue since upgrading from 7.0.3 to 7.1.0 (enterprise version) yesterday.

I have multiple alerts set up as "All time (real-time)", and on previous versions of Splunk I would receive one email when the alert triggered.

Now I receive a constant stream of the same email until I disable the alert in Splunk.

I tried setting default_backfull to false in my limits.conf as described above, but it hasn't helped unfortunately.

I'm still investigating, so I'll drop any other details I find in here...

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Tue, 29 Sep 2020 19:41:15 GMT

Hi agamemnon23, I have met up with Splunk support team in live troubleshooting session and the conclusion is that the complex search query is causing the issue we are facing. And this is only happening on Splunk 7.1.0.

To illustrate more...for search query, (index="test_index"), this will only trigger one alert per result. But for search query, (index="test_index" | table _raw), the repeating alert trigger problem will reappear.

I will keep you posted if I got further updates from Splunk regarding this issue.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

agamemnon23 — Tue, 29 Sep 2020 19:24:29 GMT

Hi pweijian, thanks for posting this update - I really appreciate it!

I have just tried a similar search to the one you suggested, and I am getting the same results as you:

source="/path/to/my/syslog/files/*"
sourcetype="syslog"
| search "my_search_string"

This sends one alert ^^^

source="/path/to/my/syslog/files/*"
sourcetype="syslog"
| search "my_search_string"
| table my_field1, my_field2, my_field3

This sends multiple continuous alerts ^^^

My searches really are not that complex, so I am surprised at the difference.
I will also raise a support ticket with Splunk.

Thanks again, and likewise - I will let you know if I find out anything more.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

jitumanidas — Thu, 03 May 2018 12:59:48 GMT

This issue i faced it for our customer, but once i restarted all the services and now its working fine.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Tue, 08 May 2018 01:57:23 GMT

Hi agamemnon23, I have some updates from Splunk support. He is suspecting that this is a potential bug and he has filed an internal ticket for Splunk engineering team to look into this issue. Will keep you posted if there is more update from them.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

agamemnon23 — Thu, 10 May 2018 16:21:01 GMT

Thanks for the update peijian!

We are still experiencing the issue, but we've reconfigured most of our alerts to use simpler queries in the meantime. I haven't raised a support ticket as yet unfortunately due to other work.

I'll keep you posted when I do/when I find out any more.

Re: Why is there an All Time (real-time) Alert Trigger in Splunk Enterprise 7.1.0?

pweijian — Thu, 24 May 2018 08:24:54 GMT

Hi agamemnon23, Splunk support has confirmed this issue as a bug and have added this issues into the known issue for Version 7.1.0. There is no workaround for this issue in Version 7.1.0. and they will fix this problem in Release 7.1.2. You can refer to bug number SPL-154136 in http://docs.splunk.com/Documentation/Splunk/7.1.0/ReleaseNotes/KnownIssues