Splunk Enterprise Security

Splunk Enterprise Security: Why are notable events not created and notable alert_action is not triggered?

Splunk Employee
Splunk Employee

One of my Splunk Enterprise Security customer's complained that sometimes the notable events are not created even when the corresponding raw data is there.
So I checked the scheduler log and found that there were cases that the "notable" alert_action was not triggered when the result_count was not "0".

Please see the red box in the screen capture below. This correlation search runs every 2 minutes. At 13:00:54, result_count was "1", but "notable" alert_action was not triggered.

alt text

This problem (missing notable events) happens randomly across almost all correlation searches they have, several times in a day.
Unfortunately, I couldn't find any clue from the splunkd log so far. There were no error or warning messages when these problems happened.

My customer is running ES 4.1.1 on Splunk 6.4.3. (search head clustering + multisite indexer clustering environment)

Any suggestion for troubleshooting would be very appreciated.
Thank you in advance.

0 Karma

Splunk Employee
Splunk Employee

Check the throttling of notable events for that search. If the search is throttling notable events, and the "randomly" skipped notable events fall into that time window, that would explain why a notable event isn't being created.

0 Karma

Engager

Did you find a resolution to this problem? I'm experiencing a very similar issue. Thanks

0 Karma

Splunk Employee
Splunk Employee

Hi smoir. Thank you for your answer.
Already checked that. Throttling was not configured.

0 Karma

Splunk Employee
Splunk Employee

Hi Smoir. I just found an ongoing support case which has very similar symptom with my case. I think, I need to contact support team. Thank you.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!