I have set up an alert using a "Saved search" in Splunk Enterprise Security. I am throttling alerts for an hour when src, dest, and threat identifier are same. For the most part, this alert is working fine. However, every now and then I see anomalies like the ones below:
Some time, I see that same alerts are fired twice in quick succession.
Other times, I see the same alert fired a second time before Hour limit of throttling is over (e.g. second alert after 30-40 mins of first alert).
The event ID and event hash are different for different alerts. It seems 'notable' macro is assigning a different event_id to the same event. Now, I am wondering how is that possible and how do I fix this?
What are your settings for Time Range? Specifically, earliest time, latest time, cron schedule, and schedule window?
It sounds like your saved search is running twice over the same data and generating different notable events for the same data. Throttling applies to each occurrence of the saved search, but has no affect on subsequent runs of that saved search. It could be something else but that's what I'd look at first.
earliest time : -10m@m,
latest time: @m,
cron Schedule : Every 10 mins (*/10 * * * *) - This comment page has modified my cron impression to this.
schedule window (I think, we are referring to Time window here) : In my opinion, it should not have any impact as my search has earliest and latest time, however, I have set it as "Last 15 mins".
Search is set to run on cron schedule.