topic Re: Why are we getting excessive number of alerts? in Splunk Search

Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 13:32:25 GMT

We have an All time (real time) alert which produced 315 alerts in the first eight hours of the day.

When running the search query of the alert for these eight hours, we get six events.

The alert itself is as simple as it gets -

index=<index name>
AND (category="Web Attack"
NOT src IN (<set of IPs>)
)

| table <set of fields>

What's going on here?

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Mon, 12 Aug 2019 15:32:42 GMT

hi @danielbb - Can you please post the alert configuration, particularly interested in the real time look back wondow

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 15:36:20 GMT

Is this the right view @Sukisen1981?

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Mon, 12 Aug 2019 16:01:15 GMT

hi @danielbb - see this, https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Specifyrealtimewindowsinyoursearch
and this
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Specifyrealtimewindowsinyoursearch

Try setting the default_backfill to false and see?

[realtime]

default_backfill =
* Specifies if windowed real-time searches should backfill events
* Defaults to true

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 16:45:54 GMT

The doc says - For windowed real-time searches, you can backfill, but we don't use windowed real-time searches.

From the UI, the only relevant option seems to be the Expires at 10 hours. Can it have anything to do with us?

Btw, where can we set "windowed" real time searches versus "all-time" real-time searches?

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Mon, 12 Aug 2019 17:03:58 GMT

Hi @danielbb
May I ask why you need a real time alert in the first place? As a thumb rule, it is better to avoid a real time alert.
Going by the frequency of the hits you mentioned earler (6 events in 8 hrs) can you not make it a scheduled alert running say every hour / hourly frequency or even on a 3 mins scheduled window?

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Mon, 12 Aug 2019 17:05:19 GMT

another option you could try perhaps is to throttle the alerts, https://docs.splunk.com/Documentation/Splunk/7.3.1/Alert/Alertexamples

So if you throttle the alert for 1 hr from the UI does it reduce your alert received counts?

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 17:11:04 GMT

Ok, makes perfect sense, however these events have indexing delay that we can't avoid. For these 6 events the delay varies between 1.7 and 12.32 minutes.

So, is there a way to schedule these "regular" alerts based on _indextime. Meaning, we'll have the alert fires for all events that got indexed in the past 15 minutes, for example.

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Wed, 30 Sep 2020 01:41:16 GMT

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 17:36:36 GMT

It shows -

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Mon, 12 Aug 2019 17:41:16 GMT

check the statistics tab carefully...any difference in minutes between indextime and _time in the table?

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 17:52:47 GMT

Not on the first page, but we have lags for some of the events.

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Wed, 30 Sep 2020 01:41:19 GMT

ok one last test and sorry, I should have asked you before you said there are only 6 events in the last eight hours...so if you use your search criteria before the evals..you should
so just add these 2 evals before your table
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | table indextime ,_time
| eval time=strptime(indextime,"%Y-%m-%d %H:%M:%S")
in the table fields add indextime and _time along with the rest.
What i am asking is now, we should have just 6 events and in these 6 events is there a difference between indextime and _time , matching what you have describec - 1-7 ~ 12 mins delay?

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 18:05:47 GMT

Right.
I found out these six events with a similar query to yours.

The real thing for me at the moment is that -

Is there a way to schedule these "regular" alerts based on _indextime. Meaning, we'll have the alert fires for all events that got indexed in the past 15 minutes, for example.

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Wed, 30 Sep 2020 01:41:25 GMT

we perhaps need 1-2 more iterations, but I believe we are making progress 🙂
_index_earliest=-15m _index_latest=now index=your index | rest of the stuff...

Now, this should calculate only events that were indexed from 15 mins ago till now...bit closer?

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 18:53:27 GMT

I think that's it - love it @Sukisen1981 !!!

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Mon, 12 Aug 2019 19:00:32 GMT

not an issue at all .would still be interesting to see if the default_backfill=false works though 🙂 🙂

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 19:02:19 GMT

haha - funny

Re: Why are we getting excessive number of alerts?

danielbb — Mon, 12 Aug 2019 19:06:34 GMT

@Sukisen1981 - please convert to an answer...

Re: Why are we getting excessive number of alerts?

Sukisen1981 — Mon, 12 Aug 2019 19:10:19 GMT

duuno which one to do , but I will convert the last comment into an answer.
I rarely get chance to fiddle around with the backend (.conf files) as it is maintained by a different vendor...this default_backfill=false looks interesting....maybe I will play around it with in my local