topic How to make alert which will trigger when certain log occured 5 times in 15 min? in Splunk Search

How to make alert which will trigger when certain log occured 5 times in 15 min?

ivana27 — Fri, 08 Jan 2021 08:57:37 GMT

Hello,

please help. I have log (example) :

[Information] Downtime start:08/01/2021 04:39:56.997 aaxService:NotAvailable

and i would like to send email triggered by alert when this log will occur 5 times per host for 15 min?

Thank you very much

Re: How to make alert which will trigger when certain log occured 5 times in 15 min?

gcusello — Fri, 08 Jan 2021 09:00:49 GMT

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now | stats count | where count>5

Ciao.

Giuseppe

Re: How to make alert which will trigger when certain log occured 5 times in 15 min?

ivana27 — Fri, 08 Jan 2021 11:11:24 GMT

Grazie Giuseppe.

Unfornatunately doesnt work proper, its not grouping them. This is my query

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime
| where count>=2

And result is null for the last 1 hour time range which is wrong because when i change query into this

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime

I see that i have 2 logs per host for last hour

Thank you

Re: How to make alert which will trigger when certain log occured 5 times in 15 min?

gcusello — Sat, 09 Jan 2021 08:45:10 GMT

Hi @ivana27,

at first you don't need the condition "host IN (*)" because is unuseful: you always have a value for host in your events.

Then, why do you dedup for "_raw", have you some duplicated event?

At least, if you use the stats command by StartTime after dedup you'll always have only one event because it's difficoult to have in the same host more than 2 events with the same timestamp!

So if you need to have an alert when you have more than 2 events in 15 minutes period, you have to group events by time; you can do this in three ways:

using the bin (with span=15m) command before stats command;
using the timechart command with span=15m instead of stats and bin,
not setting the span period and define it in the alert (as I hinted in my answer).

For an alert the third solution is the easiest.

For a dashboard panel you can use the first or the second, the second is easier.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉