Splunk Search

How to make alert which will trigger when certain log occured 5 times in 15 min?

ivana27
Path Finder

Hello,

please help. I have log (example) :

[Information] Downtime start:08/01/2021 04:39:56.997 aaxService:NotAvailable

and i would like to send email triggered by alert when this log will occur 5 times per host for 15 min?

Thank you very much

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

0 Karma

ivana27
Path Finder

Grazie Giuseppe.

Unfornatunately doesnt work proper, its not grouping them. This is my query

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime
| where count>=2

And result is null for the last 1 hour time range which is wrong because when i change query into this

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime

I see that i have 2 logs per host for last hour

ivana27_0-1610104271542.png

 

Thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

at first you don't need the condition "host IN (*)" because is unuseful: you always have a value for host in your events.

Then, why do you dedup for "_raw", have you some duplicated event?

At least, if you use the stats command by StartTime after dedup you'll always have only one event because it's difficoult to have in the same host more than 2 events with the same timestamp!

So if you need to have an alert when you have more than 2 events in 15 minutes period, you have to group events by time; you can do this in three ways:

  • using the bin (with span=15m) command before stats command;
  • using the timechart command with span=15m instead of stats and bin,
  • not setting the span period and define it in the alert (as I hinted in my answer).

For an alert the third solution is the easiest.

For a dashboard panel you can use the first or the second, the second is easier.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...