Splunk Search

How to make alert which will trigger when certain log occured 5 times in 15 min?

ivana27
Path Finder

Hello,

please help. I have log (example) :

[Information] Downtime start:08/01/2021 04:39:56.997 aaxService:NotAvailable

and i would like to send email triggered by alert when this log will occur 5 times per host for 15 min?

Thank you very much

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

0 Karma

ivana27
Path Finder

Grazie Giuseppe.

Unfornatunately doesnt work proper, its not grouping them. This is my query

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime
| where count>=2

And result is null for the last 1 hour time range which is wrong because when i change query into this

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime

I see that i have 2 logs per host for last hour

ivana27_0-1610104271542.png

 

Thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

at first you don't need the condition "host IN (*)" because is unuseful: you always have a value for host in your events.

Then, why do you dedup for "_raw", have you some duplicated event?

At least, if you use the stats command by StartTime after dedup you'll always have only one event because it's difficoult to have in the same host more than 2 events with the same timestamp!

So if you need to have an alert when you have more than 2 events in 15 minutes period, you have to group events by time; you can do this in three ways:

  • using the bin (with span=15m) command before stats command;
  • using the timechart command with span=15m instead of stats and bin,
  • not setting the span period and define it in the alert (as I hinted in my answer).

For an alert the third solution is the easiest.

For a dashboard panel you can use the first or the second, the second is easier.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

Get Updates on the Splunk Community!

🌟 From Audit Chaos to Clarity: Welcoming Audit Trail v2

🗣 You Spoke, We Listened Audit Trail v2 wasn’t written in isolation—it was shaped by your voices. In ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

 Prepare to elevate your security operations with the powerful upgrade to Splunk Enterprise Security 8.x! This ...

Get Early Access to AI Playbook Authoring: Apply for the Alpha Private Preview ...

Passionate about security automation? Apply now to our AI Playbook Authoring Alpha private preview ...