Splunk Search

How to make alert which will trigger when certain log occured 5 times in 15 min?

ivana27
Path Finder

Hello,

please help. I have log (example) :

[Information] Downtime start:08/01/2021 04:39:56.997 aaxService:NotAvailable

and i would like to send email triggered by alert when this log will occur 5 times per host for 15 min?

Thank you very much

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

0 Karma

ivana27
Path Finder

Grazie Giuseppe.

Unfornatunately doesnt work proper, its not grouping them. This is my query

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime
| where count>=2

And result is null for the last 1 hour time range which is wrong because when i change query into this

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime

I see that i have 2 logs per host for last hour

ivana27_0-1610104271542.png

 

Thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

at first you don't need the condition "host IN (*)" because is unuseful: you always have a value for host in your events.

Then, why do you dedup for "_raw", have you some duplicated event?

At least, if you use the stats command by StartTime after dedup you'll always have only one event because it's difficoult to have in the same host more than 2 events with the same timestamp!

So if you need to have an alert when you have more than 2 events in 15 minutes period, you have to group events by time; you can do this in three ways:

  • using the bin (with span=15m) command before stats command;
  • using the timechart command with span=15m instead of stats and bin,
  • not setting the span period and define it in the alert (as I hinted in my answer).

For an alert the third solution is the easiest.

For a dashboard panel you can use the first or the second, the second is easier.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

Get Updates on the Splunk Community!

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

New Release | Splunk Cloud Platform 10.1.2507

Hello Splunk Community!We are thrilled to announce the General Availability of Splunk Cloud Platform 10.1.2507 ...

🌟 From Audit Chaos to Clarity: Welcoming Audit Trail v2

🗣 You Spoke, We Listened  Audit Trail v2 wasn’t written in isolation—it was shaped by your voices.  In ...