Splunk Search

How to make alert which will trigger when certain log occured 5 times in 15 min?

ivana27
Path Finder

Hello,

please help. I have log (example) :

[Information] Downtime start:08/01/2021 04:39:56.997 aaxService:NotAvailable

and i would like to send email triggered by alert when this log will occur 5 times per host for 15 min?

Thank you very much

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

You have to schedule an alert every 15 minutes, running a search like this:

index=your_index "aaxService:NotAvailable" earliest=-15m@m latest=now
| stats count
| where count>5

Ciao.

Giuseppe

0 Karma

ivana27
Path Finder

Grazie Giuseppe.

Unfornatunately doesnt work proper, its not grouping them. This is my query

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime
| where count>=2

And result is null for the last 1 hour time range which is wrong because when i change query into this

index=pkg_prespvm "aaxService:NotAvailable" sourcetype = sigma host IN (*)
| dedup _raw
| eval StartTime=strftime(_time, "%d/%m/%Y %H:%M:%S")
| stats count as "NumberErrors" by host,StartTime

I see that i have 2 logs per host for last hour

ivana27_0-1610104271542.png

 

Thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ivana27,

at first you don't need the condition "host IN (*)" because is unuseful: you always have a value for host in your events.

Then, why do you dedup for "_raw", have you some duplicated event?

At least, if you use the stats command by StartTime after dedup you'll always have only one event because it's difficoult to have in the same host more than 2 events with the same timestamp!

So if you need to have an alert when you have more than 2 events in 15 minutes period, you have to group events by time; you can do this in three ways:

  • using the bin (with span=15m) command before stats command;
  • using the timechart command with span=15m instead of stats and bin,
  • not setting the span period and define it in the alert (as I hinted in my answer).

For an alert the third solution is the easiest.

For a dashboard panel you can use the first or the second, the second is easier.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...