Alerting

Customizing the time scope of alerts?

AlexRP
Explorer

First time posting here, and I'm a new user to Splunk. I'd love to get some advice on setting up an alert.

I want it to trigger at 8am, 12pm, 4pm, and 8pm. I've set my Cron schedule to "* 8,12,16,20 * * *".  For the search's time scope, I'd like the following

  • 8am trigger should have a search range of -12 hours to the current time.
  • 12pm, 4pm, and 8pm triggers should have a search range of -4 hours to the current time

I've set my range time to be -12 (earliest) to the current time (now), but the 12pm, 4pm, and 8pm triggers are getting results that had already been part of the result set from the 8am trigger.

Does Splunk know when a result has been previously reported, or is there a way I can filter those out using the search query?

How does the expire parameter work? Can I leverage it in a way that I won't get previously reported results?

Would I have to set up a separate alert for the 8am trigger, even though (aside from the 12 hour lookback) it does the same thing and serves the same purpose of an alert that would encompass the other times?

Here's what search and the time range looks like on my alert. Thanks in advance for the guidance!

index="slPlatform" environment="Development" Application="AP_post_EmployeePayload_To_EmployeeProfile" | where eventLogLevel like "CRITICAL%"

 

AlexRP_0-1671242080930.png

 

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @AlexRP ,

you have to add an additional condition to your search:

<your_search> earliest=-12h@h latest=@h
| eval timeframe=if(strftime(now(),"%H")=8,43200,14400)
| where now()-_time<=timeframe
| ...

In this way you take only the events in the timeframe, that's variable, and discard the other events.

Ciao.

Giuseppe

View solution in original post

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @AlexRP ,

you have to add an additional condition to your search:

<your_search> earliest=-12h@h latest=@h
| eval timeframe=if(strftime(now(),"%H")=8,43200,14400)
| where now()-_time<=timeframe
| ...

In this way you take only the events in the timeframe, that's variable, and discard the other events.

Ciao.

Giuseppe

Tags (1)
0 Karma

AlexRP
Explorer

wow! thank you so much for this solution. i didn't realize i could open up the query in this way.  i've learned something new. thanks again!

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @AlexRP ,

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

AlexRP
Explorer

Hey Gisuppe

I was trying out this solution. Can you clarify something for me, please? In the following, what does 43100 and 14400 represent? 

Is it possible just to add 8 hours to the 'earliest' variable and have my search use that value instead, effectively always using the last 4 hours when my search triggers at 12p, 4p, and 8p?

eval timeframe=if(strftime(now(),"%H")=8,43200,14400)

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @AlexRP,

43100 and 14400 are the seconds in 12 and 4 hours: 3600*12 and 3600*4.

About your second question, you can customize the eval condition as you like, adding other conditions and/or values.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...