topic Re: How to set up Splunk alert for changing logs? in Alerting

How to set up Splunk alert for changing logs?

logloganathan — Mon, 16 Apr 2018 15:20:51 GMT

I want to setup alert for changing logs.
The service name changed to success from failure then it writes to a log file and vice-versa but we don't know when its value change occurs. It may occur in 5 minutes or 1 hour or 1 day and the log file contains information like:

success
failure
success
failure

I want to alert if only failure happens for a particular time, say 5 minutes (no success occur for 5 minutes)
if success occur after failure then it wont alert
Please help me on this issue

Re: How to set up Splunk alert for changing logs?

ssadanala1 — Tue, 17 Apr 2018 12:48:01 GMT

You need to schedule the matching search for every 5 minutes.

Splunk search would be like this base search |stats count by respecfield |where count >0 and will set the trigger ocndition to send an email if results are greater than 0 .

Splunk runs the search for every 5 mins as scheduled but only notifies you when the result count is greater than condition given .

Re: How to set up Splunk alert for changing logs?

logloganathan — Tue, 17 Apr 2018 12:54:19 GMT

it wont help for my requirement.
Thanks for your help

Re: How to set up Splunk alert for changing logs?

logloganathan — Wed, 18 Apr 2018 09:41:33 GMT

Could anyone please help me in this issue?

Re: How to set up Splunk alert for changing logs?

Javip — Wed, 18 Apr 2018 10:17:33 GMT

Hi!

How many time can pass between "failure" and "success" events in your log?

Re: How to set up Splunk alert for changing logs?

logloganathan — Wed, 18 Apr 2018 13:19:15 GMT

Hi Javip,
within 2 minutes...failure come back to success..but sometime it not happen and i need alert for this.
Could you please help

Re: How to set up Splunk alert for changing logs?

Javip — Wed, 18 Apr 2018 13:44:08 GMT

Hi,

As ssadanala1 said create a cheduled search to do something like this every 5 minutes, asking for your last 24h events perhaps:

| inputlookup test_failure.csv 
| search status="failure" OR status="succcess" 
| head 1 
| eval event_time=_time, time_now=Now() 
| eval diff=time_now-event_time 
| search diff>=300 AND status="failure"

There, in my example, I filter events with status="failure" OR status="succcess"
Then I only wants the most recent event, I use head 1
The next I do is to create 2 timestamp variables, one with event time and the other one the result of Now() command
I rest both values and finally I search if my last event has status=failure and if diff value is >300 then I send an alert.

Obviously to can add more fields and send or use them for your alert/email.

Tell me if these ideas can help you.

Re: How to set up Splunk alert for changing logs?

logloganathan — Wed, 18 Apr 2018 14:04:18 GMT

but if success comes first and followed by failure then it wont alert within 5 minutes of time frame

Re: How to set up Splunk alert for changing logs?

Javip — Wed, 18 Apr 2018 14:18:09 GMT

if success comes first, it's your most recent event, nothing happens

if failure is your most recent event, then you'll send an alert when passed at least 5 minutes

you only send alert or not according to two conditions: status="failure" and you've not received a success event in 5 minutes or more.

Re: How to set up Splunk alert for changing logs?

logloganathan — Wed, 18 Apr 2018 14:46:07 GMT

Wow!!.....greatest knowledge !!..

Re: How to set up Splunk alert for changing logs?

logloganathan — Tue, 29 Sep 2020 19:08:56 GMT

Hi Javip,
i have modified query for this requirement

index=ABC host=ABC sourcetype=ABC "Success" OR "Failure"
| head 1
| eval event_time=_time, time_now=Now()
| eval diff=time_now-event_time
| where diff>=300 AND like(_raw, "%Failure%")