How to set up Splunk alert for changing logs?

[logloganathan]

I want to setup alert for changing logs.
The service name changed to success from failure then it writes to a log file and vice-versa but we don't know when its value change occurs. It may occur in 5 minutes or 1 hour or 1 day and the log file contains information like:

success
failure
success
failure

I want to alert if only failure happens for a particular time, say 5 minutes (no success occur for 5 minutes)
if success occur after failure then it wont alert
Please help me on this issue

[Javip · Accepted solution]

Hi,

As ssadanala1 said create a cheduled search to do something like this every 5 minutes, asking for your last 24h events perhaps:

| inputlookup test_failure.csv 
| search status="failure" OR status="succcess" 
| head 1 
| eval event_time=_time, time_now=Now() 
| eval diff=time_now-event_time 
| search diff>=300 AND status="failure"

There, in my example, I filter events with status="failure" OR status="succcess"
Then I only wants the most recent event, I use head 1
The next I do is to create 2 timestamp variables, one with event time and the other one the result of Now() command
I rest both values and finally I search if my last event has status=failure and if diff value is >300 then I send an alert.

Obviously to can add more fields and send or use them for your alert/email.

Tell me if these ideas can help you.

J.

[logloganathan]

but if success comes first and followed by failure then it wont alert within 5 minutes of time frame

[Javip]

if success comes first, it's your most recent event, nothing happens

if failure is your most recent event, then you'll send an alert when passed at least 5 minutes

you only send alert or not according to two conditions: status="failure" and you've not received a success event in 5 minutes or more.

[logloganathan]

Wow!!.....greatest knowledge !!..

[logloganathan]

Hi Javip,
i have modified query for this requirement

index=ABC host=ABC sourcetype=ABC "Success" OR "Failure"
| head 1
| eval event_time=_time, time_now=Now()
| eval diff=time_now-event_time
| where diff>=300 AND like(_raw, "%Failure%")

[ssadanala1]

You need to schedule the matching search for every 5 minutes.

Splunk search would be like this base search |stats count by respecfield |where count >0 and will set the trigger ocndition to send an email if results are greater than 0 .

Splunk runs the search for every 5 mins as scheduled but only notifies you when the result count is greater than condition given .

[logloganathan]

it wont help for my requirement.
Thanks for your help

[logloganathan]

Could anyone please help me in this issue?

[Javip]

Hi!

How many time can pass between "failure" and "success" events in your log?

J.

[logloganathan]

Hi Javip,
within 2 minutes...failure come back to success..but sometime it not happen and i need alert for this.
Could you please help