Alerting

How to set up Splunk alert for changing logs?

logloganathan
Motivator

I want to setup alert for changing logs.
The service name changed to success from failure then it writes to a log file and vice-versa but we don't know when its value change occurs. It may occur in 5 minutes or 1 hour or 1 day and the log file contains information like:

success
failure
success
failure

I want to alert if only failure happens for a particular time, say 5 minutes (no success occur for 5 minutes)
if success occur after failure then it wont alert
Please help me on this issue

Tags (3)
0 Karma
1 Solution

Javip
Path Finder

Hi,

As ssadanala1 said create a cheduled search to do something like this every 5 minutes, asking for your last 24h events perhaps:

| inputlookup test_failure.csv 
| search status="failure" OR status="succcess" 
| head 1 
| eval event_time=_time, time_now=Now() 
| eval diff=time_now-event_time 
| search diff>=300 AND status="failure"

There, in my example, I filter events with status="failure" OR status="succcess"
Then I only wants the most recent event, I use head 1
The next I do is to create 2 timestamp variables, one with event time and the other one the result of Now() command
I rest both values and finally I search if my last event has status=failure and if diff value is >300 then I send an alert.

Obviously to can add more fields and send or use them for your alert/email.

Tell me if these ideas can help you.

J.

View solution in original post

Javip
Path Finder

Hi,

As ssadanala1 said create a cheduled search to do something like this every 5 minutes, asking for your last 24h events perhaps:

| inputlookup test_failure.csv 
| search status="failure" OR status="succcess" 
| head 1 
| eval event_time=_time, time_now=Now() 
| eval diff=time_now-event_time 
| search diff>=300 AND status="failure"

There, in my example, I filter events with status="failure" OR status="succcess"
Then I only wants the most recent event, I use head 1
The next I do is to create 2 timestamp variables, one with event time and the other one the result of Now() command
I rest both values and finally I search if my last event has status=failure and if diff value is >300 then I send an alert.

Obviously to can add more fields and send or use them for your alert/email.

Tell me if these ideas can help you.

J.

View solution in original post

logloganathan
Motivator

but if success comes first and followed by failure then it wont alert within 5 minutes of time frame

0 Karma

Javip
Path Finder

if success comes first, it's your most recent event, nothing happens

if failure is your most recent event, then you'll send an alert when passed at least 5 minutes

you only send alert or not according to two conditions: status="failure" and you've not received a success event in 5 minutes or more.

logloganathan
Motivator

Wow!!.....greatest knowledge !!..

0 Karma

logloganathan
Motivator

Hi Javip,
i have modified query for this requirement

index=ABC host=ABC sourcetype=ABC "Success" OR "Failure"
| head 1
| eval event_time=_time, time_now=Now()
| eval diff=time_now-event_time
| where diff>=300 AND like(_raw, "%Failure%")

0 Karma

ssadanala1
Contributor

You need to schedule the matching search for every 5 minutes.

Splunk search would be like this base search |stats count by respecfield |where count >0 and will set the trigger ocndition to send an email if results are greater than 0 .

Splunk runs the search for every 5 mins as scheduled but only notifies you when the result count is greater than condition given .

logloganathan
Motivator

it wont help for my requirement.
Thanks for your help

0 Karma

logloganathan
Motivator

Could anyone please help me in this issue?

0 Karma

Javip
Path Finder

Hi!

How many time can pass between "failure" and "success" events in your log?

J.

0 Karma

logloganathan
Motivator

Hi Javip,
within 2 minutes...failure come back to success..but sometime it not happen and i need alert for this.
Could you please help

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.