I want to setup alert for changing logs.
The service name changed to success from failure then it writes to a log file and vice-versa but we don't know when its value change occurs. It may occur in 5 minutes or 1 hour or 1 day and the log file contains information like:
success
failure
success
failure
I want to alert if only failure happens for a particular time, say 5 minutes (no success occur for 5 minutes)
if success occur after failure then it wont alert
Please help me on this issue
Hi,
As ssadanala1 said create a cheduled search to do something like this every 5 minutes, asking for your last 24h events perhaps:
| inputlookup test_failure.csv
| search status="failure" OR status="succcess"
| head 1
| eval event_time=_time, time_now=Now()
| eval diff=time_now-event_time
| search diff>=300 AND status="failure"
There, in my example, I filter events with status="failure" OR status="succcess"
Then I only wants the most recent event, I use head 1
The next I do is to create 2 timestamp variables, one with event time and the other one the result of Now() command
I rest both values and finally I search if my last event has status=failure and if diff value is >300 then I send an alert.
Obviously to can add more fields and send or use them for your alert/email.
Tell me if these ideas can help you.
J.
Hi,
As ssadanala1 said create a cheduled search to do something like this every 5 minutes, asking for your last 24h events perhaps:
| inputlookup test_failure.csv
| search status="failure" OR status="succcess"
| head 1
| eval event_time=_time, time_now=Now()
| eval diff=time_now-event_time
| search diff>=300 AND status="failure"
There, in my example, I filter events with status="failure" OR status="succcess"
Then I only wants the most recent event, I use head 1
The next I do is to create 2 timestamp variables, one with event time and the other one the result of Now() command
I rest both values and finally I search if my last event has status=failure and if diff value is >300 then I send an alert.
Obviously to can add more fields and send or use them for your alert/email.
Tell me if these ideas can help you.
J.
but if success comes first and followed by failure then it wont alert within 5 minutes of time frame
if success comes first, it's your most recent event, nothing happens
if failure is your most recent event, then you'll send an alert when passed at least 5 minutes
you only send alert or not according to two conditions: status="failure" and you've not received a success event in 5 minutes or more.
Wow!!.....greatest knowledge !!..
Hi Javip,
i have modified query for this requirement
index=ABC host=ABC sourcetype=ABC "Success" OR "Failure"
| head 1
| eval event_time=_time, time_now=Now()
| eval diff=time_now-event_time
| where diff>=300 AND like(_raw, "%Failure%")
You need to schedule the matching search for every 5 minutes.
Splunk search would be like this base search |stats count by respecfield |where count >0 and will set the trigger ocndition to send an email if results are greater than 0 .
Splunk runs the search for every 5 mins as scheduled but only notifies you when the result count is greater than condition given .
it wont help for my requirement.
Thanks for your help
Could anyone please help me in this issue?
Hi!
How many time can pass between "failure" and "success" events in your log?
J.
Hi Javip,
within 2 minutes...failure come back to success..but sometime it not happen and i need alert for this.
Could you please help