How to alert when a certain event does not occur after a certain interval?

[ashokapex]

Hi ,

I am newbie and don't know much about Splunk. I have to create an alert, but don't know how to do this. Below is my sample data.
I want to fire an alert for ActionId="13333" if it did not appear after certain interval. How do I do this? please help me.

I have these logs:
log Action1d ="12222" timestamp : 12:00 AM
log Action1d ="13333" timestamp : 12:00 AM
log Action1d ="12222" timestamp : 2:00 AM
log Action1d ="12222" timestamp : 4:00 AM

[Richfez]

As a new Splunk user, welcome! You might find some of these free Education Videos helpful.

In general it's a little harder to detect and alert on events that aren't there, but here's something simple to try.

index=myIndex ActionId="13333" earliest=-1h | stats count

You may or may not need to indicate index=whateverindex. It depends on where your data is - if you don't know, it's probably in main which is likely searchable by default so you wont' need that part.

I made it search back over the past hour earliest=-1h with the idea that I run the search once per hour. If you need shorter you'll have to do a cron job - let us know exactly how long (and if the search I wrote above is sort of right), and I can give you the cron and more specific instructions for that.

Once you have the search returning a number of how many of those events happened in the past hour, click Save As from the upper right and pick Alert.

Give it a name, make sure it's set to a Scheduled search and make it Run Every Hour.

Trigger Alert when Number of Results is equal to 0.

Make sure to only have it trigger Once.

Then for a trigger action click "Create Email" and follow that wizard (or whatever fancy thing they put into place nowadays). NOTE, you'll have to put in email server settings in Settings, Server Settings, Email Settings if you haven't done so already.

[ashokapex]

Thank you so much to answer .

Above answer is good, but i dont want to pass hard coded "ActionId="13333", which i dont know,

I have a list of ActioId in my log and i have to fire alert, if nothing happen after 1st entry of any action id.

Example :

ActionId =1 "inital Request"
ActionId =2 "inital Request"
ActionId =3 "inital Request"
ActionId =4 "inital Request"

Later Log added:
ActionId =1 "Request Change "
ActionId =2 "Request Change "
ActionId =3 "Request Change "

Later Log Added

ActionId =1 "Request Change "
ActionId =3 "Request Change "

I want fire alert for ActionId 4 , because nothing happen.

let me know any question, its really complex and i dont have any one around.

Thanks
Ashok-

[Richfez]

Ah a bit more complex, but still able to be done. Try a transaction.

Let's start with just a simple one run over the past hour or so:

index=myIndex | transaction ActionId maxspan=30m 

In it I'm assuming that the maximum time between the start and ending events in one sequence of ActionId will take less than 30 minutes. Adjust as necessary.

If you run that, do you get your events grouped together byActionId? If not, just reply back with what you see and we can straighten it out for you. If they do, your battle has been won! Well, mostly. 🙂

So if it works and creates the transactions, you should have new fields duration and eventcount. Try

index=myIndex | transaction ActionId maxspan=30m | table ActionId, _time, duration, eventcount

Just to see.

But that's only half your problem. Think for a second: if it takes an average AcctionId 15 minutes to "do its thing", then you can't reasonably find incomplete ActionIds until at least 15 minutes later, right? So, if we assume and hourly search, let's go from -75 minutes to -15 minutes. But, we want the transactions to extend to recent data, not stop 15 minutes ago. So, I was thinking about the usual way to solve this with subsearches, and thought there could be an easier way.

Let's build a search that goes back 75 minutes, create transactions out of -75m to now, but then trim off any that started in the past 15m (900 seconds).

index=myIndex earliest=-75m | transaction ActionId maxspan=30m
| eval trim_time=now()-900 | search _time<trim_time

You can pipe that to the table like above to see what it does, but that should be your search. Please check it!!! I think it'll work, and I think it's working in my test data, but my test data is not like your data!

For your once-per-hour alert, then tell it to alert when eventcount =1.

In fact, if you pipe it to the table, you could have the alert send you the actual items in an email - just click the
option under alert action Email for Inline Table