Getting Data In

How to use time elapsed after an event and a missing event to create an alert?

alexm2a
Engager

Hi there,

 

I have 2 messages that log when a job is run, which share a job_id field 

event_name=process.start 

event_name=process.end

 

I'm trying to create an alert that fires if there is an event_name=process.start , but no event_name=process.end , after 3 hours.

I've seen lots of examples of using transactions between 2 events to get the duration, but not any if an event is missing.

Many thanks, apologies if this is a noob question

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @alexm2a,

please try something like this:

index=your_index (event_name="process.start" OR event_name="process.end")
| transaction maxspan=3h startswith=event_name=process.start endswith=event_name=process.end
| where eventcount=1 AND event_name="process.start"

Ciao.

Giuseppe

alexm2a
Engager

Hi Giuseppe, thank you for your assistance.

 

When I try that I get no results even though I can see a job more than 4 hours ago that had a start but no end has happened.  

If I remove the

| where eventcount=1 AND event_name="process.start"


and use 


| table event_name, jobId 

 
I cannot see my stuck jobId in the list

Seems like the transaction is removing the event that does not have a process.end 

Any ideas? 

 

Many thanks

Alex

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...