Getting Data In

How to use time elapsed after an event and a missing event to create an alert?

alexm2a
Engager

Hi there,

 

I have 2 messages that log when a job is run, which share a job_id field 

event_name=process.start 

event_name=process.end

 

I'm trying to create an alert that fires if there is an event_name=process.start , but no event_name=process.end , after 3 hours.

I've seen lots of examples of using transactions between 2 events to get the duration, but not any if an event is missing.

Many thanks, apologies if this is a noob question

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @alexm2a,

please try something like this:

index=your_index (event_name="process.start" OR event_name="process.end")
| transaction maxspan=3h startswith=event_name=process.start endswith=event_name=process.end
| where eventcount=1 AND event_name="process.start"

Ciao.

Giuseppe

alexm2a
Engager

Hi Giuseppe, thank you for your assistance.

 

When I try that I get no results even though I can see a job more than 4 hours ago that had a start but no end has happened.  

If I remove the

| where eventcount=1 AND event_name="process.start"


and use 


| table event_name, jobId 

 
I cannot see my stuck jobId in the list

Seems like the transaction is removing the event that does not have a process.end 

Any ideas? 

 

Many thanks

Alex

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...