Splunk Search

How to Check the times from 2 Events and alert if over 15 mins

LizAndy123
Path Finder

So I have an Index which contains the following

"Starting iteration"on 1 event and "Stopping iteration" on another event

I want to get the time taken from event 1 to event 2.

And if over 15 mins then I can setup an alert to warn me 

 

Labels (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

That sounds right 😀

View solution in original post

0 Karma

LizAndy123
Path Finder

So I have looked at my Events

And it does have a Common Unique ID on each start and stop event

Example

Starting iteration - 17000000

Stopping iteration - 17000000

So I guess I need to extract that number and perform a duration for this.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

That sounds right 😀

0 Karma

LizAndy123
Path Finder

So I am 99% there

New Search

Index=xxxxx "Starting iteration" OR "Stopping iteration"

| timechart count spac=15m by Series

| rex "Starting\siteration[\s\-]+[(?<start_reg_id>[^\s]+)"

| rex "Stopping\siteration[\s\-]+(?<stop_reg_id>[^:\s]+)"

| eval Start_Reg_ID=start_reg_id

| eval Stop_Reg_ID=stop_reg_id

 

When I run I get a count of 2 - Which is the start and the stop of the same ID

It shows the Time Stamp and a count of 2 - when I see the events it is correct

What I need to do is tell me if it was over 15 mins - maybe I need to redo the Timespan or put more time comments in...sorry I am a newbe

I have got the result and it collorates the start and finish but now how do I say over 15 mins thats long

0 Karma

LizAndy123
Path Finder

So I have got it working 99%

I did something like this

Index=xxxxxx "Starting iteration" OR "Stopping iteration"

| stats earliest(_time) as Start,latest(_time) as Stopped

| eval Taken=tostring(Stopped-Start)

| eval Taken=Taken/60

| eval Time_Taken=(if(Taken>15,"Not Good","Good"))

| where Time_Taken="Not Good"

| table Start Stopper Time_Taken

Now it shows Not Good if over 15 mins

The issue is how to set the alert properly - as if I set to check every 15 mins - it may overlap 2 starts - Example Started at 7pm and finished at 7.08pm Alert checks at like 7.25pm for the last 15 mins and it sees 7.08pm at Stopped then 7.15pm Start and maybe finished at 7.24pm - If that make sense to you guru's

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

How are the two events linked? Can there be more that one "start" before any "stops"? Can "start/Stop" pair be intertwined? How frequently do you want to check?

0 Karma

LizAndy123
Path Finder

So the Events basically have a start every 15 mins

We have one event saying Starting and when it finishes within the 15 mins then it will says Stopped 

 

once I know that time then I can alert the team IF it takes over 15 mins since it could be an issue

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try setting your alert to look back at least 15 minutes and use a search like this

| eval starttime=if(event="Starting",_time,null())
| eval stoptime=if(event="Stopping",_time,null())
| sort 0 _time desc
| streamstats time_window=15m latest(stoptime) as nextStop
| eval alert=if(isnull(nextStop) and time() - starttime > 15*60, "missing", null())
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...