Splunk Search

Calculate total duration when many transactions are overlap in the time

wcastillocruz
Path Finder

Hello dear community,
help me on this issue please.
When using the concurrency command to find out if transactions overlap in time, and if so, is it possible to calculate the total duration of the incident taking the overlap into account.

for example :

transaction 1:
start -> 10 a.m.
end -> 11 a.m.

Transaction 2:
start -> 10:30 am
end -> 11:30 am

transaction 1 concerns process1 and transaction 2 concern porcess2 but the two transactions correspond to the same application X

before, to calculate the total duration of the incident on application X
I added the duration of transaction 1 + the duration of transaction 2.
this is the correct way when incidents (transactions) do not overlap, but when they overlap as in the previous example. the total incident duration of the application is equal to 1h30 and not to 2h.

using concurrency command can we calculate this duration?

Labels (5)
Tags (1)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| makeresults | eval _raw="time,application,transaction,action
10:00,A,1,start
12:00,A,1,end
11:00,A,2,start
13:00,A,2,end
15:00,A,3,start
17:00,A,3,end
10:00,B,1,start
12:00,B,1,end
13:00,B,2,start
15:00,B,2,end
16:00,B,3,start
18:00,B,3,end"
| multikv forceheader=1
| fields - _time _raw linecount
| eval _time=strptime(strftime(now(),"%Y/%m/%d")."T".time.":00","%Y/%m/%dT%H:%M:%S")
| fields - time
| sort application _time
| eval process=if(action="start",1,-1)
| streamstats sum(process) as concurrent by application
| eval starttime=if(concurrent=1 AND process=1,_time,null)
| eval endtime=if(concurrent=0 AND process=-1,_time,null)
| filldown starttime
| eval duration=endtime-starttime
| stats sum(duration) as duration by application
| eval duration=tostring(duration,"duration")

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Can you identify which message belongs to which transaction? If so, you can "group" the events by that identifier and determine your duration from that.

for example :

10 a.m transaction 1 start
10:30 a.m transaction 2 start
11 a.m transaction 1 end
11:30 a.m transaction 2 end

wcastillocruz
Path Finder

Hi @ITWhisperer,
Thank you for your answer.

yes, I can identify each transaction with a unique ID.
I'm already doing this. or do you mean to combine my two transactions into one?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

How do you know that transaction 1 and transaction 2 are part of the same incident?

If I understand correctly, you want the time between first start (10 am) and last end (11:30 am), is that right?

If the first transaction finished at 11 am and the second one started at 12pm, both transactions taking an hour, would you want the duration to be 2 hours or 3 hours?

wcastillocruz
Path Finder

@ITWhisperer,
Thanks again for your reply.

well,
I identify each transaction thanks to a series of data that each transaction contains.
this is the start of my search:

index = index1
| eval ID = Service + "_" + Env + "_" + Apps + "_" + Function
| addinfo
| transaction ID startswith = (severity = 2) endswith = (severity = 0 OR severity = 1 OR severity = -1) maxevents = 4

"If I understand correctly, you want the time between first start (10 am) and last end (11:30 am), is that right?"
yes, at this time the duration of the incident for the Application is 1:30 hour.

but if
transaction 1 =
start 10:00 AM
end 11:00 AM

transaction 2 =
start 11:00 AM
end 12:00 PM

in this case the duration is equal to the sum of the duration transaction 1 + duration transaction 2
note: I have several Applications, and my ID allows me to separate transactions.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

It is still not clear what ties transaction 1 to transaction 2. Suppose there was a third transaction (transaction 3) which was part of a different "incident". How would you know whether to consider the start of transaction 1 and the end of transaction 2, or the start of transaction 1 and the end of transaction 3?

0 Karma

wcastillocruz
Path Finder

Here is an example of two transactions and underlined in blue which allows me to differentiate them

 

wcastillocruz_0-1616665472529.png

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
index = index1
| eval IncidentID = Service + "_" + Env + "_" + Apps
| stats first(_time) as start last(_time) as end by IncidentID
| eval duration=end-start

wcastillocruz
Path Finder

@ITWhisperer 
it is not as easy as it may seem.
I have listed several months of events, this is not a specific case.
moreover I can index several events for the same application in a day, and it is possible that these events do not overlap and which do not follow one another.

by following your solution I get this:

wcastillocruz_1-1616668116901.png

while for application A for example, nothing happened between 1:00 p.m. and 2:00 p.m. and with your solution I add this time to my total incident duration

thank you very much anyway.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK I got it - I had misread your first graphic. Yes, it is a bit more complicated.

ITWhisperer
SplunkTrust
SplunkTrust
| makeresults | eval _raw="time,application,transaction,action
10:00,A,1,start
12:00,A,1,end
11:00,A,2,start
13:00,A,2,end
15:00,A,3,start
17:00,A,3,end
10:00,B,1,start
12:00,B,1,end
13:00,B,2,start
15:00,B,2,end
16:00,B,3,start
18:00,B,3,end"
| multikv forceheader=1
| fields - _time _raw linecount
| eval _time=strptime(strftime(now(),"%Y/%m/%d")."T".time.":00","%Y/%m/%dT%H:%M:%S")
| fields - time
| sort application _time
| eval process=if(action="start",1,-1)
| streamstats sum(process) as concurrent by application
| eval starttime=if(concurrent=1 AND process=1,_time,null)
| eval endtime=if(concurrent=0 AND process=-1,_time,null)
| filldown starttime
| eval duration=endtime-starttime
| stats sum(duration) as duration by application
| eval duration=tostring(duration,"duration")

ITWhisperer
SplunkTrust
SplunkTrust

Essentially, use streamstats to keep a running total of active processes, note when the first of an overlapping set starts and when the last of the overlapping set ends, then find the difference and add all the differences to give a total processing time

wcastillocruz
Path Finder

it's not you but me who explains badly

Thank a lot

0 Karma

wcastillocruz
Path Finder

that explains better than me 🙂 

wcastillocruz_0-1616664266345.png

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

That looks like you want the first message from Application A and the last message from Application A. Can an application handle more than one incident? Do you need to be able to distinguish between the incidents an application is handling? How would you do this?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...