Splunk Query to get time taken by each application in a transaction

[jyothishtj]

Hi All,

I am new to splunk. I got a transaction which is flowing through multiple applications. I got a requirement to find out the time taken in each application for the transaction. The logs from all the application contains the same "transaction_id". The log entry from each application will contain a seprate index(e.g, for app1 app1_index, for app2 app2_index,for app3*app3_index* etc.,)

the flow will be like app1->app2->app3->app2->app1.

I am looking for a splunk query to identify the time taken by app1(the time between 1st entry of app1_index and last entry of app1_index in the intial flow from app1->app2). The problem here is that more than one entry will be present for each application(for example in app1->app2 will have an entry for app1 and app2-> app1 will have an entry).

I was trying to use the "transaction" for this. But when using transaction along with the index field, I can get only the transaction with the same index start and end. But in my case there will be 2 flow with the same index.
Is there any way to achieve this requirement?. Can you please help here. Any pointer will be appreciated.

[to4kawa]

 <your search to get all the events from one or multiple indexes> 
| reverse
| streamstats range(_time) as duration by transaction_id index
| rename COMMENT as "untill here, duration of each flow by transaction id"
| rename COMMENT as "from here, duration of total flow by transaction id"
| stats sum(duration) as duration by transaciotn_id

Hi, @jyothishtj
The unit is seconds.

[woodcock]

Show us sample event data and a mockup of the desired final output FOR THOSE EVENTS.

[jyothishtj]

Hi Woodcock,

PFB the link with sample event details(created a mock up screenshot as the original one is sensitive data)

https://photos.app.goo.gl/DSwpLzyfdRYGbPLo6

I have the Transaction Id present in all events in the link, not able to upload image file, . I am just checking ,is there any option to get the time spend in each application for each flow for the transaction (for example looking for a separate entry for the time taken in app1 before app2 started processing and the time taken in app1 when app1 get the final result from app2 etc.,)
The expected result is something like .. each entry is the column name(excuse me for the formatting)

1.app1(Flow1) Start time 2.app1(Flow1) End time 3.app2(FLow1)Start Time 4.app2(flow1) End Time .............
5. app1(flow2)start time 6.app1(flow2) End TIme

[ololdach]

Hi,

I assume, that the transaction_id is unique and will identify all events that are connected with the transaction. With this assumption I suggest something like this:

<your search to get all the events from one or multiple indexes> | stats earliest(_time) as start_time, latest(_time) as end_time by transaction_id | eval time_used=end_time-start_time

This will get you a field time_used that holds the time between the first event and the last event in seconds. If you want to get the time over multiple transaction_id values, you would probably do something like this:

<your search to get all the events from one or multiple indexes>
| eval key=[<subsearch to produce transaction_id #1>| return $transaction_id].[<subsearch to produce transaction_id #2>| return $transaction_id]
| stats values(transaction_id) as transactions, earliest(_time) as start_time, latest(_time) as end_time by key
| eval time_used=end_time-start_time | fields - key

[jyothishtj]

Thanks Ololdash for the response. In my case, I have the Transaction Id present in all events(PFB the event sample(created based on original events) in the link, not able to upload image file), . I am just checking ,is there any option to get the time spend in each application for each flow for the transaction (for example looking for a separate entry for the time taken in app1 before app2 started processing and the time taken in app1 when app1 get the final result from app2 etc.,)

https://photos.app.goo.gl/DSwpLzyfdRYGbPLo6

example of result expecting is someting like

each number below is a column(excuse me for the formatting issue)
1.app1(Flow1) Start time 2.app1(Flow1) End time 3.app2(FLow1)Start Time 4.app2(flow1) End Time .............
5. app1(flow2)start time 6.app1(flow2) End TIme

[to4kawa]

Transaction ID is NOT Unique?

[jyothishtj]

Transaction ID is unique and same in all the events for a transaction. But, I am trying to find out the time each application spend for each flow in the transaction. Actually the flow is mentioned as transaction1 ,transaction2 etc., in the image.

https://photos.app.goo.gl/DSwpLzyfdRYGbPLo6

Hope I am not creating confusion.