Is there a way in Splunk to compare and find delta of the duration from the previous runs?

[maniishpawar]

Hi All,

We are generating a log that records in and out timestamp in epoch for a specific set of transactions and we have been doing this for a while. In order to test this api, we run our load test at specific times for 1 hour and that generate the logs with transaction ids, keywords and the in and out timestamps etc. see the sample below.

We are querying this data and calculating duration.  Is there a way in splunk to compare and find delta of the duration from the previous runs.

Every run has a specific timestamps and we were adding it in the SPL itself like earliest="08/23/2022:20:45:00" latest="08/23/2022:21:55:00"

=============================================================

sample log

2022/08/23 21:54:38,918 INFO [XXXX.CPU_LITE @67166e0a] [LoggerMessageProcessor ] [ ] [ ] [] - End Workflow: flow1 | LogID: 104 |{
"Trans-Id": "cf18655a-5d1a-4867-b500-c4ba5bee9333",
"AppId": "somepapi"
} | OutTimestamp : 1661306078918
2022/08/23 21:54:37,819 INFO [XXXX.CPU_INTENSIVE @2c86def1] [LoggerMessageProcessor ] [ ] [ ] [] - Start Workflow: flow1 | LogID: 104 |{
"Trans-Id": "cf18655a-5d1a-4867-b500-c4ba5bee9333",
"AppId":"somepapi"
} | InTimestamp : 1661306077819

[ITWhisperer]

How do you identify from the previous log events, which is the corresponding previous run, e.g. would "Start Workflow: flow1 | LogID: 104" in this run have a corresponding "Start Workflow: flow1 | LogID: 104" in the previous run?

[maniishpawar]

I am not sure if I follow your question. But yes it will have corresponding start and end flow but transid are different. Events are in a pair of start workflow and end workflow, there is a common transid between these two events, which is auto generated. 

What I am trying to find out is below. 

In the month of July lets say there were 1 million event pairs ( so total count 2 mil)  and these transactions have average of 1second response time or duration. 

I want to compare this with the month of Aug , where we currently have 900,000 event pairs. I want to find how much delta/deviation we are seeing as compared to July in a chart.

[ITWhisperer]

So, for each pair, do you already know how to calculate the duration?

Are you just after determining the average for each month?

| bin _time span=mon
| stats avg(duration) as avg_duration by _time