Splunk Search

Response Time from Splunk Logs using Transaction starts with and within an single event

sen8sen
Engager

I am trying to calculate the duration/timetaken between 2 strings in an event using transaction starts with and endswith and it is not giving the expected and the format is different, I wanted a simple format with HH:MM:SS

Example of event returned :

[main] 19:28:06,435[batchLogId=, clientCode=] INFO org.ets.ereg.batch.common.listener.BaseJobListener.beforeJob(BaseJobListener.java:89) - Start : Before Job *************
...
...
...
...

[main] 20:05:07,411[batchLogId=15304309, clientCode=] INFO org.ets.ereg.batch.common.listener.BaseJobListener.afterJob(BaseJobListener.java:163) - End : After Job ***********

My Requirement :

I wanted to calculate my time taken or duration based on the timings in front of these . Between Start and End After Job.

My Query is

index="ereg-prod" source="jobs.*log" | transaction startswith="Start : Before Job" endswith="End : After Job" | rex field=source "/*/logs/job-(?\S+).log"

I tried time chart and _time what is the exact way to get it. Any suggestions would be helpful.

Labels (1)
Tags (1)
0 Karma
1 Solution

PavelP
Motivator

Hello @sen8sen,

please consult https://docs.splunk.com/Documentation/Splunk/8.0.3/SearchReference/Transaction

the transaction command adds two fields to the raw events, duration and eventcount. The values in the duration field show the difference between the timestamps for the first and last events in the transaction.

So basically the transaction command do it for you already and you can use this field directly:

index="ereg-prod" source="jobs.log" | transaction startswith="Start : Before Job" endswith="End : After Job" | rex field=source "//logs/job-(?\S+).log" |table _raw duration

View solution in original post

0 Karma

PavelP
Motivator

Hello @sen8sen,

please consult https://docs.splunk.com/Documentation/Splunk/8.0.3/SearchReference/Transaction

the transaction command adds two fields to the raw events, duration and eventcount. The values in the duration field show the difference between the timestamps for the first and last events in the transaction.

So basically the transaction command do it for you already and you can use this field directly:

index="ereg-prod" source="jobs.log" | transaction startswith="Start : Before Job" endswith="End : After Job" | rex field=source "//logs/job-(?\S+).log" |table _raw duration
0 Karma

sen8sen
Engager

Hi ,

Thanks , it does work . I too have another field which comes within the event and I am trying to calculate the duration by that field. My query is below

index="ereg-prod" | transaction time maxspan=1h startswith="Start : Before Job" endswith="End : After Job" | rex field=source "*/logs/job-(?\S+).log" | table duration, BatchJobName

basically I am trying to calculate the duration in HH/MM/SS for the batchjobs and there are some batch jobs which runs hourly and some every 15 min within a hour and how can arrive at the average running duration of these jobs. If I try my search | | chart values(duration) over _time by BatchJobName will it work. Any suggestions would be helpful.

0 Karma

PavelP
Motivator

Hi @sen8sen,

try to build your transactions based on a uniq common field (BatchJobName):

  index="ereg-prod" | transaction BatchJobName time maxspan=1h startswith="Start : Before Job" endswith="End : After Job" | rex field=source "*/logs/job-(?\S+).log" | table duration, BatchJobName

BTW: what you try to achive/extract with your rex command? Your rex command missing a named caputring group, see https://docs.splunk.com/Documentation/Splunk/8.0.3/SearchReference/Rex

0 Karma

sen8sen
Engager

hi,

here is my query

index="ereg-prod" source="/export/Apps/ereg-prod/batch/archive/.*log" | transaction time startswith="Start : Before Job" endswith="End : After Job" | rex field=source "ereg-prod/batch/archive/logs/job-(?\S+).log" | rex field=source "ereg-prod/batch/archive/logs/job-(?\S+).log" | chart avg(duration) as "Duration in Seconds" by BatchJobName

It is returning the results but the unable to format for HH:MM:SS which would be useful. Would it be possible to achieve that

0 Karma

PavelP
Motivator
0 Karma

sen8sen
Engager

Hi ,

thanks a lot -:)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...