Splunk Search

Fastest way to get log file duration?

jameshgibson
Path Finder

We have a batch runner that outputs a log file for each run. We would like to be able to get the duration of each log file so we can see how the batch times change when we make modifications.

Currently I am using:

source=/some/source* | transaction maxevents=-1 maxpause=-1 maxspan=1d keepevicted=t source | table duration, source

Unfortunatly because the log files are very large this is incredibly slow and even causes splunk to run out of memory. Is there a quicker way?

NOTE: The files do not have a consistant ending line to match on but all start with the same statement.

Tags (3)
1 Solution

kristian_kolb
Ultra Champion

Maybe I'm not seeing the whole picture, but what about

... | stats min(_time) AS start max(_time) AS stop by source| eval dur = stop - start | ...

building transactions require some memory, as you've noted..

/Kristian

View solution in original post

kristian_kolb
Ultra Champion

Maybe I'm not seeing the whole picture, but what about

... | stats min(_time) AS start max(_time) AS stop by source| eval dur = stop - start | ...

building transactions require some memory, as you've noted..

/Kristian

jameshgibson
Path Finder

thanks, thats what I was looking for 🙂

0 Karma

jameshgibson
Path Finder

The report as is doesn't run fully so I can't create a summary. Also the table ends up having multiple durations for source which doesn't make sense so I think the transaction command is either broken or I am not using it properly.

0 Karma

MuS
Legend

have you thought about summary indexing your search?

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...

Index This | How many sevens are there between 1 and 100?

August 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...