Splunk Search

search start time

guilhem
Contributor

HI,

I would like to know if it's possible in the earliest / latest fields of a search to have something like:

index=myindex earliest="the time this search has started"-X seconds latest="the time this search has started" -Y seconds

Is it possible?

The aim is to run a saved search that populates a summary index, but I need to backfill this index with a search that contains earliest=-20h latest=-10h, but running the search as it is with the fill_summary_index.py command line returns no results because events I need to backfill occurs a long time before -20h (I need to backfill 4 month of datas).

I think I can have an eval statement that would compute what I want, and then just use where, but it would be very inefficient time wise.

Thanks,


EDIT

Thanks to the answers, I found that I can use earliest= [some search | return result] to populate the earliest field to look for data when I want. Problem is that I can't find a way to say:

earliest=["search that returns the starting scheduled time of this saved search"]

The keyword 'now' returns the starting time of the search when put inside the earliest field, but it's not what I want, I would like to have the starting SCHEDULED time, not the actual time I run the search.

I don't know if it's the right way to do it, and if there is another way, I would gladly try it.

0 Karma
1 Solution

guilhem
Contributor

Thank you very much martin for the help. I finally found the rest of the solution from here, use | addinfo, and info_min_time to retrieve the starting time of the search. So the final answer is (if you want earliest to start 20 hours before the scheduled time of the search eg):

index=_internal earliest=[ search index=_internal | head 1| addinfo | eval test=info_min_time-20*3600 | return $test]


EDIT

simplified, optimized, cleaned version:

index=_internal earliest=[ stats count | addinfo | eval test=relative_time(info_min_time, "-20h") | return $test]

View solution in original post

0 Karma

guilhem
Contributor

Thank you very much martin for the help. I finally found the rest of the solution from here, use | addinfo, and info_min_time to retrieve the starting time of the search. So the final answer is (if you want earliest to start 20 hours before the scheduled time of the search eg):

index=_internal earliest=[ search index=_internal | head 1| addinfo | eval test=info_min_time-20*3600 | return $test]


EDIT

simplified, optimized, cleaned version:

index=_internal earliest=[ stats count | addinfo | eval test=relative_time(info_min_time, "-20h") | return $test]

0 Karma

guilhem
Contributor

Thank you very much, I have learned a lots of very valuable things on splunk today. It's been a loooong time since I want to find a way to have something evalued whithout pulling datas:

  • doesn't work, so I used to play with index=X | head 1 so it was really fast. Yours (using stats) is even better.

Never used relative_time before. It's really clean and less "hacky" than what I did. I update the answer

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Since you're not actually using any data from _internal you could replace that with a call to stats or something else that doesn't cause any data to be loaded:

index=_internal earliest=[ stats count | addinfo | eval test=relative_time(info_min_time, "-20h") | return $test]

Additionally, you can let Splunk's relative time syntax do the time fiddling for you in case it gets more complicated.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If you can express your time fields using eval you can do a subsearch for each:

index=myindex earliest=[some search | eval earliest=something | return $earliest] latest=[some search | eval latest=something | return $latest] | ...

martin_mueller
SplunkTrust
SplunkTrust

Isn't the time the search was run equal to now from the search's point of view?

0 Karma

guilhem
Contributor

nice! Another tip I didn't know.

I am still trying to find a way to get the search scheduled time start, but I didn't find it yet. Does scheduled search have a special field containing their scheduled time?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Your first attempt can be made to work like this:

index=_internal earliest=[ stats count | eval test="-24h" | return $test]

The dollar sign changes the behaviour of return, returning only the value itself instead of key=value as usual.

0 Karma

guilhem
Contributor

I just need to figure out how to get the time the search was scheduled, instead of the "now" time.

0 Karma

guilhem
Contributor

Peoblem solved, in fact result contained "test=-24h" and not only "-24h" as I was expected.

The correct search is then:

index=_internal [search index=_internal | head 1 | eval earliest="-24h" | return earliest]

which works wonderfully.

Many thanks for the help.

0 Karma

guilhem
Contributor

I have tried it but I couldn't make it works:

index=_internal earliest=[ search index=_internal | head 1 | return "-24h"]
, also tried:

index=_internal earliest=[ search index=_internal | head 1 | eval test="-24h" | return test]

but it always returns:

Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the left hand side.

0 Karma

guilhem
Contributor

THAT is awesome, didn't know you could run subsearches after an '=' !!!

Is it possible to run a subsearch like this: 'eval=[some subsearch]' ? It looks increadibly powerfull and will solve many performance problem that I have. Gonna test it right away.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...