Reporting

accessing starting and ending time of a scheduled search inside the search

guilhem
Contributor

I would like to access to the starting and ending time of a scheduled search inside the search itself, if I want to run it afterwards with the right parameters.

If I put the | addinfo command inside the scheduled search it gives me the info_min_time and info_max_time but these contains the time when the search was ran, not the time the search was scheduled to run.

In fact I want to access this time (when run with the fill_summary_index.py script):

Started job 'XXXX_cHJvZF9zdW1tYXJ5X3JldHVybmluZ19zZWdtZW50X2xhc3RIb3Vy_at_1363205100_2ca7751a68b1bed6' for saved search 'prod_summary_returning_segment_lastHour', UTC = 1363205100 (Wed Mar 13 21:05:00 2013)

Thanks!


EDIT:

"For saved search 'saved_search_name'
Executing search to find existing data: 'search splunk_server=local index=my_index source="saved_search_name" | stats count by search_now'
  waiting for job sid = '1363940863.11827'  ... finished
Out of 10 scheduled times, 3 will be skipped because they already exist.

*** Spawning a total of 7 searches (max 8 concurrent) ***
Started job 'XXXXcHJvZF9zdW1tYXJ5X3JldHVybmluZ19zZWdtZW50X2xhc3RIb3Vy_at_1363259100_86ab6683906837c8' for saved search 'prod_summary_returning_segment_lastHour', UTC = 1363259100 (Thu Mar 14 12:05:00 2013)

This is the time (UTC = ...) that I want to access, even if I run the search in an unscheduled manner, the real scheduled time is still computed by the python script somehow. I want to know if I can access this value inside the search itself?

Thanks for the answers!

Guilhem

0 Karma
1 Solution

guilhem
Contributor

So after messing around a bit, I have found a clunky way to achieve what I wanted, may it helps other!

First I have created a macro, that takes the last search of the jobs list with the given search name, and return its earliest time (MY_MACRO):

MY_MACRO(search_name, "hours to subtract before earliest schedule time")
[| rest /services/search/jobs
| search label="$arg1$"
| head 1
| eval earliest=substr(earliestTime, 1, 19)
| eval earliest=strptime(earliest,"%Y-%m-%dT%H:%M:%S")
| eval earliest=earliest-$arg2$*3600
| eval earliest=round(earliest, 0)
| return earliest]

Note that you shouldn't return $earliest as in a macro it will not work (maybe interpreted as an argument cause of the $ sign, see here.

Then, take the search you want to backfill summary index, let's take index=MY_INDEX as an example (name it MY_SEARCH), and add the MY_MACRO value at the end, so the final search is:

index=MY_INDEX MY_MACRO(MY_SEARCH, "hours to subtract before earliest schedule time")

You can then use fill_summary_index.py with this search, the earliest time will be the scheduled time (the UTC+ ... time you can see in the console log), and not the time you run the search.

Thanks very much to Ayn, Martin and alacercogitatus for the help, I think I'm done with it.

View solution in original post

0 Karma

guilhem
Contributor

So after messing around a bit, I have found a clunky way to achieve what I wanted, may it helps other!

First I have created a macro, that takes the last search of the jobs list with the given search name, and return its earliest time (MY_MACRO):

MY_MACRO(search_name, "hours to subtract before earliest schedule time")
[| rest /services/search/jobs
| search label="$arg1$"
| head 1
| eval earliest=substr(earliestTime, 1, 19)
| eval earliest=strptime(earliest,"%Y-%m-%dT%H:%M:%S")
| eval earliest=earliest-$arg2$*3600
| eval earliest=round(earliest, 0)
| return earliest]

Note that you shouldn't return $earliest as in a macro it will not work (maybe interpreted as an argument cause of the $ sign, see here.

Then, take the search you want to backfill summary index, let's take index=MY_INDEX as an example (name it MY_SEARCH), and add the MY_MACRO value at the end, so the final search is:

index=MY_INDEX MY_MACRO(MY_SEARCH, "hours to subtract before earliest schedule time")

You can then use fill_summary_index.py with this search, the earliest time will be the scheduled time (the UTC+ ... time you can see in the console log), and not the time you run the search.

Thanks very much to Ayn, Martin and alacercogitatus for the help, I think I'm done with it.

0 Karma

Ayn
Legend

You could get this from the REST endpoint /saved/searches, see more here: http://docs.splunk.com/Documentation/Splunk/5.0.2/RESTAPI/RESTsearch#saved.2Fsearches.2F.7Bname.7D

Specifically the next_scheduled_time value should be what you want.

0 Karma

guilhem
Contributor

Yes I do, but I don't see all of my saved search inside. Only a very small portion of them, can't tell what is their common factor?

Same in the /services/scheduled/, I don't have any of my search here, just an empty view folder.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

You should have /services/saved/searches/ though.

0 Karma

guilhem
Contributor

Thank you very much. I have tested the rest keyword and it works very weel. I was able to (almost) solve the problem. I still have a parsing error but it may be due to the macro I use (following topic here:
http://splunk-base.splunk.com/answers/80765/debugging-a-search-ran-from-python-script)

Also it looks like the endpoint you give ( saved/searches) doesn't exists in my system. I only have inputs in the /services/search.

0 Karma

Ayn
Legend

Note that you can access it using the rest command in the search language.

0 Karma

guilhem
Contributor

I have never used this before (the REST API). Gonna need some time to test if I can make it work. Thanks for the answer.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Wouldn't this be what now() returns?

0 Karma

guilhem
Contributor

Yes, I see where is the confusion. But when you use the script fill_summary_index.py, in the console log, you can see (see edited post, not enough room here)

0 Karma

Ayn
Legend

I don't really get how you mean by scheduled here - if you run it manually on 3/1/2013 it's obviously not scheduled, so there's no way of accessing a scheduled time that you're not using anyway.

0 Karma

guilhem
Contributor

The now() description is:

now(): This function takes no arguments and returns the time that the search was started. The time is represented in Unix time or seconds since epoch.

The 'search was started' is a bit confusing. Let's say this search was scheduled to run the 01/01/2013 and I run it the 03/01/2013, what will now() return: 01/01/2013 or 03/01/2013?

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...