Summary Index Backfiller fill_summary_index.py broken? "No scheduled times for your time range"

[Olli1919]

Hi Fellow Splunkers,

After having upgraded to 6.4.1 yesterday, I had a go with fill_summary_index.py again, and noticed that am still getting the same error "No scheduled times for your time range", which I had been getting since a year or longer.

Being in dire need of it this time, I tried to backfill different searches, e.g. using sistats, using collect, basic and cron scheduling. To my surprise even the most basic search (basic scheduling 1 hour, using sistats and flagging "enable summary indexing" in the webgui search edit form), was getting the same error "No scheduled times...".

Taking a deeper look at it with @daubsi_2, we found that:

• manually searching the designated REST endpoint (saved/searches/{name}/scheduled_times) returned the needed list "scheduled_times"

    <s:list>
      <s:item>1465292100</s:item>
      <s:item>1465293000</s:item>
      ...
  

• Looking at the function output of getSavedSearchWithTimes in fill_summary_index.py, this returned the definition of the search in question, but without scheduled_times. It looks pretty much as if only search definition without scheduled_times was queried

• Taking a look a saved.py and entity.py, we could not make out where the needed REST URI .../scheduled_times would be prepended

• Changing a few lines to make fill_summary_index.py use the REST endpoint (saved/searches/{name}/scheduled_times) made the script run again as it used to. It found the scheduled_times just happily.

How is your mileage with fill_summary_index.py? Does it work as intended in 6.3+? As it is probably in heavy use out there, I would be a little surprised if it's really going to the wrong endpoint.

Olli

[daubsi_2 · Accepted solution]

For your convenience, here's a diff so you can apply the patch:

--

--- fill_summary_index.py       2016-05-13 00:05:14.000000000 +0200
+++ fill_summary_index_oko3.py  2016-06-10 15:17:18.000000000 +0200
@@ -9,6 +9,7 @@
 import splunk.auth
 import splunk.search
 import splunk.util
+import splunk.entity as entity
 import splunk.clilib.bundle_paths as bundle_paths

 app = None
@@ -158,12 +159,12 @@
       -timefield <string>   The field in summary index data that contains the
                             scheduled time of the saved search that generated
                             that data
-
+
   1 - For a full description of Splunk relative time strings, see:
       http://docs.splunk.com/Documentation/Splunk/6.4.1/User/ChangeTheTimeRangeOfYourSearch#Syntax_for_rel...
   '''
   sys.exit(0)
-
+
 def getBoolArg(opt, val):
   try:
     return splunk.util.normalizeBoolean(val, enableStrictMode=True)
@@ -349,7 +350,8 @@
     cur_st_list = []
     index = indexarg
     try:
-      ent = splunk.search.getSavedSearchWithTimes(ssname, et, lt, namespace=app, sessionKey=sk, owner=owner)
+      #ent = splunk.search.getSavedSearchWithTimes(ssname, et, lt, namespace=app, sessionKey=sk, owner=owner)
+      ent = entity.getEntity("/saved/searches", ssname, uri="/servicesNS/"+owner+"/"+app+"/saved/searches/"+ssname+"/scheduled_times", earliest_time=et, latest_time=lt, namespace=app, owner=owner, sessionKey=sk)
       if 'scheduled_times' in ent:
         for st in ent['scheduled_times']:
           cur_st_list.append((ssname, st))

[Olli1919]

Here is a quick hack to fix this - if anyone needs it, too. Beware of mistakes, use at you own risk, I am no pro at that.

First, check if you are getting the same error:
- Is your search configured correctly (enabled, is_scheduled, has summary_index action)?
- Check if the REST Endpoint returns the scheduled times correctly, when queried manually:

curl -k -u <user>:<pass> https://localhost:8089/servicesNS/<user>/<app>/saved/searches/<searchname>/scheduled_times --get -d earliest_time=-5d -d latest_time=now

This REST query should return a block with the scheduled_times you queried for (-5d until now, as set above), e.g.:

<s:key name="scheduled_times">
          <s:list>
            <s:item>1465548600</s:item>
            <s:item>1465549500</s:item>
            <s:item>1465550400</s:item>
            ....
          </s:list>
</s:key>

If it does not return this list of scheduled_times of jobs, there is something else wrong. This hack won't help you until you get schedules_times.

To integrate this hack, make the following changes to fill_summary_index.py:
Within the imports at the beginning add the following line, after line 11 "import splunk.util" would be a nice place:

import splunk.entity as entity

Comment out line 352 like this:

#ent = splunk.search.getSavedSearchWithTimes(ssname, et, lt, namespace=app, sessionKey=sk, owner=owner)

Insert this replacement statement in the following line 353:

ent = entity.getEntity("/saved/searches", ssname, uri="/servicesNS/"+owner+"/"+app+"/saved/searches/"+ssname+"/scheduled_times", earliest_time=et, latest_time=lt, namespace=app, owner=owner, sessionKey=sk)

What this does is that it leaves out the call to the wrapper function getSavedSearchWithTimes in saved.py and calls the getEntity function from entity.py directly, with an override to the correct REST Endpoint (in field "uri").

As said, use at own risk. If anyone has a nicer solution, feel free to post here.

Please upvote of you have the same problem and this helped you.