Summary index missing random events

[phemmer]

I'm trying to set up some summary indexes, but the summary index is missing random events. The scheduled search job is running, but the data is just not in the index.

For example:

Notice that the event for 11:09 is missing. Yet when I look in the job activity, the job fired off:

Note that the job at 11:10 fills in the summary index data for 11:09. Below is from the job inspection output. The times fit right where the 11:09 event is supposed to be.

This isn't a case of the job running too long. As you can see in the job list, it completes in less than a second.
The query that is running is very simple:

host=iad1bf5* program=ltm request request="GET /" | stats dc(client_ip)

I have a copy of the job inspection output, as well as the search.log, and can provide any info needed from there.
This is with Splunk 6.3.2

[srujan9292]

You can schedule the saved search to run 15 minutes earlier.
For example, if the time is 8:30.000 AM, schedule the saved search to run from -30min to -15min.
in this case, the events will not be missed out.

[woodcock]

What you are probably missing is that the event that was not included was not present in Splunk at the time the SI-populating search ran. Run this search:

 host=iad1bf5* program=ltm request request="GET /" | eval lag=tostring((_indextime - _time), "duration")

If the lag of any event is larger than the width of your timepicker range used by your SI-populating search, then it will be missed. This is why I generally suggest people use Accelerated Data Models + tstats instead of Summary Index. It has all of the advantages but none of the weaknesses, the largest of which is mishandling (missing) of late-arriving events.

[phemmer · Accepted solution]

Finally figured this out. We have multiple search heads, and the job was running on a random one, and not replicating the data in any way.

Fixed it by configuring the search heads to forward their data to the indexers: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata