Splunk Search

Summary index missing random events

phemmer
Path Finder

I'm trying to set up some summary indexes, but the summary index is missing random events. The scheduled search job is running, but the data is just not in the index.

For example:
missing events
Notice that the event for 11:09 is missing. Yet when I look in the job activity, the job fired off:
job activity
Note that the job at 11:10 fills in the summary index data for 11:09. Below is from the job inspection output. The times fit right where the 11:09 event is supposed to be.

This isn't a case of the job running too long. As you can see in the job list, it completes in less than a second.
The query that is running is very simple:

host=iad1bf5* program=ltm request request="GET /" | stats dc(client_ip)

I have a copy of the job inspection output, as well as the search.log, and can provide any info needed from there.
This is with Splunk 6.3.2

Tags (1)
0 Karma
1 Solution

phemmer
Path Finder

Finally figured this out. We have multiple search heads, and the job was running on a random one, and not replicating the data in any way.

Fixed it by configuring the search heads to forward their data to the indexers: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

View solution in original post

0 Karma

srujan9292
Explorer

You can schedule the saved search to run 15 minutes earlier.
For example, if the time is 8:30.000 AM, schedule the saved search to run from -30min to -15min.
in this case, the events will not be missed out.

0 Karma

woodcock
Esteemed Legend

What you are probably missing is that the event that was not included was not present in Splunk at the time the SI-populating search ran. Run this search:

 host=iad1bf5* program=ltm request request="GET /" | eval lag=tostring((_indextime - _time), "duration")

If the lag of any event is larger than the width of your timepicker range used by your SI-populating search, then it will be missed. This is why I generally suggest people use Accelerated Data Models + tstats instead of Summary Index. It has all of the advantages but none of the weaknesses, the largest of which is mishandling (missing) of late-arriving events.

0 Karma

phemmer
Path Finder

Finally figured this out. We have multiple search heads, and the job was running on a random one, and not replicating the data in any way.

Fixed it by configuring the search heads to forward their data to the indexers: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...