Summary Index skipping hosts

[Branden]

I have a summary index that collects stdout from a script that we run on all our hosts (SplunkLightForwarder). The search runs every 5 minutes looks like this:

sourcetype="datapath-adapter" | head 1 | multikv | sistats list(Select), list(Name), list(State), list(Errors) by Name,host

and

report=adapter

When I go to retrieve the data, it works fine:

index=si_hosts report="dpadapter"

EXCEPT it only contains information for three out of my twenty-four hosts. I check the orig_host field and, sure enough, only 3 values listed.

Why would the summary index choose only three hosts to index? There's nothing particular unique about those hosts, it just seems to random.

Is this a known issue by any chance?

[Branden]

Not even close.
I have a script that runs a command every 30 minutes. Splunk captures the stdout from that command and indexes it.
Even tho my saved search runs every 5 minutes, it'll probably capture an event once per 30 minutes per host. And it's just several lines of output.
I only have about 8 servers that run this script so it's no where close to 10k.

[araitz]

How many results are you getting per run? More than 10k?

[southeringtonp]

Is there a possibility that one or more of your fields going into sistats has a null value? The stats family of commands will ignore events with null values by default.

Assuming this is what's going on, you can use the fillnull command to insert default values before calling sistats.

http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Fillnull

[Branden]

I don't think that's the case. Certainly not over the past 24 hours, which is when I started indexing. Thanks though.