topic Re: Summary Index skipping hosts in Knowledge Management

Summary Index skipping hosts

Branden — Fri, 29 Oct 2010 00:53:53 GMT

I have a summary index that collects stdout from a script that we run on all our hosts (SplunkLightForwarder). The search runs every 5 minutes looks like this:

sourcetype="datapath-adapter" | head 1 | multikv | sistats list(Select), list(Name), list(State), list(Errors) by Name,host

and

report=adapter

When I go to retrieve the data, it works fine:

index=si_hosts report="dpadapter"

EXCEPT it only contains information for three out of my twenty-four hosts. I check the orig_host field and, sure enough, only 3 values listed.

Why would the summary index choose only three hosts to index? There's nothing particular unique about those hosts, it just seems to random.

Is this a known issue by any chance?

Re: Summary Index skipping hosts

southeringtonp — Fri, 29 Oct 2010 01:01:05 GMT

Is there a possibility that one or more of your fields going into sistats has a null value? The stats family of commands will ignore events with null values by default.

Assuming this is what's going on, you can use the fillnull command to insert default values before calling sistats.

http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Fillnull

Re: Summary Index skipping hosts

Branden — Fri, 29 Oct 2010 01:31:12 GMT

I don't think that's the case. Certainly not over the past 24 hours, which is when I started indexing. Thanks though.

Re: Summary Index skipping hosts

araitz — Fri, 29 Oct 2010 03:18:48 GMT

How many results are you getting per run? More than 10k?

Re: Summary Index skipping hosts

Branden — Fri, 29 Oct 2010 21:17:51 GMT

Not even close.
I have a script that runs a command every 30 minutes. Splunk captures the stdout from that command and indexes it.
Even tho my saved search runs every 5 minutes, it'll probably capture an event once per 30 minutes per host. And it's just several lines of output.
I only have about 8 servers that run this script so it's no where close to 10k.