I have a summary index that collects stdout from a script that we run on all our hosts (SplunkLightForwarder). The search runs every 5 minutes looks like this:
sourcetype="datapath-adapter" | head 1 | multikv | sistats list(Select), list(Name), list(State), list(Errors) by Name,host
When I go to retrieve the data, it works fine:
EXCEPT it only contains information for three out of my twenty-four hosts. I check the orig_host field and, sure enough, only 3 values listed.
Why would the summary index choose only three hosts to index? There's nothing particular unique about those hosts, it just seems to random.
Is this a known issue by any chance?
Not even close.
I have a script that runs a command every 30 minutes. Splunk captures the stdout from that command and indexes it.
Even tho my saved search runs every 5 minutes, it'll probably capture an event once per 30 minutes per host. And it's just several lines of output.
I only have about 8 servers that run this script so it's no where close to 10k.
Is there a possibility that one or more of your fields going into
sistats has a null value? The
stats family of commands will ignore events with null values by default.
Assuming this is what's going on, you can use the
fillnull command to insert default values before calling