Knowledge Management

Summary Index skipping hosts

Branden
Builder

I have a summary index that collects stdout from a script that we run on all our hosts (SplunkLightForwarder). The search runs every 5 minutes looks like this:

sourcetype="datapath-adapter" | head 1 | multikv | sistats list(Select), list(Name), list(State), list(Errors) by Name,host

and

report=adapter

When I go to retrieve the data, it works fine:

index=si_hosts report="dpadapter"

EXCEPT it only contains information for three out of my twenty-four hosts. I check the orig_host field and, sure enough, only 3 values listed.

Why would the summary index choose only three hosts to index? There's nothing particular unique about those hosts, it just seems to random.

Is this a known issue by any chance?

Tags (1)
0 Karma

Branden
Builder

Not even close.
I have a script that runs a command every 30 minutes. Splunk captures the stdout from that command and indexes it.
Even tho my saved search runs every 5 minutes, it'll probably capture an event once per 30 minutes per host. And it's just several lines of output.
I only have about 8 servers that run this script so it's no where close to 10k.

0 Karma

araitz
Splunk Employee
Splunk Employee

How many results are you getting per run? More than 10k?

0 Karma

southeringtonp
Motivator

Is there a possibility that one or more of your fields going into sistats has a null value? The stats family of commands will ignore events with null values by default.

Assuming this is what's going on, you can use the fillnull command to insert default values before calling sistats.

http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Fillnull

0 Karma

Branden
Builder

I don't think that's the case. Certainly not over the past 24 hours, which is when I started indexing. Thanks though.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...