Splunk app ingesting excess data

[roberteves]

Hi all, somewhat new to Splunk, hopefully I'll describe the issue well...

I've setup a deployment app to forward logs from an (non-splunk) application for which there is no current app on Splunkbase. I configured the index and log file locatiosn to go to the index oempi. I configured a specific server class for these servers.

I also have a server class for Splunk App for *nix which has these servers as well as other Linux servers used for other purposes. That app is configured to send data to the os index.

For some reason I'm getting all the data from the Splunk App for *nix in the oempi index  as well as the os index.

Below are the settings for the two apps:

head Splunk_TA_nix/local/inputs.conf
# Copyright (C) 2020 Splunk Inc. All Rights Reserved.
[default]
index = os

[script://./bin/vmstat.sh]
interval = 60
sourcetype = vmstat
source = vmstat
disabled = 1

head OpenEMPI/local/inputs.conf
[default]
index = oempi

[monitor:///sysnet/openempi/openempi-3.5.7/logs]
whitelist=(\.log|\.out|\.txt)
disabled = 0

[monitor:///sysnet/openempi/openempi-3.5.7/openempi-entity-3.5.7/logs]
whitelist=(\.log)
disabled = 0

[hmallett · Accepted solution]

Where you have a [default] stanza in each inputs.conf, those stanzas are not scoped to the inputs.conf file. Instead all the inputs.conf files are aggregated, and only one of the [default] stanzas "wins". (Configuration file order precedence is documented here)

I would expect your inputs.confs to look like:

[script://./bin/vmstat.sh]
index = os
interval = 60
sourcetype = vmstat
source = vmstat
disabled = 1

and

[monitor:///sysnet/openempi/openempi-3.5.7/logs]
index = oempi
whitelist=(\.log|\.out|\.txt)
disabled = 0

[monitor:///sysnet/openempi/openempi-3.5.7/openempi-entity-3.5.7/logs]
index = oempi
whitelist=(\.log)
disabled = 0

[roberteves]

Sorry for the delay, I was thinking it might be something like that. Making those changes resolved the issue. Thanks!

[somesoni2]

Try to run btool on your forwarder to see what type of monitoring config is active.

$Splunk_home/bin/splunk btool inputs list --debug

[roberteves]

Thanks, I'm not really familiar with the output from that, but I do see this:

/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf              [monitor:///var/log]
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf              blacklist = (lastlog|anaconda\.syslog|audit\.log.*)
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf              disabled = 0
/opt/splunkforwarder/etc/system/local/inputs.conf                          host = <my host>
/opt/splunkforwarder/etc/apps/OpenEMPI/local/inputs.conf                   index = oempi
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf              whitelist = (\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out)

Why would it be combining the configuration from those two apps? I understand system defaults may be used where they're not specified in an app, but I expected these to be two separate apps and the configuration for either wouldn't affect the other