outputs.conf question about duplicates caused by referencing indexers by IP vs FQDN?

[calvinmcelroy]

I have suspicious that my outputs.conf configuration files are causing some unwanted data cloning in my forwarders. I am trying to make sense of some weird behavior I am observing, I am hoping someone can fact-check my assumptions for validity, or tell me what if I am not understanding this issue correctly. 

I have a UF on a syslog server. On the UF is a variety of apps, only a few of which possess a outputs.conf file. 

If I search for outputs.conf files, these are the 4 that I find:

./apps/SplunkUniversalForwarder/default/outputs.conf
./apps/comp_all_forwarder_outputs/local/outputs.conf
./apps/comp_all_outputs/local/outputs.conf
./system/default/outputs.conf

Based on the conf file hierarchy rules, I would expect that the two with ./local/outputs.conf would take priority over the other two with ./default/outputs.conf.

Taking a look at each file, one is specifying indexer peers by FQDN, and the other is specifying the peers as IP addresses. Since both files have the same priority, and they are not the same conf file, would this create a scenario where Splunk sends data to the indexer tier twice (once for each outputs.conf file) cloning the data into the same indexing tier?

/opt/splunkforwarder/etc/apps/comp_all_outputs/local/outputs.conf

[tcpout]
defaultGroup = primary_indexers
[tcpout:primary_indexers]
server = spkidx01.comp.com:9997, spkidx02.comp.com:9997, spkidx03.comp.com:9997
autoLB = true

/opt/splunkforwarder/etc/apps/comp_all_forwarder_outputs/local/outputs.conf

[tcpout]
defaultGroup = primary_indexers
[tcpout:primary_indexers]
server = 10.15.4.229:9997, 10.15.5.85:9997, 10.15.4.250:9997

The IP Addresses listed resolve to the FQDNs in the previous outputs.conf file. I would expect Splunk or maybe the OS would call these two separate outputs.conf files

TIA!

[calvinmcelroy]

Thank you both, this was helpful. 

I did find at least one issue causing double events, but the problem doesn't seem to be as wide spread as I thought. Now I know how to verify this.

[somesoni2]

The configuration file precedence is described in following Splunk documentation:

https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Wheretofindtheconfigurationfiles 

For outputs.conf file, the precedence would be this:

$SPLUNK_HOME/etc/system/local/*
$SPLUNK_HOME/etc/apps/A/local/* ... $SPLUNK_HOME/etc/apps/z/local/*
$SPLUNK_HOME/etc/apps/A/default/* ... $SPLUNK_HOME/etc/apps/z/default/*
$SPLUNK_HOME/etc/system/default/*

 Based on this, your /opt/splunkforwarder/etc/apps/comp_all_forwarder_outputs/local/outputs.conf version would take precedense.

As Richard mentioned, please run btool command (on your UF back end) to see what configurations are taking place.

Example btool command:

/opt/splunkforwarder/bin/splunk btool outputs list --debug | grep -v system/default

[richgalloway · Accepted solution]

Use btool to see exactly what Splunk is using for its outputs config.

splunk btool --debug outputs list

If it doesn't shed light on the problem then post the output here for us to look at.