Getting Data In

outputs.conf question about duplicates caused by referencing indexers by IP vs FQDN?

calvinmcelroy
Path Finder

I have suspicious that my outputs.conf configuration files are causing some unwanted data cloning in my forwarders. I am trying to make sense of some weird behavior I am observing, I am hoping someone can fact-check my assumptions for validity, or tell me what if I am not understanding this issue correctly. 

I have a UF on a syslog server. On the UF is a variety of apps, only a few of which possess a outputs.conf file. 

If I search for outputs.conf files, these are the 4 that I find:

 

 

./apps/SplunkUniversalForwarder/default/outputs.conf
./apps/comp_all_forwarder_outputs/local/outputs.conf
./apps/comp_all_outputs/local/outputs.conf
./system/default/outputs.conf

 

 

Based on the conf file hierarchy rules, I would expect that the two with ./local/outputs.conf would take priority over the other two with ./default/outputs.conf.

Taking a look at each file, one is specifying indexer peers by FQDN, and the other is specifying the peers as IP addresses. Since both files have the same priority, and they are not the same conf file, would this create a scenario where Splunk sends data to the indexer tier twice (once for each outputs.conf file) cloning the data into the same indexing tier?

/opt/splunkforwarder/etc/apps/comp_all_outputs/local/outputs.conf

 

 

[tcpout]
defaultGroup = primary_indexers
[tcpout:primary_indexers]
server = spkidx01.comp.com:9997, spkidx02.comp.com:9997, spkidx03.comp.com:9997
autoLB = true

 

 

 

/opt/splunkforwarder/etc/apps/comp_all_forwarder_outputs/local/outputs.conf

 

 

[tcpout]
defaultGroup = primary_indexers
[tcpout:primary_indexers]
server = 10.15.4.229:9997, 10.15.5.85:9997, 10.15.4.250:9997

 

 

The IP Addresses listed resolve to the FQDNs in the previous outputs.conf file. I would expect Splunk or maybe the OS would call these two separate outputs.conf files

 

TIA!

 

 

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Use btool to see exactly what Splunk is using for its outputs config.

splunk btool --debug outputs list

If it doesn't shed light on the problem then post the output here for us to look at.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

calvinmcelroy
Path Finder

Thank you both, this was helpful. 

I did find at least one issue causing double events, but the problem doesn't seem to be as wide spread as I thought. Now I know how to verify this.

0 Karma

somesoni2
Revered Legend

The configuration file precedence is described in following Splunk documentation:

https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Wheretofindtheconfigurationfiles 

For outputs.conf file, the precedence would be this:

$SPLUNK_HOME/etc/system/local/*
$SPLUNK_HOME/etc/apps/A/local/* ... $SPLUNK_HOME/etc/apps/z/local/*
$SPLUNK_HOME/etc/apps/A/default/* ... $SPLUNK_HOME/etc/apps/z/default/*
$SPLUNK_HOME/etc/system/default/*

 Based on this, your /opt/splunkforwarder/etc/apps/comp_all_forwarder_outputs/local/outputs.conf version would take precedense.

As Richard mentioned, please run btool command (on your UF back end) to see what configurations are taking place.

Example btool command:

/opt/splunkforwarder/bin/splunk btool outputs list --debug | grep -v system/default

richgalloway
SplunkTrust
SplunkTrust

Use btool to see exactly what Splunk is using for its outputs config.

splunk btool --debug outputs list

If it doesn't shed light on the problem then post the output here for us to look at.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...