If there are multiple outputs.conf files (apps vs system), how do they merge, or which one takes precedence?

[jlemoine]

I've inherited an older Splunk instance (6.2.2) that appears to have multiple /local/outputs.conf files. While I'm familiar with the precedence inside of the outputs.conf file that I'm used to working with inside of /system/local/ (Global, Target, Single,) I don't know who wins the fight when there are multiple /local/outputs.conf files. Can anyone shed some light on the order of precedence for the following:

[splunk@superawesomeserver etc]$ locate outputs.conf
/data/splunkforwarder/etc/system/local/outputs.conf
/opt/splunkforwarder/etc/apps/uni_splunk_forwarders/local/outputs.conf
/opt/splunkforwarder/etc/system/local/outputs.conf

Obviously, there appears to be two $SPLUNK_HOME's, which is another issue altogether, but when it comes down to the apps vs system outputs.conf, how do they merge?

Thanks in advance!

[earlhelms]

btool is your friend - https://docs.splunk.com/Documentation/Splunk/6.5.2/Troubleshooting/Usebtooltotroubleshootconfigurati...
IMO, it is the best way to validate what values are set to.

[minx66]

So if there are multipsle input files under different app folders but there are some of the same log types in them, which apps will get preference?

[dmaislin_splunk · Accepted solution]

All covered in depth here: https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Wheretofindtheconfigurationfiles

Basically:
etc/system/default loads first at the bottom of the totem pole
etc/apps/appname/default
etc/apps/appname/local
etc/system/local is highest

All the config files with the same name are merged and same settings under same name stanzas are overwritten based on the precedence docs I listed above.

[jlemoine]

Will an etc/system/default/outputs.conf override an etc/apps/appname/local/outputs.conf?

[dmaislin_splunk]

No, you shouldn't be messing with anything in the system/defaults folder anyway. The app takes precedence.

[jlemoine]

Sweet and simple, thank you!