Splunk precedence issue

[rameshlpatel]

Hi,

I have outputs.conf file under etc/system/local folder with following conf.

[tcpout-server://10.248.180.196:9997]
[tcpout:default-autolb-group]
server = 10.248.180.196:9997

In addition, I deployed app with outputs.conf (with following conf) from deployment server to etc/app dir.

[tcpout-server://alpputl018:9997]

[tcpout:default-autolb-group]
server = alpputl018:9997

Ideally app folder outputs.conf should override system/local outputs.conf. means ideally logs should be forward to alpputl018, but in my scenario its still pointing to old indexer i.e. 10.248.180.196.

In addition. forwarder logs are forwarding to new indexer but not application log.

This issue is really strength to me and not working as per splunk precedence theory.

Please help me out to understand this issue.

[sowings]

$SPLUNK_HOME/etc/system/local takes precedence over any app config (whether local OR default) in $SPLUNK_HOME/etc/apps. If you are using the deployment server, you are best served by not placing any local (site-specific) configs in $SPLUNK_HOME/etc/system/local, since these cannot be overridden by apps sent by the deployment server.

Because of the precedence rules set out in $SPLUNK_HOME/etc/system/default/conf.conf, the behavior that [~rameshlpatel] is observing is correct, even if it's not what's intended.

[sowings]

btw, "cd $SPLUNK_HOME/etc/system/default ; grep conf conf.conf | grep -v confdb". The apps provided from a cluster master (placed in the slave-apps folder on the clustered indexer) override even system/local!

[rameshlpatel]

Thanks for clearing my doubts.

[kheli]

indexing is global context so config in /etc/system/local will take precendence.

You can also use btool command to find all outputs.conf value in a splunk instance.

http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurati...

If you cannot see application logs are being indexed, make sure the index for the application log is created in the indexer and data input has been configured properly in forwarder.

[rameshlpatel]

index has been created in new indexer and monitoring path is also properly configured in forwarders.

[rameshlpatel]

I ran btool and its showing old one. Now problem is how I override this configuration with new from deployment server ?