Can someone help me understand how my current outputs.conf settings work?

[splunkbacon]

A splunk engineer told us to deploy an app with the deployment server (to universal forwarders) that contained the outputs.conf file. the problem is that even with this app deployed running btool still shows that the config is being taken from system/local/outputs.conf --which is what I expected based on the documents from Splunk....maybe he was confused....

Anyway the real problem is I don't understand the correlation between a few values in outputs.conf

[target-broker:deploymentServer]
targetUri = clustermaster:8089

[tcpout]
defaultGroup=my_indexers

[tcpout: my_indexers]
server=1.1.1.1:9997,1.1.1.2:9997

[tcpout-server://1.1.1.1:9997]

[tcpout: my_LB_indexers]
autoLBFrequency=30
server=1.1.1.1:9997,1.1.1.2:9997,1.1.1.3:9997
useACK=true

After reading the outputs.conf documentation on Splunk I'm confused about a few things:

1) What is the result of the above config? Does the default group attribute mean that it will never send to the my_LB_indexers group? Or does the universal forwarder always send to all output groups? Since duplicate values exist would it still only send to each one once?

2) is the single server stanza even needed? I saw one splunk document that said it was optional and another that said it's always needed.

3) If the config makes it only send to my_indexers, does it ever fail over to another group if the my_indexers group becomes unavailable?

[pradeepkumarg]

Data is routed only to defaultGroup unless you specify in your inputs using _TCP_ROUTING = my_LB_indexers

[splunkbacon]

The weird thing is that the above config file exists in system/local/default....so it should be THE config...yet we see traffic on 9997 to the third server in my_LB_indexers. I looked at inputs.conf and I didn't see that option...let me check a few other places.

[woodcock]

No, it doesn't exists there; that's a typo. If you mean $SPLUNK_HOME/etc/system/local/ then yes: that is THE setting.

[woodcock]

This part points you to a Deployment Server:

[target-broker:deploymentServer]
targetUri = clustermaster:8089

This part defines the destination Indexers:

 [tcpout]
 defaultGroup=my_indexers

 [tcpout: my_indexers]
 server=1.1.1.1:9997,1.1.1.2:9997

This part is an empty header that tells me that you used the CLI to set your indexers. It does nothing and can/should be deleted.

[tcpout-server://1.1.1.1:9997]

This part is probably a staged upgrade setting that will take effect if you change defaultGroup=my_indexers todefaultGroup=my_LB_indexers, WHICH MIGHT BE HAPPENING IN ANOTHER outputs.confFILE SOMEWHERE ELSE!!!

[tcpout: my_LB_indexers]
 autoLBFrequency=30
 server=1.1.1.1:9997,1.1.1.2:9997,1.1.1.3:9997
 useACK=true

To see what your settings REALLY are run this:

splunk btool outputs list --debug

[xpac]

Is it only me, or is the [target-broker:deploymentServer] part in the wrong .conf file?
Shouldn't that be in deploymentclient.conf?

[splunkbacon]

yeah i think so. I just copy/pasted this from the file the engineer looked at.

[xpac]

It should be either moved or you should check if you have another setting like this in the right file, somewhere. 🙂

[woodcock]

You are right; that should be in deploymentclienty.conf!

[splunkbacon]

Thank you. Those are the things that i did (btools) and conclusions I came to as well. Does the tcpout: my_LB_indexers not do anything on it's own? btools shows defualtgroup is still set to my_indexers. I'm just kind of confused on whether it will send to every group in outsputs.conf, or if the defaultGroup didn't exist what would it do? not send anywhere? or send to all? I thought I saw something in splunk docs that said if multiple target groups exists that it would send to all of them.

[woodcock]

If you never see it tracked back to the [tcpout] stanza from another file, then it is completely inert.