Forward data to Indexer cluster

[nravichandran]

I am in the middle of understanding an already built environment and trying to figure out how a splunk universal forward is configured. A brief about the environment , 3 search heads, 2 indexers, 1 deployment server and license master, and master node.

In one of the forwarder configuration is configured as deployment client. But i don't find the outputs.conf either in apps or in system folders. But the forwarder is sending data to the indexers. Is there a way to find out how it sends by CLI or Any other conf file?

Thank you in advance.

[mattymo · Accepted solution]

Hi nravichandran!

Try running the 'list forward-server' command from the forwarder itself when looking to confirm if, and to whom, the forwarder is sending:

splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk list forward-server
Your session is invalid.  Please login.
Splunk username: admin
Password: 
Active forwards:
    10.10.31.216:9997
Configured but inactive forwards:
    None

Also, btool is a must! Do yourself a huge favor and explore it as part of getting to know this enviro:

https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Usebtooltotroubleshootconfigurati...

Splunk forwarders sending data must have an outputs.conf. You can use btool to get splunk to tell you, what configs, are coming from where:

Here's an example

splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk btool outputs list --debug 
/opt/splunkforwarder/etc/system/default/outputs.conf                        [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf                        priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf                        type = udp
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           defaultGroup = n00b-splkidx-02
/opt/splunkforwarder/etc/system/default/outputs.conf                        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout-server://10.10.31.216:9997]
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout:n00b-splkidx-02]
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           server = 10.10.31.216:9997

For windows CLI help see: https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/AbouttheCLI

[nravichandran]

Thank you very much for a detailed reply. I was able to figure out that the outputs.conf are under /apps//local folder. I was exicited to run the btool but it does not work for me. I have a root account and run ./splunk cmd btool outpus list --debug. It does not give results nor throw any error.

[mattymo]

you need to be under /opt/splunkforwarder/bin if it is a universal forwarder.

also watch the typos!

./splunk cmd btool outputs list --debug

[nravichandran]

Thank you very much!