Getting Data In

Forward data to Indexer cluster

nravichandran
Communicator

I am in the middle of understanding an already built environment and trying to figure out how a splunk universal forward is configured. A brief about the environment , 3 search heads, 2 indexers, 1 deployment server and license master, and master node.

In one of the forwarder configuration is configured as deployment client. But i don't find the outputs.conf either in apps or in system folders. But the forwarder is sending data to the indexers. Is there a way to find out how it sends by CLI or Any other conf file?

Thank you in advance.

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Hi nravichandran!

Try running the 'list forward-server' command from the forwarder itself when looking to confirm if, and to whom, the forwarder is sending:

splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk list forward-server
Your session is invalid.  Please login.
Splunk username: admin
Password: 
Active forwards:
    10.10.31.216:9997
Configured but inactive forwards:
    None

Also, btool is a must! Do yourself a huge favor and explore it as part of getting to know this enviro:

https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Usebtooltotroubleshootconfigurati...

Splunk forwarders sending data must have an outputs.conf. You can use btool to get splunk to tell you, what configs, are coming from where:

Here's an example

splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk btool outputs list --debug 
/opt/splunkforwarder/etc/system/default/outputs.conf                        [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf                        priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf                        type = udp
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           defaultGroup = n00b-splkidx-02
/opt/splunkforwarder/etc/system/default/outputs.conf                        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout-server://10.10.31.216:9997]
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout:n00b-splkidx-02]
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           server = 10.10.31.216:9997

For windows CLI help see: https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/AbouttheCLI

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

Hi nravichandran!

Try running the 'list forward-server' command from the forwarder itself when looking to confirm if, and to whom, the forwarder is sending:

splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk list forward-server
Your session is invalid.  Please login.
Splunk username: admin
Password: 
Active forwards:
    10.10.31.216:9997
Configured but inactive forwards:
    None

Also, btool is a must! Do yourself a huge favor and explore it as part of getting to know this enviro:

https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Usebtooltotroubleshootconfigurati...

Splunk forwarders sending data must have an outputs.conf. You can use btool to get splunk to tell you, what configs, are coming from where:

Here's an example

splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk btool outputs list --debug 
/opt/splunkforwarder/etc/system/default/outputs.conf                        [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf                        priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf                        type = udp
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           defaultGroup = n00b-splkidx-02
/opt/splunkforwarder/etc/system/default/outputs.conf                        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout-server://10.10.31.216:9997]
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           [tcpout:n00b-splkidx-02]
/opt/splunkforwarder/etc/apps/n00blab_ufw_base/local/outputs.conf           server = 10.10.31.216:9997

For windows CLI help see: https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/AbouttheCLI

- MattyMo

nravichandran
Communicator

Thank you very much for a detailed reply. I was able to figure out that the outputs.conf are under /apps//local folder. I was exicited to run the btool but it does not work for me. I have a root account and run ./splunk cmd btool outpus list --debug. It does not give results nor throw any error.

0 Karma

mattymo
Splunk Employee
Splunk Employee

you need to be under /opt/splunkforwarder/bin if it is a universal forwarder.

also watch the typos!

./splunk cmd btool outputs list --debug

- MattyMo
0 Karma

nravichandran
Communicator

Thank you very much!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...