Way to monitor splunkforwarder

[virginiatech199]

How would I monitor splunkforwarder traffic output to indexers from the splunkforwarder? Have the target indexer listed in outputs.conf; url and port.

Monitor meaning is there traffic flowing to the indexers?

This solution does not involve logging into Splunk dashboard -- must be performed from the splunkforwarder.

Thanks for suggestions!

🙂

[virginiatech199]

Thanks for suggestions. Have to monitor splunk state FROM the splunkforwarder. Tried a 5 second tcpdump to target indexers but sometimes it sees nothing, sometimes sees a lot of traffic.....will try the splunkd.log suggestion.

[sandeepmakkena]

If it is a universal forwarder, the monitoring would be whether it is forwarding the data to a specific index or not. I best approach would create an alert.
Something like this

     | metasearch index=_internal 
     | eval host=upper(host) 
     | stats count by host
     | append [ | inputlookup Perimeter.csv | eval count=0 | eval host=upper(host) | fields host count] 
     | stats sum(count) AS Total by host 
     | where Total=0 

You can set the time interval depending on the data it is forwarding.

Hope this helps, Thanks!

[ivanreis]

The splunkd.log file have the information for the UF activity, so you can take a look on this file direct on the UF itself. It should be under /splunkforwarder/var/log/splunk.

My suggestions is to use the Management Console, and you can enable it to monitor the universal forwarder tier. It is not required to index any data, because Splunk by default is already collecting this data in the _internal index. The _internal index, does not count agains the license.
To enable the MC to monitor the splunk forwarders, check this document -> https://docs.splunk.com/Documentation/Splunk/7.3.2/DMC/Configureforwardermonitoring