Monitoring linux forward to ensure communication

timmy13 · ‎03-27-2015

We encountered a forwarder that was not sending data to the indexer nor phoning home. This forwarder's data is a dependancy of many other processes. We found that the splunkd processes were orphaned, killed them, and indexing began working. Is there a way to monitor that splunk is actually sending data?

aljohnson_splun · ‎03-27-2015

Are you really actually looking to see if data is being sent ? Or are you maybe more interested in whether or not its being received ?

I would use the

Distributed Management Console

Then check out Indexing Performance

Or check out this answer, where they suggest going to

http://54.183.100.158:8000/en-US/app/search/indexing_volume

in order to split by host.

Or maybe you'd rather go old school, ssh into your indexer, and

sudo tcpdump port 9997

where 9997 is the port that you are forwarding to.

Or maybe you'd just like to use the _internal index. Or you could check splunkd.log.

See this answer for quite a few more options.

A link from the Wiki on troubleshooting forwarding.