Getting Data In
Highlighted

Why can't I see my forwarder data on my search head or the indexer on separate Centos Linux VMs?

Path Finder

I have one Search Head(SH)/DS, one indexer, and one forwarder all on separate Centos Linux VMs.

I cannot see any forwarder data on the SH or the indexer.

If I run the following on the indexer index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp I get no result found.

If I run the following on the indexer: index=_internal */pathwheremydataresides/* I get results so that path that is being monitored by the forwarder shows up in the internal log

If I run the following on the indexer index=_internal source=*metrics.log* group=queue tcpout | stats count by name I get no result found.

If I run the following on the forwarder $SPLUNK_HOME/bin/splunk search 'index=_internal source=*metrics.log* destHost | dedup destHost' I get "FATAL: Unable to read the job status"

If I run the following on the forwarder ./splunk list monitor it successfully shows my path that I am monitoring.

If I run a search on a different index that was previously set up with one shot data on that machine (not forwarder data) I get results.

The indexes that I created on each of indexers are reserved for loading forwarded data only. I am not mixing data sources.

Under the forward management panel, I can see all of my apps, all of my clients as well as my server classes correctly associating apps to clients

All of the search peers are connected and healthy.

All apps have been successfully deployed out.

inputs.conf & outputs.conf look as expected on the forwarder

inputs.conf look as expected on the indexer

Any guidance?

0 Karma
Highlighted

Re: Why can't I see my forwarder data on my search head or the indexer on separate Centos Linux VMs?

Path Finder

I am going to answer my own question here since I solved it in case it may be of benefit to someone else.

First, verify that there is not a firewall issue on the indexer servers by doing "telnet local 9997" If this works (which mine did) your good to go here. If it does not work check your iptables.

Second, manually check that there is connectivity between the FW and INX (and not with ping) by doing "telnet ipaddressINDX 9997".
If this fails(which mine did not) is a network firewall problem. BTW I am assuming that you have already verified that the correct hostname is shown in the server.conf file on all servers.

Third, with all apps pushed out successfully by your DS verify that Splunk is happy with all of your conf files by doing "./splunk cmd btool check". This will make sure all the conf files are written in a way that will be accepted by Splunk.

Fourth, verify that the path where you data resides is being monitored by running "./splunk list monitor"

Lastly if all is well from everything above (as it was in my case) go and look at the following file to verify that the forwarder is successfully connecting to your indexer (which mine was not in my case):
/opt/splunkforwarder/var/log/splunk/splunkd.log

If there is no connection, reboot your forwarder. That is what I did and it worked.

0 Karma