Not receiving data from linux universal forwarders

[dlynum]

I'm running Splunk 4.2.3 on Centos 6, and universal forwarder 4.2.3 on (2) Centos 5 servers and (2) Windows 2008 R2 Servers. I installed the universal forwarder onto the Centos and Windows servers within 30 minutes of each other. I'm seeing data from the Windows servers in my Splunk host/server, but now the Centos servers. Each of the forwarders uses the same ip address and port number for the host/receiver. A "netstat -an | grep port number" run on the Centos forwarders shows that they are connected to my host/receiver. The "/opt/splunkforwarder/var/log/splunk/splunkd.log" also shows that the forwarder is connected to the host/received. I've restarted the forwarder service on each forwarder, and changed the forwarder log afterward. In each case the forwarder connects to the host/receiver. I have no errors in the log file, no errors in /var/log/messages. The forwarder service starts without error. Obviously there isn't a firewall on the forwarders that's blocking the communications between the host and the forwarder, as shown by the output of "netstat -an | grep port number". I did check "inputstatus" as recommended in another users's question, but being new to Splunk, I couldn't find anything that looked wrong.

So, does anyone have any ideas?

[mikelanghorst · Accepted solution]

Do you have anything configured to monitor on the CentOS server? With a default install a Universal Forwarder only monitors the $SPLUNK_HOME/var/log/splunk directory.

On the forwarder, what does "$SPLUNK_HOME/bin/splunk list monitor" report?

[mikelanghorst]

Oh, and if it helped, please mark this as answered. There should be a check mark for the answer available for the person that submitted the question.

[mikelanghorst]

dlynum - Take a look at the "Splunk for Unix and Linux" app available on SplunkBase. This would pickup the standard /var/logs and such. It also includes some scripted inputs you may find helpful. http://splunk-base.splunk.com/apps/22314/splunk-for-unix-and-linux

[dlynum]

Mike,

With further googling I found that in order to monitor directories other than the default, I needed to add "monitor:/// directory to monitor". Now that I've done this I'm seeing what I need to see from one of my Linux forwarders. Once I did that, I'm now seeing this particular Linux forwarder in the list of hosts on the search summary page. So I've been able my original question.

[dlynum]

I did some further googling on the problem I'm having, and ran into this, http://splunk-base.splunk.com/answers/465/ive-set-up-a-forwarder-but-im-not-receving-any-events-on-t...

When I ran this command on my Splunk host, index=_internal source=*metrics.log tcpin_connections,
I did in fact find my linux forwarder. But on the search summary page on my host, only (3) hosts are listed. Those hosts are the Splunk host itself, and my Windows servers. So even though I can now find my linux forwarder, it's not listed in the host section of the search summary page. Why is that? Should it be listed?

[dlynum]

Voltaire,

I didn't specify an index. It checked the output.cfg on both of my Windows forwarders, and neither of their output.cfg files have an index specified. But of course I'm receiving data from the Windows forwarders just fine. Also, the output.cfg file for both the Windows and Linux forwarders is identical. When I look through the Splunk log files on my Splunk host, here's what I see for one of my Linux forwarders:

09-23-2011 11:32:31.143 -0700 INFO TcpInputProc - Waiting for connection from src=xxx.xxx.xxx.xxx:36569 to close before shutting down TcpInputProcessor.

[Voltaire]

What index did you configure for your light forwarders to use when you setup the data inputs? Check that value, and do a search on that index from your main indexer.From search app, "index="yourindex" * alltime, otherwise check the splunkd logs, etc... in your SSPLUNK path/var/log directories for any errors or crashed.