Can't get Universal Forwarder to work.

[sackerman]

I have successfully installed the receiving server, setup the receiver, opened firewall ports. In setting up the forwarder after entering the server ip/username/password info it succeeded. I setup 1 line in the inputs.conf: [monitor:///var/log/httpd/error_log]
and have restarted splunk on the forwarder and restarted Apache on the forwarded to generate some error messages. I ran the 'list monitor' command on the forwarder and it showed that it was indeed monitoring '/var/log/httpd/error_log' (as well as the splunk logs). However there is nothing showing up on the receiver and there are new entries in the 'error_log' and I am not sure where to start looking.

[sackerman]

Found another entry in the forwarder splunkd.log:

11-12-2011 11:06:48.173 -0500 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying...

After searching I only found references to solutions when your metrics.log has 'blocked=true' but have found no entries in either the forwarder and receiver metrics log.

[sackerman]

Okay, I think I may have found the problem
forwarder:metrics.log:

11-12-2011 11:46:57.189 -0500 INFO StatusMgr - destHost=173-160-51-65-colorado.hfc.comcastbusiness.net, destIp=173.160.51.65, destPort=9997, eventType=connect_fail, publisher=tcpout, sourcePort=8089, statusee=TcpOutputProcessor

But I am not sure what the cause is. I have the source port opened up on the forwarder and the destination port opened up on the receiver. I have looked in the 'messages' log on the receiver and I don't see that the connection has been blocked. I am looking in the same log on the forwarder and don't see that the connection has been blocked. So what's next?

[sackerman]

*******/opt/splunkforwarder/bin/splunk list forward-server

Active forwards:
None
Configured but inactive forwards:
173.160.xx.xx:9997

*******inputs.conf
[default]
host = u15437226.onlinehome-server.com

[monitor:///var/log/httpd/error_log]
[monitor:///var/log/httpd/ssl_error_log]

*******outputs.conf
[tcpout]
defaultGroup = 173.160.xx.xx_9997
disabled = false

[tcpout:173.160.xx.xx_9997]
server = 173.160.xx.xx:9997

[tcpout-server://173.160.xx.xx:9997]

Not sure why it lists 'Configured but inactive forwards'?

[mikelanghorst]

It would be helpful to see your inputs.conf on the indexer and outputs.conf on the forwarder. You can obscure or modify any sensitive data.

What's the output of ./splunk list forward-server on the forwarder?

[mikesaia]

Check the /opt/splunk/var/log/splunk/splunkd and metrics.log on the forwarder and receiver. You should be able to get some good info from there as a starting point. On the receiver to view the log directory you can just run a search, index="_internal" to see the splunk log messages. But on the universal forwarder since there is no web interface you will have to manually view the log files. If you want to show some output from those logs that would help.