I have banged my head for about 2 hours trying to figure out why my universal forwarder won't transfer data to my Heavy Forwarder.
Steps I have done:
[tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = ip:port [tcpout-server://ip:port]
[default] host = hostname
Universal Forwarder log:
09-20-2018 09:15:52.949 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 249499 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. 09-20-2018 09:15:57.504 -0400 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 09-20-2018 09:15:57.504 -0400 INFO DC:PhonehomeThread - Attempted handshake 2430 times. Will try to re-subscribe to handshake reply 09-20-2018 09:16:00.296 -0400 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr: 09-20-2018 09:16:00.296 -0400 INFO HttpPubSubConnection - Could not obtain connection, will retry after=72.778 seconds. 09-20-2018 09:16:09.515 -0400 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
I have also restarted both my heavy forwarder and universal forwarder.
When running the ./splunk list forward-server, my server and IP is listed under "Configured but inactive"
In that case you could try the following:
1.Restart the indexers and confirm that they are listening on port 9997 in their inputs.conf and ensure these servers match those defined in your outputs.conf for UF. Once the indexers are ready to recieve data on 997 (./splunk display listen)...
2.Restart the forwarders and run './splunk display forward-server' again to see if forwarding is activated. This should have cleared it up, if not, re-inspect your configurations
Hope that helps.
Are you sure port 80 is available for Splunk? Or is somethine else running on port 80
(im not a windows guy) but is your firewall open for external connections (looks like you checked it form local host 0.0.0.0:80)
Thanks for your help! Using
netstat -tulpn | grep LISTEN I could see that 0.0.0.0:80 is being used by splunkd, and is LISTEN mode. My "windows client" I referred to from above is my personal computer. I'm able to ping and tcping.exe the port successfully; however, my universal forwarder just doesn't want to budge.
So what errors are in the logs before that WARN at the first line of the logs you shared? Because none of these log events say anything about why it fails to connect to the forwarder, that happened before the warning that that output is blocked.
What does the inputs.conf look like on your heavy forwarder?
I setup the receiving / sending on the Heavy Forwarder; however inputs.conf in system/local has:
host = ip-10-227-226-139.ec2.internal