Why is my universal forwarder not forwarding?

[thomastaylor]

Hello all!

I have banged my head for about 2 hours trying to figure out why my universal forwarder won't transfer data to my Heavy Forwarder.

Steps I have done:

• Opened receiving port 80 on Heavy Forwarder
• The heavy forwarder's forwarding port has been configured correctly (HTTP data inputs forward correctly)
• netstat -lpnt -> shows that 0.0.0.0:80 is in LISTEN mode
• Using tcping.exe from my Windows client, I was able to successfully /tcping.exe server-ip port
• Ping server-ip is successful
• I added a forwarder server ./splunk add forward-server server-ip:port with correct port
• I added the server to NO_PROXY env var

outputs.conf:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = ip:port

[tcpout-server://ip:port]

inputs.conf

[default]
host = hostname

Universal Forwarder log:

09-20-2018 09:15:52.949 -0400 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 249499 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
09-20-2018 09:15:57.504 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
09-20-2018 09:15:57.504 -0400 INFO  DC:PhonehomeThread - Attempted handshake 2430 times. Will try to re-subscribe to handshake reply
09-20-2018 09:16:00.296 -0400 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: 
09-20-2018 09:16:00.296 -0400 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=72.778 seconds.
09-20-2018 09:16:09.515 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

I have also restarted both my heavy forwarder and universal forwarder.

When running the ./splunk list forward-server, my server and IP is listed under "Configured but inactive"

Any thoughts?

[mguhad]

In that case you could try the following:
1.Restart the indexers and confirm that they are listening on port 9997 in their inputs.conf and ensure these servers match those defined in your outputs.conf for UF. Once the indexers are ready to recieve data on 997 (./splunk display listen)...

2.Restart the forwarders and run './splunk display forward-server' again to see if forwarding is activated. This should have cleared it up, if not, re-inspect your configurations

1. If the above two method fail, you could reset the fishbucket or reset the individual checkpoint for the concered input file using the btprobe command.

Hope that helps.

[teunlaan]

Are you sure port 80 is available for Splunk? Or is somethine else running on port 80
(im not a windows guy) but is your firewall open for external connections (looks like you checked it form local host 0.0.0.0:80)

[thomastaylor]

Thanks for your help! Using netstat -tulpn | grep LISTEN I could see that 0.0.0.0:80 is being used by splunkd, and is LISTEN mode. My "windows client" I referred to from above is my personal computer. I'm able to ping and tcping.exe the port successfully; however, my universal forwarder just doesn't want to budge.

[FrankVl]

So what errors are in the logs before that WARN at the first line of the logs you shared? Because none of these log events say anything about why it fails to connect to the forwarder, that happened before the warning that that output is blocked.

What does the inputs.conf look like on your heavy forwarder?

[thomastaylor]

I setup the receiving / sending on the Heavy Forwarder; however inputs.conf in system/local has:
[default]
host = ip-10-227-226-139.ec2.internal