topic Re: Why is my universal forwarder not forwarding? in Getting Data In

Why is my universal forwarder not forwarding?

thomastaylor — Fri, 21 Sep 2018 15:53:47 GMT

Hello all!

I have banged my head for about 2 hours trying to figure out why my universal forwarder won't transfer data to my Heavy Forwarder.

Steps I have done:

Opened receiving port 80 on Heavy Forwarder
The heavy forwarder's forwarding port has been configured correctly (HTTP data inputs forward correctly)
netstat -lpnt -> shows that 0.0.0.0:80 is in LISTEN mode
Using tcping.exe from my Windows client, I was able to successfully /tcping.exe server-ip port
Ping server-ip is successful
I added a forwarder server ./splunk add forward-server server-ip:port with correct port
I added the server to NO_PROXY env var

outputs.conf:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = ip:port

[tcpout-server://ip:port]

inputs.conf

[default]
host = hostname

Universal Forwarder log:

09-20-2018 09:15:52.949 -0400 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 249499 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
09-20-2018 09:15:57.504 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
09-20-2018 09:15:57.504 -0400 INFO  DC:PhonehomeThread - Attempted handshake 2430 times. Will try to re-subscribe to handshake reply
09-20-2018 09:16:00.296 -0400 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: 
09-20-2018 09:16:00.296 -0400 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=72.778 seconds.
09-20-2018 09:16:09.515 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

I have also restarted both my heavy forwarder and universal forwarder.

When running the ./splunk list forward-server, my server and IP is listed under "Configured but inactive"

Any thoughts?

Re: Why is my universal forwarder not forwarding?

teunlaan — Wed, 26 Sep 2018 07:52:27 GMT

Are you sure port 80 is available for Splunk? Or is somethine else running on port 80
(im not a windows guy) but is your firewall open for external connections (looks like you checked it form local host 0.0.0.0:80)

Re: Why is my universal forwarder not forwarding?

thomastaylor — Wed, 26 Sep 2018 14:45:43 GMT

Thanks for your help! Using netstat -tulpn | grep LISTEN I could see that 0.0.0.0:80 is being used by splunkd, and is LISTEN mode. My "windows client" I referred to from above is my personal computer. I'm able to ping and tcping.exe the port successfully; however, my universal forwarder just doesn't want to budge.

Re: Why is my universal forwarder not forwarding?

FrankVl — Wed, 26 Sep 2018 14:57:45 GMT

So what errors are in the logs before that WARN at the first line of the logs you shared? Because none of these log events say anything about why it fails to connect to the forwarder, that happened before the warning that that output is blocked.

What does the inputs.conf look like on your heavy forwarder?

Re: Why is my universal forwarder not forwarding?

thomastaylor — Wed, 26 Sep 2018 15:17:04 GMT

I setup the receiving / sending on the Heavy Forwarder; however inputs.conf in system/local has:
[default]
host = ip-10-227-226-139.ec2.internal

Re: Why is my universal forwarder not forwarding?

mguhad — Thu, 30 May 2019 22:09:34 GMT

In that case you could try the following:
1.Restart the indexers and confirm that they are listening on port 9997 in their inputs.conf and ensure these servers match those defined in your outputs.conf for UF. Once the indexers are ready to recieve data on 997 (./splunk display listen)...

2.Restart the forwarders and run './splunk display forward-server' again to see if forwarding is activated. This should have cleared it up, if not, re-inspect your configurations

If the above two method fail, you could reset the fishbucket or reset the individual checkpoint for the concered input file using the btprobe command.

Hope that helps.