Getting Data In

Why is my universal forwarder not forwarding?


Hello all!

I have banged my head for about 2 hours trying to figure out why my universal forwarder won't transfer data to my Heavy Forwarder.

Steps I have done:

  • Opened receiving port 80 on Heavy Forwarder
  • The heavy forwarder's forwarding port has been configured correctly (HTTP data inputs forward correctly)
  • netstat -lpnt -> shows that is in LISTEN mode
  • Using tcping.exe from my Windows client, I was able to successfully /tcping.exe server-ip port
  • Ping server-ip is successful
  • I added a forwarder server ./splunk add forward-server server-ip:port with correct port
  • I added the server to NO_PROXY env var


defaultGroup = default-autolb-group

server = ip:port



host = hostname

Universal Forwarder log:

09-20-2018 09:15:52.949 -0400 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 249499 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
09-20-2018 09:15:57.504 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
09-20-2018 09:15:57.504 -0400 INFO  DC:PhonehomeThread - Attempted handshake 2430 times. Will try to re-subscribe to handshake reply
09-20-2018 09:16:00.296 -0400 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: 
09-20-2018 09:16:00.296 -0400 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=72.778 seconds.
09-20-2018 09:16:09.515 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

I have also restarted both my heavy forwarder and universal forwarder.

When running the ./splunk list forward-server, my server and IP is listed under "Configured but inactive"

Any thoughts?

0 Karma


In that case you could try the following:
1.Restart the indexers and confirm that they are listening on port 9997 in their inputs.conf and ensure these servers match those defined in your outputs.conf for UF. Once the indexers are ready to recieve data on 997 (./splunk display listen)...

2.Restart the forwarders and run './splunk display forward-server' again to see if forwarding is activated. This should have cleared it up, if not, re-inspect your configurations

  1. If the above two method fail, you could reset the fishbucket or reset the individual checkpoint for the concered input file using the btprobe command.

Hope that helps.

0 Karma


Are you sure port 80 is available for Splunk? Or is somethine else running on port 80
(im not a windows guy) but is your firewall open for external connections (looks like you checked it form local host

0 Karma


Thanks for your help! Using netstat -tulpn | grep LISTEN I could see that is being used by splunkd, and is LISTEN mode. My "windows client" I referred to from above is my personal computer. I'm able to ping and tcping.exe the port successfully; however, my universal forwarder just doesn't want to budge.

0 Karma

Ultra Champion

So what errors are in the logs before that WARN at the first line of the logs you shared? Because none of these log events say anything about why it fails to connect to the forwarder, that happened before the warning that that output is blocked.

What does the inputs.conf look like on your heavy forwarder?

0 Karma


I setup the receiving / sending on the Heavy Forwarder; however inputs.conf in system/local has:
host = ip-10-227-226-139.ec2.internal

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...