Issue with forwarder. Couldn't complete HTTP request: Connection timed out

[tribunal]

I have several forwarders, all installed on Ubuntu 14.04 boxes. One of them stopped working but the rest are fine. After troubleshooting, the only difference on the one not working from the others is that when I try these commands:

./splunk list forward-server
./splunk show deploy-poll

I get an error which is "Couldn't complete HTTP request: Connection timed out"

These commands work on my other forwarders and immediately ask me for my credentials. When I try these commands on the box that isn't working, it takes about 30 seconds and then gives me that error. I can't find any information about this error online (I find the error but not anything about why a connection would time out. The outputs.conf file is the same on every box and any other .conf file I know about is the same.

Anyone know what would cause this or even a log file I can view that might give me a clue? Thanks!

[mayurr98]

go to
Etc/system/local/inputs.conf

[splunktcp://9997]
connection_host = none

restart Splunk server and it will be fixed. DNS is holding it all up.

[woodcock]

Make sure that a firewall is not running and blocking ports.

[DalJeanis]

I'd check the ports on the box. When it seems like a box isn't listening, it's possible that it isn't listening.

[ddrillic]

Cheerful place to start at I can't find my data!

Especially the section which says -

-- Are my forwarders connecting to my receiver? Which IP addresses are connecting to Splunk as inputs, and how many times is each IP logged in metrics.log?

[tribunal]

Thank you for the input. I ran the command "index=_internal source=metrics.log tcpin_connections | stats count by sourceIp" in Splunk and the IP address of the box is showing up. Does this mean that it is sending something to Splunk but Splunk is not displaying the events? What could cause Splunk to get events but not display them?