Getting Data In
Highlighted

Issue with forwarder. Couldn't complete HTTP request: Connection timed out

New Member

I have several forwarders, all installed on Ubuntu 14.04 boxes. One of them stopped working but the rest are fine. After troubleshooting, the only difference on the one not working from the others is that when I try these commands:

./splunk list forward-server
./splunk show deploy-poll

I get an error which is "Couldn't complete HTTP request: Connection timed out"

These commands work on my other forwarders and immediately ask me for my credentials. When I try these commands on the box that isn't working, it takes about 30 seconds and then gives me that error. I can't find any information about this error online (I find the error but not anything about why a connection would time out. The outputs.conf file is the same on every box and any other .conf file I know about is the same.

Anyone know what would cause this or even a log file I can view that might give me a clue? Thanks!

0 Karma
Highlighted

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

Ultra Champion

Cheerful place to start at I can't find my data!

Especially the section which says -

-- Are my forwarders connecting to my receiver? Which IP addresses are connecting to Splunk as inputs, and how many times is each IP logged in metrics.log?

0 Karma
Highlighted

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

New Member

Thank you for the input. I ran the command "index=internal source=metrics.log tcpinconnections | stats count by sourceIp" in Splunk and the IP address of the box is showing up. Does this mean that it is sending something to Splunk but Splunk is not displaying the events? What could cause Splunk to get events but not display them?

0 Karma
Highlighted

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

SplunkTrust
SplunkTrust

I'd check the ports on the box. When it seems like a box isn't listening, it's possible that it isn't listening.

0 Karma
Highlighted

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

Esteemed Legend

Make sure that a firewall is not running and blocking ports.

0 Karma
Highlighted

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

SplunkTrust
SplunkTrust

go to
Etc/system/local/inputs.conf

[splunktcp://9997]
connection_host = none

restart Splunk server and it will be fixed. DNS is holding it all up.

0 Karma