topic Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out in Getting Data In

Issue with forwarder. Couldn't complete HTTP request: Connection timed out

tribunal — Wed, 06 Dec 2017 19:49:27 GMT

I have several forwarders, all installed on Ubuntu 14.04 boxes. One of them stopped working but the rest are fine. After troubleshooting, the only difference on the one not working from the others is that when I try these commands:

./splunk list forward-server
./splunk show deploy-poll

I get an error which is "Couldn't complete HTTP request: Connection timed out"

These commands work on my other forwarders and immediately ask me for my credentials. When I try these commands on the box that isn't working, it takes about 30 seconds and then gives me that error. I can't find any information about this error online (I find the error but not anything about why a connection would time out. The outputs.conf file is the same on every box and any other .conf file I know about is the same.

Anyone know what would cause this or even a log file I can view that might give me a clue? Thanks!

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

ddrillic — Wed, 06 Dec 2017 20:37:09 GMT

Cheerful place to start at I can't find my data!

Especially the section which says -

-- Are my forwarders connecting to my receiver? Which IP addresses are connecting to Splunk as inputs, and how many times is each IP logged in metrics.log?

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

DalJeanis — Wed, 06 Dec 2017 21:40:23 GMT

I'd check the ports on the box. When it seems like a box isn't listening, it's possible that it isn't listening.

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

tribunal — Tue, 29 Sep 2020 17:06:23 GMT

Thank you for the input. I ran the command "index=_internal source=metrics.log tcpin_connections | stats count by sourceIp" in Splunk and the IP address of the box is showing up. Does this mean that it is sending something to Splunk but Splunk is not displaying the events? What could cause Splunk to get events but not display them?

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

woodcock — Fri, 08 Dec 2017 06:26:50 GMT

Make sure that a firewall is not running and blocking ports.

Re: Issue with forwarder. Couldn't complete HTTP request: Connection timed out

mayurr98 — Fri, 08 Dec 2017 08:54:43 GMT

go to
Etc/system/local/inputs.conf

[splunktcp://9997]
connection_host = none

restart Splunk server and it will be fixed. DNS is holding it all up.