topic Re: Universal Forwarder not sending data - timed out in Getting Data In

Universal Forwarder not sending data - timed out

john_byun — Wed, 06 Nov 2013 18:40:06 GMT

I've installed a universal forwarder on a linux box and configured it, but I'm getting the following errors. I'm running 5.0.1 and the indexer is currently listening on 9997:

From indexer:

11-05-2013 14:02:50.585 -0800 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:60599. Timeout

From forwarder:
11-05-2013 20:23:49.189 -0500 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).

11-05-2013 20:23:49.475 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.

11-05-2013 20:24:06.849 -0500 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin

11-05-2013 20:37:09.152 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:37:09.152 -0500 INFO TcpOutputProc - Detected connection to =xx.xx.xx.xx:9997 closed

11-05-2013 20:37:09.152 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997

11-05-2013 20:37:09.153 -0500 INFO TcpOutputProc - Closing stream for idx==xx.xx.xx.xx:9997

11-05-2013 20:37:29.621 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:37:49.155 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.

11-05-2013 20:42:09.152 -0500 WARN TcpOutputProc - Shutdown timed out for xx.xx.xx.xx:9997

11-05-2013 20:50:39.168 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Detected connection to xx.xx.xx.xx:9997 closed

11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997

11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Closing stream for idx=xx.xx.xx.xx:9997

11-05-2013 20:51:00.107 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:55:39.110 -0500 WARN TcpOutputProc - Shutdown timed out for xx.xx.xx.xx:9997

11-05-2013 20:56:09.110 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:57:49.114 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.

11-05-2013 21:03:09.116 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Detected connection to xx.xx.xx.xx:9997 closed

11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997

11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Closing stream for idx=xx.xx.xx.xx:9997

11-05-2013 21:03:29.601 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 21:04:09.117 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

Here is the configuration on the forwarder:

outputs.conf

[tcpout]
defaultGroup = default

[tcpout:default]
server = xx.xx.xx.xx:9997

[tcpout-server://xx.xx.xx.xx:9997]

Re: Universal Forwarder not sending data - timed out

lguinn2 — Wed, 06 Nov 2013 18:54:26 GMT

My first thought is that port 9997 is blocked. You should make sure that the port is open from the indexer to the forwarder.

Re: Universal Forwarder not sending data - timed out

grijhwani — Wed, 06 Nov 2013 18:57:00 GMT

Do you definitely have appropriate routing? Since you have redacted your source address it impossible to know if this is relevant. Are there firewalls intervening? Do they have rules to allow TCP on 9997 from source to indexer?

Re: Universal Forwarder not sending data - timed out

john_byun — Wed, 06 Nov 2013 18:58:54 GMT

It's not being blocked. I can successfully telnet to port 9997 from the forwarder to the indexer.

Also, if it were blocked, I would not see the error message above from the indexer.

Re: Universal Forwarder not sending data - timed out

lguinn2 — Wed, 06 Nov 2013 21:21:50 GMT

What is the configuration on the indexer? Specifically, what is in the inputs.conf stanza that set up the tcpinput on 9997?

Re: Universal Forwarder not sending data - timed out

john_byun — Wed, 06 Nov 2013 23:29:52 GMT

Hmm...even though it was showing in the web gui, I couldn't find it in any of the inputs.conf files. I confirmed it was listening on 9997 using netstat.

In any case, I explicitly added it to my inputs.conf from the /splunk/etc/apps/search/local folder.

[splunktcp://9997]
connection_host = dns

I am still not seeing any data come in.

Re: Universal Forwarder not sending data - timed out

Akili — Tue, 24 Dec 2013 10:20:57 GMT

had the same problem, couldnt connect to indexer
in windows for universal forwarder installation ( 5.0.4) please check the files in:
path /SplunkUniversalForwarder/etc/system/local
replace the config files under with those from:
path /SplunkUniversalForwarder/etc/apps/Windows/local
restart splunkforwarder:
splunk restart

it should get connected
in splunk host i can see the forwarder has been connected and it has send logs. i had activated some advanced audit features.

Re: Universal Forwarder not sending data - timed out

kuido7Xdoc — Wed, 23 Apr 2014 18:01:22 GMT

add manually into file
opt/splunk/etc/system/local/inputs.conf

[splunktcp://9997]
disabled = 0

Re: Universal Forwarder not sending data - timed out

hsesterhenn_spl — Fri, 22 Jul 2016 15:39:14 GMT

Well, old question but maybe worth to comment:

Remember to check you have a rule in inputs.conf somewhere.

Check this with

splunk btool inputs list --debug | less

and search for a stanza where there is NO "disable = 1" entry!

HTH,

Holger