I've installed a universal forwarder on a linux box and configured it, but I'm getting the following errors. I'm running 5.0.1 and the indexer is currently listening on 9997:
From indexer:
11-05-2013 14:02:50.585 -0800 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:60599. Timeout
From forwarder:
11-05-2013 20:23:49.189 -0500 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
11-05-2013 20:23:49.475 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.
11-05-2013 20:24:06.849 -0500 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin
11-05-2013 20:37:09.152 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out
11-05-2013 20:37:09.152 -0500 INFO TcpOutputProc - Detected connection to =xx.xx.xx.xx:9997 closed
11-05-2013 20:37:09.152 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997
11-05-2013 20:37:09.153 -0500 INFO TcpOutputProc - Closing stream for idx==xx.xx.xx.xx:9997
11-05-2013 20:37:29.621 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out
11-05-2013 20:37:49.155 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.
11-05-2013 20:42:09.152 -0500 WARN TcpOutputProc - Shutdown timed out for xx.xx.xx.xx:9997
11-05-2013 20:50:39.168 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out
11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Detected connection to xx.xx.xx.xx:9997 closed
11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997
11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Closing stream for idx=xx.xx.xx.xx:9997
11-05-2013 20:51:00.107 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out
11-05-2013 20:55:39.110 -0500 WARN TcpOutputProc - Shutdown timed out for xx.xx.xx.xx:9997
11-05-2013 20:56:09.110 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out
11-05-2013 20:57:49.114 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.
11-05-2013 21:03:09.116 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out
11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Detected connection to xx.xx.xx.xx:9997 closed
11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997
11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Closing stream for idx=xx.xx.xx.xx:9997
11-05-2013 21:03:29.601 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out
11-05-2013 21:04:09.117 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out
Here is the configuration on the forwarder:
outputs.conf
[tcpout]
defaultGroup = default
[tcpout:default]
server = xx.xx.xx.xx:9997
[tcpout-server://xx.xx.xx.xx:9997]
Well, old question but maybe worth to comment:
Remember to check you have a rule in inputs.conf somewhere.
Check this with
splunk btool inputs list --debug | less
and search for a stanza where there is NO "disable = 1" entry!
HTH,
Holger
add manually into file
opt/splunk/etc/system/local/inputs.conf
[splunktcp://9997]
disabled = 0
had the same problem, couldnt connect to indexer
in windows for universal forwarder installation ( 5.0.4) please check the files in:
path /SplunkUniversalForwarder/etc/system/local
replace the config files under with those from:
path /SplunkUniversalForwarder/etc/apps/Windows/local
restart splunkforwarder:
splunk restart
it should get connected
in splunk host i can see the forwarder has been connected and it has send logs. i had activated some advanced audit features.
What is the configuration on the indexer? Specifically, what is in the inputs.conf
stanza that set up the tcpinput
on 9997?
Hmm...even though it was showing in the web gui, I couldn't find it in any of the inputs.conf files. I confirmed it was listening on 9997 using netstat.
In any case, I explicitly added it to my inputs.conf from the /splunk/etc/apps/search/local folder.
[splunktcp://9997]
connection_host = dns
I am still not seeing any data come in.
Do you definitely have appropriate routing? Since you have redacted your source address it impossible to know if this is relevant. Are there firewalls intervening? Do they have rules to allow TCP on 9997 from source to indexer?
My first thought is that port 9997 is blocked. You should make sure that the port is open from the indexer to the forwarder.
It's not being blocked. I can successfully telnet to port 9997 from the forwarder to the indexer.
Also, if it were blocked, I would not see the error message above from the indexer.