How to activate forwarder server?

[jhl226116]

Hi Guys,

I am struggling to send data from remote machine to Splunk server. I have tried the steps mentioned in the link but still no luck:
https://answers.splunk.com/answers/48760/how-to-activate-forward-server.html

Can anyone tell me how to activate forward server?

Running Splunk server and Forwarder on virtual Ubuntu platform.

Indexer: 10.10.50.49
Universal Forwarder: 10.10.50.18

root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.49:9997
Added forwarding to: 10.10.50.49:9997.

root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
10.10.50.49:9997

Port 9997 has been enabled in the Indexer.

root@indexer: /opt/splunk/bin# ./splunk list forward-server
Active forwards: 
SPsvr:9997
Configured but inactive forwards:
None

I can ping between Indexer =(10.10.50.49) and forwarder(10.10.50.18) vice-versa
I have disabled Ubuntu firewall on both Indexer and Forwarder

root@indexer:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

root@forwarder:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

Not sure if my outputs.conf is configured correctly. I checked the document but am not exactly sure. Here is my outputs.conf from the forwarder:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.10.50.49:9997

[tcpout-server://10.10.50.49:9997]

If someone can tell me what I'm doing wrong or how I can resolve this issue, I would really appreciate it.

I'm almost close to giving up if there's no concrete answer on this. I'd like to at least know what else I can do from here.

Thanks,

[skoelpin]

Don't give up.. You narrowed down the problem and have a quick and easy way to test.

[jhl226116]

Thanks for encouraging me, I have narrowed down the problem and getting closer to resolving it since I got the telnet session working now from forwarder to the indexer.

Continuing to research for answers and solutions..

[skoelpin]

On the server with the universal forwarder, go to /opt/splunk/var/log/splunk/splunkd.log and see if there's any errors.

Can you also post the contents of your inputs.conf?

[jhl226116]

There were around 2468 lines of logs in the forawrder. This is the first time I checked as I only just came to know where to check the logs. I'm just pasting last page of the logs FYI.

root@forwarder#nano /opt/splunkforwarder/var/log/splunk/splunkd.log

03-21-2017 13:37:17.842 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:37:17.842 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:37:47.689 +1100 ERROR TcpOutputProc - Can't find or illegal IP address or Name: SPsvr
03-21-2017 13:37:47.690 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:37:47.690 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:38:13.058 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 8800 seconds.
03-21-2017 13:38:17.553 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:38:17.553 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:38:47.401 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:38:47.401 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:39:17.247 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:39:17.247 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:39:47.093 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:39:47.093 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:39:53.068 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 8900 seconds.
03-21-2017 13:40:16.941 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:40:16.941 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:40:46.790 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:40:46.790 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:41:16.635 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:41:16.635 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:41:33.076 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9000 seconds.
03-21-2017 13:41:46.482 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:41:46.482 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:42:16.327 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:42:16.327 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:42:46.174 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:42:46.174 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:43:13.085 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9100 seconds.
03-21-2017 13:43:16.019 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:43:16.019 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:43:45.874 +1100 ERROR TcpOutputProc - Can't find or illegal IP address or Name: SPsvr
03-21-2017 13:43:45.874 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:43:45.874 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:44:15.721 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:44:15.721 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:44:45.570 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:44:45.570 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:44:53.094 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9200 seconds.
03-21-2017 13:45:15.456 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:45:15.456 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:45:45.305 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:45:45.305 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:46:15.154 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:46:15.154 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:46:33.110 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9300 seconds.
03-21-2017 13:46:45.012 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:46:45.012 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:47:14.861 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:47:14.861 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed

• Here's my inputs.conf from the forwarder. Not sure if it's configured properly, I don't exactly know where to look.

Forwarder Inputs.Conf

[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup

[blacklist:$SPLUNK_HOME/etc/auth]

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME/etc/splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[batch://$SPLUNK_HOME/var/spool/splunk]
move_policy = sinkhole
crcSalt =

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
queue = stashparsing
sourcetype = stash_new
move_policy = sinkhole
crcSalt =

[fschange:$SPLUNK_HOME/etc]

poll every 10 minutes

pollPeriod = 600

generate audit events into the audit index, instead of fschange events

signedaudit=true
recurse=true
followLinks=false
hashMaxSize=-1
fullEvent=false
sendEventMaxSize=-1
filesPerDelay = 10
delayInMills = 100

[udp]
connection_host=ip

[tcp]
acceptFrom=*
connection_host=dns

[splunktcp]
route=has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
acceptFrom=*
connection_host=ip

[script]
interval = 60.0
start_by_shell = true

[SSL]
default cipher suites that splunk allows. Change this if you wish to increase the security
of SSL connections, or to lower it if you having trouble connecting to splunk.
cipherSuite = ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
allowSslRenegotiation = true
sslQuietShutdown = false

Allow only sslv3 and above connections
sslVersions = *,-ssl2

[skoelpin]

So it looks like your being blocked by a firewall or there is nothing listening.. Did you enable your indexer to listen on port 9997? If not then go to Settings>Receiving and Listening and add 9997

[jhl226116]

9997 was already added and enabled on the indexer.

Ubuntu firewalls on both hosts has already been disabled.

I have just created a new rule in Cisco ASA firewall in the network lab to allow necessary Splunk ports to communicate between the indexer and forwarder.

Ports allowed from any source any to any destination within my internal network range.

Ports allowed:
TCP 8000 - Spluk Web
TCP 8080 - Indexer to Indexer Replication
TCP 8088 - mgmt for myself only
TCP 8089 - mgmt
TCP 9997 - Indexing
UDP 514 - Syslog

Also ICMP, domain, http, https has always been enabled already.

[skoelpin]

Can you do a telnet from your forwarder to your indexer to verify you can connect?

telnet IndexerIP 9997

[jhl226116]

I was unable to telnet from forwarder to the indexer. Only can ping vice-versa. Where does this give you indication?

root@forwarder:~# telnet 10.10.50.49
Trying 10.10.50.49...
telnet: unable to connect to remote host: Connection refused

root@forwarder:~# ping 10.10.50.49
PING 10.10.50.49 (10.10.50.49) 56(84) bytes of data.
64 bytes from 10.10.50.49: icmp_seq=1 ttl=64 time=0.213 ms
64 bytes from 10.10.50.49: icmp_seq=2 ttl=64 time=0.227 ms
64 bytes from 10.10.50.49: icmp_seq=3 ttl=64 time=0.233 ms
64 bytes from 10.10.50.49: icmp_seq=4 ttl=64 time=0.259 ms
64 bytes from 10.10.50.49: icmp_seq=5 ttl=64 time=0.207 ms
64 bytes from 10.10.50.49: icmp_seq=6 ttl=64 time=0.259 ms
64 bytes from 10.10.50.49: icmp_seq=7 ttl=64 time=0.210 ms
64 bytes from 10.10.50.49: icmp_seq=8 ttl=64 time=0.253 ms

[skoelpin]

This means your port is blocked which is the reason why data is not being sent via 9997 and also explains the connection refused message in your splunkd logs.

You need to go take another look at your firewall settings and test its actually open by using telnet

[skoelpin]

So you said its working now. Are you able to forward events to the indexer now via port 9997?

[jhl226116]

I still can't get past the credentials when trying telnet from forwarder (10.10.50.18) to the indexer (10.10.50.49). It's not accepting my password.... I used the same password that I used to logon to the Linux graphical desktop but it's not accepting it.
Still trying to figure out how to get past this point... maybe trying to even reset Ubuntu password all together since it's not letting me through.

root@forwarder:~# telnet 10.10.50.49
Trying 10.10.50.49...
Connected to 10.10.50.49.
Escape character is '^]'.
Ubuntu 16.04.1 LTS
SPsvr login: admin
Password:

Login failed.

[jhl226116]

I suppose you were right it seems telnet wasn't enabled somewhere but now I have enabled it and am now able to telnet to the Indexer from the forwarder.

root@SPsvr:/opt/splunk/bin# nc -v localhost 23
Connection to localhost 23 port [tcp/telnet] succeeded!

root@SPsvr:/opt/splunk/bin# netstat -nat | grep 23
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:48838 127.0.0.1:23 ESTABLISHED
tcp 0 0 10.10.50.49:49508 74.125.23.189:443 ESTABLISHED
tcp 0 0 10.10.50.49:50208 74.125.23.189:443 ESTABLISHED
tcp 0 0 127.0.0.1:23 127.0.0.1:48838 ESTABLISHED

root@SPsvr:/opt/splunk/bin# sudo netstat -tanpu | grep ":23"
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 7270/inetd
tcp 0 0 127.0.0.1:48838 127.0.0.1:23 ESTABLISHED 9735/nc
tcp 0 0 127.0.0.1:23 127.0.0.1:48838 ESTABLISHED 9736/in.telnetd

root@SPsvr:/opt/splunk/bin# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN 968/dnsmasq
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 7270/inetd
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 25816/splunkd
tcp 0 0 0.0.0.0:8191 0.0.0.0:* LISTEN 25826/mongod
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 25816/splunkd
tcp 0 0 127.0.0.1:8065 0.0.0.0:* LISTEN 25939/python
udp 0 0 0.0.0.0:5353 0.0.0.0:* 2458/chrome
udp 0 0 0.0.0.0:5353 0.0.0.0:* 2458/chrome
udp 0 0 0.0.0.0:5353 0.0.0.0:* 589/avahi-daemon: r
udp 0 0 0.0.0.0:38843 0.0.0.0:* 589/avahi-daemon: r
udp 0 0 127.0.1.1:53 0.0.0.0:* 968/dnsmasq
udp 0 0 0.0.0.0:68 0.0.0.0:* 949/dhclient
udp 0 0 0.0.0.0:514 0.0.0.0:* 25816/splunkd
udp 0 0 0.0.0.0:631 0.0.0.0:* 30478/cups-browsed
udp6 0 0 :::60258 :::* 589/avahi-daemon: r
udp6 0 0 :::5353 :::* 2458/chrome
udp6 0 0 :::5353 :::* 589/avahi-daemon: r

However I can't get past the credentials when telnetting from the forwarder (10.10.50.18) to the indexer (10.10.50.49).
I used the same credentials for everything so there's no doubt my passwords are corrrect but I'm unable to telnet into the indexer.
Previously telnet to indexer was refusing connection but now I'm one step closer. Can't stop now. Trying to figure out what the password is and how to get past this point now as it keeps saying login is incorrect.

root@forwarder:~# telnet 10.10.50.49
Trying 10.10.50.49...
Connected to 10.10.50.49.
Escape character is '^]'.
Ubuntu 16.04.1 LTS
SPsvr login: admin
Password: