topic Re: How to activate forwarder server? in Getting Data In

How to activate forwarder server?

jhl226116 — Tue, 21 Mar 2017 01:06:30 GMT

Hi Guys,

I am struggling to send data from remote machine to Splunk server. I have tried the steps mentioned in the link but still no luck:
https://answers.splunk.com/answers/48760/how-to-activate-forward-server.html

Can anyone tell me how to activate forward server?

Running Splunk server and Forwarder on virtual Ubuntu platform.

Indexer: 10.10.50.49
Universal Forwarder: 10.10.50.18

root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.49:9997
Added forwarding to: 10.10.50.49:9997.

root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
10.10.50.49:9997

Port 9997 has been enabled in the Indexer.

root@indexer: /opt/splunk/bin# ./splunk list forward-server
Active forwards: 
SPsvr:9997
Configured but inactive forwards:
None

I can ping between Indexer =(10.10.50.49) and forwarder(10.10.50.18) vice-versa
I have disabled Ubuntu firewall on both Indexer and Forwarder

root@indexer:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

root@forwarder:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

Not sure if my outputs.conf is configured correctly. I checked the document but am not exactly sure. Here is my outputs.conf from the forwarder:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.10.50.49:9997

[tcpout-server://10.10.50.49:9997]

If someone can tell me what I'm doing wrong or how I can resolve this issue, I would really appreciate it.

I'm almost close to giving up if there's no concrete answer on this. I'd like to at least know what else I can do from here.

Thanks,

Re: How to activate forwarder server?

skoelpin — Tue, 21 Mar 2017 02:14:15 GMT

On the server with the universal forwarder, go to /opt/splunk/var/log/splunk/splunkd.log and see if there's any errors.

Can you also post the contents of your inputs.conf?

Re: How to activate forwarder server?

jhl226116 — Tue, 29 Sep 2020 13:18:53 GMT

There were around 2468 lines of logs in the forawrder. This is the first time I checked as I only just came to know where to check the logs. I'm just pasting last page of the logs FYI.

root@forwarder#nano /opt/splunkforwarder/var/log/splunk/splunkd.log

03-21-2017 13:37:17.842 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:37:17.842 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:37:47.689 +1100 ERROR TcpOutputProc - Can't find or illegal IP address or Name: SPsvr
03-21-2017 13:37:47.690 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:37:47.690 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:38:13.058 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 8800 seconds.
03-21-2017 13:38:17.553 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:38:17.553 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:38:47.401 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:38:47.401 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:39:17.247 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:39:17.247 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:39:47.093 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:39:47.093 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:39:53.068 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 8900 seconds.
03-21-2017 13:40:16.941 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:40:16.941 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:40:46.790 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:40:46.790 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:41:16.635 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:41:16.635 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:41:33.076 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9000 seconds.
03-21-2017 13:41:46.482 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:41:46.482 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:42:16.327 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:42:16.327 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:42:46.174 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:42:46.174 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:43:13.085 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9100 seconds.
03-21-2017 13:43:16.019 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:43:16.019 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:43:45.874 +1100 ERROR TcpOutputProc - Can't find or illegal IP address or Name: SPsvr
03-21-2017 13:43:45.874 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:43:45.874 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:44:15.721 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:44:15.721 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:44:45.570 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:44:45.570 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:44:53.094 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9200 seconds.
03-21-2017 13:45:15.456 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:45:15.456 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:45:45.305 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:45:45.305 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:46:15.154 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:46:15.154 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:46:33.110 +1100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9300 seconds.
03-21-2017 13:46:45.012 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:46:45.012 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 13:47:14.861 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 13:47:14.861 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed

Here's my inputs.conf from the forwarder. Not sure if it's configured properly, I don't exactly know where to look.

Forwarder Inputs.Conf

[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup

[blacklist:$SPLUNK_HOME/etc/auth]

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME/etc/splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[batch://$SPLUNK_HOME/var/spool/splunk]
move_policy = sinkhole
crcSalt =

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
queue = stashparsing
sourcetype = stash_new
move_policy = sinkhole
crcSalt =

[fschange:$SPLUNK_HOME/etc]

poll every 10 minutes

pollPeriod = 600

generate audit events into the audit index, instead of fschange events

signedaudit=true
recurse=true
followLinks=false
hashMaxSize=-1
fullEvent=false
sendEventMaxSize=-1
filesPerDelay = 10
delayInMills = 100

[udp]
connection_host=ip

[tcp]
acceptFrom=*
connection_host=dns

[splunktcp]
route=has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
acceptFrom=*
connection_host=ip

[script]
interval = 60.0
start_by_shell = true

[SSL]
default cipher suites that splunk allows. Change this if you wish to increase the security
of SSL connections, or to lower it if you having trouble connecting to splunk.
cipherSuite = ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
allowSslRenegotiation = true
sslQuietShutdown = false

Allow only sslv3 and above connections
sslVersions = *,-ssl2

Re: How to activate forwarder server?

esix_splunk — Tue, 21 Mar 2017 07:30:10 GMT

You are getting a connection refused message to the indexer. This means its being blocked at the network level. There is a firewall somewhere blocking this. I'd recommend disabling firewalls on both hosts as a test, but you might also have a network level firewall blocking this.

Re: How to activate forwarder server?

skoelpin — Tue, 21 Mar 2017 13:40:01 GMT

So it looks like your being blocked by a firewall or there is nothing listening.. Did you enable your indexer to listen on port 9997? If not then go to Settings>Receiving and Listening and add 9997

Re: How to activate forwarder server?

jhl226116 — Tue, 21 Mar 2017 23:26:11 GMT

9997 was already added and enabled on the indexer.

Ubuntu firewalls on both hosts has already been disabled.

I have just created a new rule in Cisco ASA firewall in the network lab to allow necessary Splunk ports to communicate between the indexer and forwarder.

Ports allowed from any source any to any destination within my internal network range.

Ports allowed:
TCP 8000 - Spluk Web
TCP 8080 - Indexer to Indexer Replication
TCP 8088 - mgmt for myself only
TCP 8089 - mgmt
TCP 9997 - Indexing
UDP 514 - Syslog

Also ICMP, domain, http, https has always been enabled already.

Re: How to activate forwarder server?

jhl226116 — Tue, 21 Mar 2017 23:31:00 GMT

Ubuntu firewalls on both hosts has already been disabled.

After seeing your post, I created a new rule in Cisco ASA firewall in the network level to allow necessary Splunk ports to communicate between the indexer and forwarder.

Ports allowed from any source any to any destination within my internal network range.

Ports allowed:
TCP 8000 - Spluk Web
TCP 8080 - Indexer to Indexer Replication
TCP 8088 - mgmt for myself only
TCP 8089 - mgmt
TCP 9997 - Indexing
UDP 514 - Syslog

Also ICMP, domain, http, https has always been enabled already.

Even after creating a new firewall rule to allow any connections between Index and forward server, it still says forwards is inactive.

root@indexer: /opt/splunk/bin# ./splunk list forward-server
Active forwards:
SPsvr:9997
Configured but inactive forwards:

I have restarted splunk and forwarder but no changes.

I can't completely shut off Cisco ASA down because there's other live traffics running on different ports in different network ranges. But this being at a network level and which I have just created a new rule for splunk ports specifically, I'm pretty sure ASA isn't the issue here.

Re: How to activate forwarder server?

MuS — Wed, 22 Mar 2017 01:02:10 GMT

Hi jhl226116,

this looks wrong:

root@indexer: /opt/splunk/bin# ./splunk list forward-server
Active forwards: 
SPsvr:9997
Configured but inactive forwards:

This tells me you enabled forwarding on the indexer but not receiving. To enable receiving on the indexer run this command:

./splunk enable listen 9997 -auth <username>:<password>

And please remember to disable the forwarding on the indexer before you enable receiving, otherwise you could create a nasty data loop 😉

Hope this helps ...

cheers, MuS

Re: How to activate forwarder server?

jhl226116 — Tue, 29 Sep 2020 13:22:19 GMT

I was trying to drill down to where the connection started failing and spotted below error message in the forwarder logs. 03-21-2017 09:07:13.538 +1100 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This pu$

I have no clue what's going on now, it's driving me nuts. I just wanna give up at this point......

nano /opt/splunkforwarder/var/log/splunk/splunkd.log03-21-2017 09:07:13.378 +1100 INFO ChunkedLBProcessor - Initializing the chunked line breaking processor
03-21-2017 09:07:13.378 +1100 INFO TcpOutputProc - Initializing with fwdtype=lwf
03-21-2017 09:07:13.388 +1100 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
03-21-2017 09:07:13.388 +1100 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
03-21-2017 09:07:13.388 +1100 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry)
03-21-2017 09:07:13.389 +1100 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to 10.10.50.49:9997
03-21-2017 09:07:13.389 +1100 INFO TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
03-21-2017 09:07:13.389 +1100 INFO TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
03-21-2017 09:07:13.390 +1100 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
03-21-2017 09:07:13.390 +1100 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
03-21-2017 09:07:13.390 +1100 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
03-21-2017 09:07:13.465 +1100 INFO PipelineComponent - Launching the pipelines for set 0.
03-21-2017 09:07:13.534 +1100 INFO TailingProcessor - TailWatcher initializing...
03-21-2017 09:07:13.534 +1100 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
03-21-2017 09:07:13.534 +1100 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
03-21-2017 09:07:13.535 +1100 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
03-21-2017 09:07:13.535 +1100 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk.
03-21-2017 09:07:13.535 +1100 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
03-21-2017 09:07:13.535 +1100 INFO TailReader - Registering metrics callback for: tailreader0
03-21-2017 09:07:13.535 +1100 INFO TailReader - Starting tailreader0 thread
03-21-2017 09:07:13.537 +1100 INFO loader - Limiting REST HTTP server to 21333 sockets
03-21-2017 09:07:13.537 +1100 INFO loader - Limiting REST HTTP server to 658 threads
03-21-2017 09:07:13.538 +1100 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This pu$
03-21-2017 09:07:13.538 +1100 INFO TailReader - Registering metrics callback for: batchreader0
03-21-2017 09:07:13.538 +1100 INFO TailReader - Starting batchreader0 thread
03-21-2017 09:07:13.539 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:07:13.539 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:07:13.544 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
03-21-2017 09:07:13.551 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
03-21-2017 09:07:13.553 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
03-21-2017 09:07:13.556 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
03-21-2017 09:07:13.558 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
03-21-2017 09:07:13.561 +1100 INFO WatchedFile - Will begin reading at offset=60531 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
03-21-2017 09:07:13.565 +1100 INFO WatchedFile - Will begin reading at offset=123 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
03-21-2017 09:07:13.568 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
03-21-2017 09:07:13.576 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
03-21-2017 09:07:13.599 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
03-21-2017 09:07:13.602 +1100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
03-21-2017 09:07:13.610 +1100 INFO WatchedFile - Will begin reading at offset=405521 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
03-21-2017 09:07:43.225 +1100 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
03-21-2017 09:07:43.227 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:07:43.227 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:08:13.073 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:08:13.074 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:08:42.951 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:08:42.952 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:09:12.805 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:09:12.805 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:09:42.662 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:09:42.663 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:10:12.513 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:10:12.513 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:10:42.371 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:10:42.371 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:11:12.223 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused
03-21-2017 09:11:12.224 +1100 ERROR TcpOutputFd - Connection to host=10.10.50.49:9997 failed
03-21-2017 09:11:42.082 +1100 WARN TcpOutputFd - Connect to 10.10.50.49:9997 failed. Connection refused

Re: How to activate forwarder server?

jhl226116 — Wed, 22 Mar 2017 02:10:45 GMT

It says Failed to create because Configuration for port 9997 already exists. Forwarding is already disabled on the indexer.

root@SPsvr:/opt/splunk/bin# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
SPsvr:9997

root@SPsvr:/opt/splunk/bin# sudo ufw disable
Firewall stopped and disabled on system startup

root@SPsvr:/opt/splunk/bin# ./splunk enable listen 9997 -auth admin:xxxxxxxx
Failed to create. Configuration for port 9997 already exists.

Re: How to activate forwarder server?

MuS — Wed, 22 Mar 2017 02:44:30 GMT

On the indexer run splunk btool outputs list --debug | grep -v default, see what custom outputs.conf you have and remove it on the indexer.

Then run splunk btool inputs list splunktcp --debug | grep -v default, check if everything is correct and also run splunk list inputstatus and check for tcp_cooked:listenerports which should be 9997.

Restart Splunk and it should work

Re: How to activate forwarder server?

jhl226116 — Wed, 22 Mar 2017 03:16:25 GMT

Commands are not working. Is there a typo somewhere?

root@SPsvr:/opt/splunk/bin# splunk btool outputs list --debug | grep -v default
splunk: command not found

root@SPsvr:~# splunk btool outputs list --debug | grep -v default
splunk: command not found

root@SPsvr:/opt/splunk/bin# splunk btool inputs list splunktcp --debug | grep -v default
splunk: command not found

root@SPsvr:/opt/splunk/bin# splunk list inputstatus
splunk: command not found

root@SPsvr:~# splunk list input status
splunk: command not found

Re: How to activate forwarder server?

skoelpin — Wed, 22 Mar 2017 16:55:22 GMT

Can you do a telnet from your forwarder to your indexer to verify you can connect?

telnet IndexerIP 9997

Re: How to activate forwarder server?

jhl226116 — Tue, 29 Sep 2020 13:22:50 GMT

I was unable to telnet from forwarder to the indexer. Only can ping vice-versa. Where does this give you indication?

root@forwarder:~# telnet 10.10.50.49
Trying 10.10.50.49...
telnet: unable to connect to remote host: Connection refused

root@forwarder:~# ping 10.10.50.49
PING 10.10.50.49 (10.10.50.49) 56(84) bytes of data.
64 bytes from 10.10.50.49: icmp_seq=1 ttl=64 time=0.213 ms
64 bytes from 10.10.50.49: icmp_seq=2 ttl=64 time=0.227 ms
64 bytes from 10.10.50.49: icmp_seq=3 ttl=64 time=0.233 ms
64 bytes from 10.10.50.49: icmp_seq=4 ttl=64 time=0.259 ms
64 bytes from 10.10.50.49: icmp_seq=5 ttl=64 time=0.207 ms
64 bytes from 10.10.50.49: icmp_seq=6 ttl=64 time=0.259 ms
64 bytes from 10.10.50.49: icmp_seq=7 ttl=64 time=0.210 ms
64 bytes from 10.10.50.49: icmp_seq=8 ttl=64 time=0.253 ms

Re: How to activate forwarder server?

MuS — Wed, 22 Mar 2017 20:58:37 GMT

use ./splunk instead of splunk

Re: How to activate forwarder server?

jhl226116 — Wed, 22 Mar 2017 21:05:16 GMT

Awesome, I can run the commands. See results below:

root@SPsvr:/opt/splunk/bin# ./splunk btool outputs list --debug | grep -v default
/opt/splunk/etc/system/local/outputs.conf [tcpout]
/opt/splunk/etc/system/local/outputs.conf [tcpout-server://SPsvr:9997]
/opt/splunk/etc/system/local/outputs.conf server = SPsvr:9997

root@SPsvr:/opt/splunk/bin# ./splunk btool inputs list splunktcp --debug | grep -v default
/opt/splunk/etc/system/local/inputs.conf host = csoc
/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
/opt/splunk/etc/apps/search/local/inputs.conf connection_host = ip
/opt/splunk/etc/system/local/inputs.conf host = csoc

root@SPsvr:/opt/splunk/bin# ./splunk list inputstatus
Cooked:tcp :
9997:127.0.0.1:8089
time opened = 2017-03-22T08:22:38+1100

tcp

ExecProcessor:exec commands :
./bin/collector.path
time opened = 2017-03-22T08:22:45+1100

./bin/dmc_config.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:50+1100
    time opened = 2017-03-22T08:22:50+1100

./bin/instrumentation.py
    exit status description = exited with code 0
    time closed = 2017-03-23T03:06:00+1100
    time opened = 2017-03-23T03:05:00+1100
    total bytes = 305

./bin/scripted_inputs/dependency_manager.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:48+1100
    time opened = 2017-03-22T08:22:48+1100

./bin/scripted_inputs/deploy_splunk_ta_paloalto.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:43+1100
    time opened = 2017-03-22T08:22:43+1100

./bin/scripted_inputs/ftr_lookups.py
    exit status description = exited with code 0
    time closed = 2017-03-22T08:22:40+1100
    time opened = 2017-03-22T08:22:40+1100

./bin/scripted_inputs/update_hosts.py
    exit status description = exited with code 0
    time closed = 2017-03-23T00:00:00+1100
    time opened = 2017-03-23T00:00:00+1100

Raw:tcp :
tcp

TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/splunk.version
file position = 70
file size = 70
percent = 100.00
type = finished reading

$SPLUNK_HOME/var/log/introspection
    type = directory

$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
    type = directory

$SPLUNK_HOME/var/spool/splunk/...stash_new
    type = directory

/opt/splunk/var/log/introspection/kvstore.log.1
    file position = 10616832
    file size = 25005470
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 42.46
    type = reading (batch)

/opt/splunk/var/log/introspection/kvstore.log.2
    file position = 24970463
    file size = 25006411
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 99.86
    type = open file

/opt/splunk/var/log/introspection/kvstore.log.4
    file position = 0
    file size = 25004216
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 0.00
    type = batch processing(toRead=25004216)

tcp_cooked:listenerports :
9997

UDP:listenerports :
514

Re: How to activate forwarder server?

skoelpin — Wed, 22 Mar 2017 23:14:11 GMT

This means your port is blocked which is the reason why data is not being sent via 9997 and also explains the connection refused message in your splunkd logs.

You need to go take another look at your firewall settings and test its actually open by using telnet

Re: How to activate forwarder server?

skoelpin — Wed, 22 Mar 2017 23:18:10 GMT

Don't give up.. You narrowed down the problem and have a quick and easy way to test.

Re: How to activate forwarder server?

jhl226116 — Thu, 23 Mar 2017 00:18:28 GMT

I suppose you were right it seems telnet wasn't enabled somewhere but now I have enabled it and am now able to telnet to the Indexer from the forwarder.

root@SPsvr:/opt/splunk/bin# nc -v localhost 23
Connection to localhost 23 port [tcp/telnet] succeeded!

root@SPsvr:/opt/splunk/bin# netstat -nat | grep 23
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:48838 127.0.0.1:23 ESTABLISHED
tcp 0 0 10.10.50.49:49508 74.125.23.189:443 ESTABLISHED
tcp 0 0 10.10.50.49:50208 74.125.23.189:443 ESTABLISHED
tcp 0 0 127.0.0.1:23 127.0.0.1:48838 ESTABLISHED

root@SPsvr:/opt/splunk/bin# sudo netstat -tanpu | grep ":23"
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 7270/inetd
tcp 0 0 127.0.0.1:48838 127.0.0.1:23 ESTABLISHED 9735/nc
tcp 0 0 127.0.0.1:23 127.0.0.1:48838 ESTABLISHED 9736/in.telnetd

root@SPsvr:/opt/splunk/bin# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN 968/dnsmasq
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 7270/inetd
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 25816/splunkd
tcp 0 0 0.0.0.0:8191 0.0.0.0:* LISTEN 25826/mongod
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 25816/splunkd
tcp 0 0 127.0.0.1:8065 0.0.0.0:* LISTEN 25939/python
udp 0 0 0.0.0.0:5353 0.0.0.0:* 2458/chrome
udp 0 0 0.0.0.0:5353 0.0.0.0:* 2458/chrome
udp 0 0 0.0.0.0:5353 0.0.0.0:* 589/avahi-daemon: r
udp 0 0 0.0.0.0:38843 0.0.0.0:* 589/avahi-daemon: r
udp 0 0 127.0.1.1:53 0.0.0.0:* 968/dnsmasq
udp 0 0 0.0.0.0:68 0.0.0.0:* 949/dhclient
udp 0 0 0.0.0.0:514 0.0.0.0:* 25816/splunkd
udp 0 0 0.0.0.0:631 0.0.0.0:* 30478/cups-browsed
udp6 0 0 :::60258 :::* 589/avahi-daemon: r
udp6 0 0 :::5353 :::* 2458/chrome
udp6 0 0 :::5353 :::* 589/avahi-daemon: r

However I can't get past the credentials when telnetting from the forwarder (10.10.50.18) to the indexer (10.10.50.49).
I used the same credentials for everything so there's no doubt my passwords are corrrect but I'm unable to telnet into the indexer.
Previously telnet to indexer was refusing connection but now I'm one step closer. Can't stop now. Trying to figure out what the password is and how to get past this point now as it keeps saying login is incorrect.

root@forwarder:~# telnet 10.10.50.49
Trying 10.10.50.49...
Connected to 10.10.50.49.
Escape character is '^]'.
Ubuntu 16.04.1 LTS
SPsvr login: admin
Password:

Re: How to activate forwarder server?

jhl226116 — Thu, 23 Mar 2017 00:24:28 GMT

Thanks for encouraging me, I have narrowed down the problem and getting closer to resolving it since I got the telnet session working now from forwarder to the indexer.

Continuing to research for answers and solutions..