Archive

How to change splunkforwarder status from Configured but inactive, to Active?

gregcoats
Explorer

I established splunk on Solaris server indexerhost, and splunk successfully searches events on indexerhost. Then, I established splunkforwarder on Solaris server forwarderhost, directing it to forward to indexerhost. Unfortunately, splunkforwarding from forwarderhost to indexerhost is not taking place, and so a splunk search of events on forwarderhost yields zero results.
How to determine if splunk on indexerhost is listening and receiving (or not) the info from the splunkforwarder on forwarderhost?

How to change splunkforwarder status from Configured but inactive, to Active?

root@forwarderhost# ${SPLUNK_HOME}/bin/splunk list forward-server

Active forwards:

    None

Configured but inactive forwards:

    indexerhost.domain.com:9997 (ssl)
Tags (1)

biofool
New Member

I also found the only way I could log into my server was
/opt/splunkforwarder/bin/./splunk list forward-server

When I tried to just add the forward server without listing first I could not login.

0 Karma

gregcoats
Explorer

I reported this to Splunk Support at https://www.splunk.com/page/submit_issue as Case 107515. Seth Garvin's response was very helpful. For reference, I am working with Splunk forwarder version 5.0.1, build 143156, for Solaris 10, SPARC, from the file splunkforwarder-5.0.1-143156-SunOS-sparc.tar.Z.

There is a bug in Splunk 5.0.1, such that in the file /opt/splunkforwarder/etc/system/local/server.conf when the sslKeysfilePassword is set to the encrypted version of the special string "password", it is misread and incorrectly set. The successful work around was to edit the file /opt/splunkforwarder/etc/system/local/server.conf, and change the sslKeysfilePassword to the unencrypted string "password". Then do /opt/splunkforwarder/bin/splunk restart. After the restart, I notice that the order of the values in server.conf had changed, and the value for sslKeysfilePassword is again displayed encrypted.

An additional problem is that, because of another bug in Splunk 5.0.1, the results of the output of /opt/splunkforwarder/bin/splunk list forward-server can be wrong. As a consequence, the Splunk supplied gauge normally used to show the status of splunk forwarder systems displays erroneous info. Seth Garvin identifies this issue as known bug SPL-55793.
After the Splunk restart, /opt/splunkforwarder/bin/splunk list forward-server continues to report that the forwarder is Configured but inactive. This is false. Going to the Splunk indexer, and doing a search for the just restarted Splunk forwarder using host=forwarderhost yielded more than 1,000 events for the past 24 hours.

As a diagnostic of the failure to communicate via SSL from Splunk forwarder to Splunk indexer, for me, an indication that SSL is working on the Splunk indexer is the presence of this message in the Splunk log.

root@indexerhost# grep "port 9997 is reserved" /opt/splunk/var/log/splunk/splunkd.log | tail -1

01-07-2013 13:06:25.325 -0500 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)

An indication that SSL is NOT working on the Splunk forwarder is the presence of these messages in the Splunk log.

root@forwarderhost# grep "SSL" /opt/splunkforwarder/var/log/splunk/splunkd.log

01-07-2013 13:10:32.279 -0500 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP DecryptFinal_ex:bad decrypt.

01-07-2013 13:10:32.279 -0500 ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server indexerhost.domain.com:9997

01-07-2013 13:10:32.282 -0500 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available

kadishmal
Engager

Using 6.0.2, but this answer didn't help.

0 Karma