https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77454#M8896 <P>I also found the only way I could log into my server was<BR /> /opt/splunkforwarder/bin/./splunk list forward-server</P> <P>When I tried to just add the forward server without listing first I could not login.</P> Fri, 11 Jan 2013 08:20:30 GMT biofool 2013-01-11T08:20:30Z https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77452#M8894 <P>I established splunk on Solaris server indexerhost, and splunk successfully searches events on indexerhost. Then, I established splunkforwarder on Solaris server forwarderhost, directing it to forward to indexerhost. Unfortunately, splunkforwarding from forwarderhost to indexerhost is not taking place, and so a splunk search of events on forwarderhost yields zero results.<BR /> How to determine if splunk on indexerhost is listening and receiving (or not) the info from the splunkforwarder on forwarderhost?</P> <P>How to change splunkforwarder status from Configured but inactive, to Active?</P> <P>root@forwarderhost# ${SPLUNK_HOME}/bin/splunk list forward-server</P> <P>Active forwards:</P> <PRE><CODE> None </CODE></PRE> <P>Configured but inactive forwards:</P> <PRE><CODE> indexerhost.domain.com:9997 (ssl) </CODE></PRE> Thu, 03 Jan 2013 20:08:11 GMT https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77452#M8894 gregcoats 2013-01-03T20:08:11Z https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77453#M8895 <P>I reported this to Splunk Support at <A href="https://www.splunk.com/page/submit_issue">https://www.splunk.com/page/submit_issue</A> as Case 107515. Seth Garvin's response was very helpful. For reference, I am working with Splunk forwarder version 5.0.1, build 143156, for Solaris 10, SPARC, from the file splunkforwarder-5.0.1-143156-SunOS-sparc.tar.Z.</P> <P>There is a bug in Splunk 5.0.1, such that in the file /opt/splunkforwarder/etc/system/local/server.conf when the sslKeysfilePassword is set to the encrypted version of the special string "password", it is misread and incorrectly set. The successful work around was to edit the file /opt/splunkforwarder/etc/system/local/server.conf, and change the sslKeysfilePassword to the unencrypted string "password". Then do /opt/splunkforwarder/bin/splunk restart. After the restart, I notice that the order of the values in server.conf had changed, and the value for sslKeysfilePassword is again displayed encrypted.</P> <P>An additional problem is that, because of another bug in Splunk 5.0.1, the results of the output of /opt/splunkforwarder/bin/splunk list forward-server can be wrong. As a consequence, the Splunk supplied gauge normally used to show the status of splunk forwarder systems displays erroneous info. Seth Garvin identifies this issue as known bug SPL-55793.<BR /> After the Splunk restart, /opt/splunkforwarder/bin/splunk list forward-server continues to report that the forwarder is Configured but inactive. This is false. Going to the Splunk indexer, and doing a search for the just restarted Splunk forwarder using host=forwarderhost yielded more than 1,000 events for the past 24 hours.</P> <P>As a diagnostic of the failure to communicate via SSL from Splunk forwarder to Splunk indexer, for me, an indication that SSL is working on the Splunk indexer is the presence of this message in the Splunk log.</P> <P>root@indexerhost# grep "port 9997 is reserved" /opt/splunk/var/log/splunk/splunkd.log | tail -1</P> <P>01-07-2013 13:06:25.325 -0500 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)</P> <P>An indication that SSL is NOT working on the Splunk forwarder is the presence of these messages in the Splunk log.</P> <P>root@forwarderhost# grep "SSL" /opt/splunkforwarder/var/log/splunk/splunkd.log</P> <P>01-07-2013 13:10:32.279 -0500 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP DecryptFinal_ex:bad decrypt.</P> <P>01-07-2013 13:10:32.279 -0500 ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server indexerhost.domain.com:9997</P> <P>01-07-2013 13:10:32.282 -0500 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available</P> Thu, 10 Jan 2013 20:38:03 GMT https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77453#M8895 gregcoats 2013-01-10T20:38:03Z https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77454#M8896 <P>I also found the only way I could log into my server was<BR /> /opt/splunkforwarder/bin/./splunk list forward-server</P> <P>When I tried to just add the forward server without listing first I could not login.</P> Fri, 11 Jan 2013 08:20:30 GMT https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77454#M8896 biofool 2013-01-11T08:20:30Z https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77455#M8897 <P>Using 6.0.2, but this answer didn't help.</P> Tue, 11 Mar 2014 08:58:06 GMT https://community.splunk.com/t5/Splunk-Enterprise/How-to-change-splunkforwarder-status-from-Configured-but/m-p/77455#M8897 kadishmal 2014-03-11T08:58:06Z