Hello everyone. I have written an SSL input app using a 3rd party certificate that will be used to deploy on all of our indexers. It contains an inputs.conf file, a pem file (for the indexer, containing the indexer cert, the key, and the rootCA cert), and the rootCA pem file. The inputs file has the correct password, I've verified numerous times. I have also checked the MD5 sum hash of the key, the indexer pem, and the certificate request. They're all correct. The cert should work.
When restarting splunk, I receive these messages in splunkd.log:
10-10-2013 13:30:11.090 +0000 ERROR TcpInputConfig - SSL server certificate not found, or password is wrong - SSL ports will not be opened
10-10-2013 13:30:11.090 +0000 ERROR TcpInputConfig - SSL context not found. Will not open splunk 2 splunk (SSL) IPv4 port 9997
When examining the inputs.conf file after restarting splunk, it does not appear that the password has been hashed. When we had originally done this configuration in $SPLUNK_HOME, after restarting splunk, the password would have been hashed. Is there any reason why this would be? At this point I'm just trying to figure out how to make this work.
Ultimately I found the issue - when the app was originally deployed, splunk copied a hashed version of the password to $SPLUNK_HOME/etc/system/local/inputs.conf in an SSL stanza. When the new app was deployed, apparently the hash for the password changed, but splunk never updates the configurationin /etc/system/local, so it was trying to use an old password against the certificate. Clearing out that SSL stanza fixed the issue - splunk recreated it with the correct password hash from the app.
Ultimately I found the issue - when the app was originally deployed, splunk copied a hashed version of the password to $SPLUNK_HOME/etc/system/local/inputs.conf in an SSL stanza. When the new app was deployed, apparently the hash for the password changed, but splunk never updates the configurationin /etc/system/local, so it was trying to use an old password against the certificate. Clearing out that SSL stanza fixed the issue - splunk recreated it with the correct password hash from the app.