Not able to get Correct Cert working on SH

[Kozanic]

We have generated an SSL Cert using internal CA server, configured to work for a number of our servers including 3 SHs.

We have created an App that pushes out web.conf file with stanza for the following items:
[settings]
privKeyPath = etc/auth/healthCerts/HealthSearcheadPrivateKey.key
serverCert = etc/auth/healthCerts/searchheadcertcombined.pem
sslVersions = tls1.2

I have confirmed that correct files are available and splunk user has access to the files, I have confirmed in btool that the above settings are in affect, yet on one of our servers, it is still using the default self-signed Cert for some reason.

The above works perfectly on the other 2 SHs, just one that it doesn't.

Have checked /etc/system/local - but there are no entries for web.conf, only in default.

I have restarted the Splunk service on the SH a number of times - but still using the default cert.

Not sure what I'm missing or what else I can check - but appreciate any suggestions people might have.

[ptang_splunk]

btool never lies and this usually means either:

• the file is not accessible by the Splunk user
• web.conf not in the right location - $SPLUNK_HOME/etc/apps//[default|local]/web.conf
• configuration file precedence - http://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/Wheretofindtheconfigurationfiles
• perhaps check file md5 checksum - maybe corrupted file?

What path does it point to when using:
splunk cmd btool web list settings --debug

[Kozanic]

Hi ptang

Running btool gives me the following outputs (only included those relevant):

/opt/splunk/etc/apps/config_SH_webconf/local/web.conf             privKeyPath = etc/auth/healthCerts/HealthSearcheadPrivateKey.key
/opt/splunk/etc/apps/config_SH_webconf/local/web.conf             serverCert = etc/auth/healthCerts/searchheadcertcombined.pem
/opt/splunk/etc/apps/config_SH_webconf/local/web.conf             sslVersions = tls1.2

Path and directory listing below match the above output:

-bash-4.2$ ls -la /opt/splunk/etc/auth/healthCerts/
total 44
drwxr-xr-x. 2 splunk splunk 4096 Apr 24 11:15 .
drwx------. 8 splunk splunk 4096 May 28 12:05 ..
-rw-r--r--. 1 splunk splunk 1704 Apr 24 11:15 HealthSearcheadPrivateKey.key
-rw-r--r--. 1 splunk splunk 6261 Apr 24 11:15 searchheadcertcombined.pem
-rw-r--r--. 1 splunk splunk 2894 Apr 24 11:15 searchheadcert.pem
-rw-r--r--. 1 splunk splunk 631 Apr 24 11:15 splunkCertConfig.conf
-rw-r--r--. 1 splunk splunk 1435 Apr 24 11:15 splunksec.csr
-rw-r--r--. 1 splunk splunk 8843 Apr 24 11:15 splunkweb.pem

MuS, Thanks for the extra info - I agree with your thought on btool so ran your command as well - just to compare:

relevant entries:

privKeyPath=etc/auth/healthCerts/HealthSearcheadPrivateKey.key
serverCert=etc/auth/healthCerts/searchheadcertcombined.pem
sslVersions=tls1.2

From this - I can only assume that things are configured correctly - yet, it's not using this cert.
Any other thoughts on why not?

[MuS]

Sorry, but this is not entirely correct. See the docs on btool http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurati...

Btool displays merged on-disk configurations. That is, btool shows you the merged settings in the .conf files. It does not necessarily show you what Splunk software is currently using. So for example if you edit a .conf file and do not restart (and the edit requires a restart), btool reports the newly edited settings rather than the settings that are currently being used. To view current in-memory configurations, query the REST endpoint /services/properties/.

or use this command:

 splunk show config web

cheers, MuS

[xpac]

Did you check splunkd.log for erorrs/warnings, or infos about cert stuff?

[Kozanic]

Hi xpac,

Yes - I did check splunkd logs for both warnings and errors - nothing obvious.

Have also tried looking for cert, privatekey and the cert name - nothing comes up suggesting errors.