Unable to configure and install certificates in Splunk Enterprise for Splunk Log Observer Connect

[rahusri2]

Hello,

I am following document: https://docs.splunk.com/Documentation/Splunk/9.4.0/Security/ConfigureandinstallcertificatesforLogObs... to configure and install certificates in Splunk Enterprise for Splunk Log Observer Connect but getting some error mentioned below.

I have generated myFinalCert.pem as per the document mentioned above, below are the server.conf and web.conf configuration.

# cat ../etc/system/local/server.conf
[general]
serverName = ip-xxxx.us-west-2.compute.internal
pass4SymmKey = $7$IHXMpPIvtTGnxEusRYk62AjBIizAQosZq0YXtUg==

[sslConfig]
serverCert = /opt/splunk/etc/auth/sloccerts/myFinalCert.pem
requireClientCert = false
sslPassword = $7$vboieDG2v4YFg8FbYxW8jDji6woyDylOKWLe8Ow==

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
peers = *
quota = MAX
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
peers = *
quota = MAX
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
peers = *
quota = MAX
stack_id = free

# cat ../etc/system/local/web.conf
[expose:tlPackage-scimGroup]
methods = GET
pattern = /identity/provisioning/v1/scim/v2/Groups/*

[expose:tlPackage-scimGroups]
methods = GET
pattern = /identity/provisioning/v1/scim/v2/Groups

[expose:tlPackage-scimUser]
methods = GET,PUT,PATCH,DELETE
pattern = /identity/provisioning/v1/scim/v2/Users/*

[expose:tlPackage-scimUsers]
methods = GET
pattern = /identity/provisioning/v1/scim/v2/Users

[settings]
enableSplunkWebSSL = true
serverCert = /opt/splunk/etc/auth/sloccerts/myFinalCert.pem
#

After making changes to server.conf, I am able to restart the splunkd service but after making changes to the web.conf, restarting the splunkd service gets stuck, below are logs related to it:

# ./splunk restart
splunkd is not running.                                    [FAILED]

Splunk> The IT Search Engine.

Checking prerequisites...
	Checking http port [8000]: open
	Checking mgmt port [8089]: open
	Checking appserver port [127.0.0.1:8065]: open
	Checking kvstore port [8191]: open
	Checking configuration... Done.
	Checking critical directories...	Done
	Checking indexes...
		Validated: _audit _configtracker _dsappevent _dsclient _dsphonehome _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main sim_metrics statsd_udp_8125_5_dec summary
	Done
	Checking filesystem compatibility...  Done
	Checking conf files for problems...
	Done
	Checking default conf files for edits...
	Validating installed files against hashes from '/opt/splunk/splunk-9.3.2-d8bb32809498-linux-2.6-x86_64-manifest'
	All installed files intact.
	Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done
                                                           [  OK  ]

Waiting for web server at https://127.0.0.1:8000 to be available...............................WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Please let me know if I am missing some thing.

Thanks

[PaulPanther]

The documentation is not correct.

You have to create two separate certificate files because the Splunk Web certificate must not contain the private key.

web certificate format:

-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the intermediate certificate)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the root certificate for the CA)...
-----END CERTIFICATE-----

server certificate format:

-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
...<Server Private Key – Passphrase protected>
 -----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the intermediate certificate)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the certificate authority certificate)...
-----END CERTIFICATE-----

Check out:

Configure and install certificates in Splunk Enterprise for Splunk Log Observer Connect - Splunk Doc...

Final configuration must look like:

web.conf

[settings]
enableSplunkWebSSL = true
privKeyPath = /opt/splunk/etc/auth/mycerts/mySplunkWebPrivateKey.key 
serverCert = /opt/splunk/etc/auth/mycerts/mySplunkWebCertificate.pem
sslPassword = <priv_key_passwd>

server.conf

[sslConfig]
serverCert = /opt/splunk/etc/auth/sloccerts/myFinalCert.pem
requireClientCert = false
sslPassword = <priv_key_passwd>

[PickleRick]

Yes. Web interface is the only "standard" (not including any unpredictable things done by add-on developers) component which behaves differently.

While all other "areas of activity" (inputs, outputs, inter-splunkd connections) require certs in a single-file form (from the top - subject cert, private key, certificate chain), web interface requires two separate files - one with the private key and another with the chained subject certificate.

And TLS-protecting your web interface while desired as a general rule has nothing to do with inputs and outputs.