Overview
Creating a TLS cert for Splunk usage is pretty much standard openssl. To make life better, use an openssl.conf file. Since I have to look up the syntax and waste a few hours every time I generate a certificate for figuring out how to generate one with a Subject Alternative Name (SAN) correctly, it's documented here.
Splunk Trust now recommends creating certs inside an app in the certs directory of the app, e.g. $SPLUNK_HOME/etc/apps/org_base/certs.
Remember, if openssl complains about library issues, as user splunk, use splunk cmd openssl ... instead of vanilla openssl ... and it will work.
See https://hurricanelabs.com/splunk-tutorials/splunk-certificates-master-guide/#:~:text=The%20same%20certificate%20material%20can%20(and%20should)%20be,using%20the%20same%20allows%20for%20ease%20of%20management. for additional information on this procedure
See https://help.splunk.com/en/splunk-enterprise/administer/manage-users-and-security/9.4/secure-splunk-platform-communications-with-transport-layer-security-certificates/configure-tls-certificate-host-name-validation-for-secured-connections-between-splunk-software-components for a lot of key details, bad settings can break even starting Splunk.
Procedure
When renewing, you will be issued a PEM and already have the key. Go directly to the section below that applies, do not recreate your key and CSR.
For purposes of continuity/sanity, we are using filenames for the cert and key of the hostname ending in .pem. For example, for host clpv003, use the filename clpv003.pem for the certificate and clpv003.key for the key.
Generate the certificate
There's several steps to this process.
openssl.conf
Create or update the openssl.conf file. It is a Good Idea® to store this file in the below referenced /opt/splunk/etc/appname/certs directory.
openssl.conf
[req] distinguished_name = req_distinguished_name req_extensions = v3_req [req_distinguished_name] countryName = Country Name (2 letter code) countryName_default = US stateOrProvinceName = State or Province Name (full name) stateOrProvinceName_default = CA localityName = Locality Name (eg, city) localityName_default = MyCity organizationalUnitName = Organizational Unit Name (eg, section) organizationalUnitName_default = ORG commonName = Common Name (eg, YOUR name) commonName_max = 64 commonName_default = splunk-test.example.com [ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = clpv003.example.com DNS.2 = splunk-dev.example.com
For the commonName_default use the CNAME pointing to the application alias for the role the host in question will serve. For DNS.1, use the FQDN of the host itself. For DNS.2, put in the CNAME for the application role again. It is an expectation and requirement of newer browsers that if a SAN is used, that it takes priority over the subject, so the subject must be listed as one of the SAN entries.
⚠️We may also need the short host name as an alternative name in the SAN for some use cases.
Make sure to update the openssl.conf as above, or it will not be useful.
While RFC 5280 section 4.2.1.6 explicitly states a Subject Alternative Name (SAN) may be a DNS (FQDN) entry, an IP, a RFC822 address, etc., some CAs do not permit anything but FQDNs. Also, while contrary to the RFC, some CAs expect the subject field to be filled out, even though if the SAN is filled out, the Subject is optional per the RFC.
Create the CSR and key
Now, generate the cert almost as expected. For documentation, we've placed the openssl.conf at /opt/splunk/etc/app/ORG_base/certs/openssl.conf. Change the app name to the base app for the Splunk instance type.
If certificates or keys already exist, alter the below script to ensure unique filenames. For purposes of this code snippet, we are working on the app named ORG_base and that app is to be deployed by the search deployer (SD) to the search head cluster. What's important is that the cert work is all done inside an app.
As splunk
install -d /opt/splunk/etc/apps/ORG_base/certs cd /opt/splunk/etc/apps/ORG_base/certs /opt/splunk/bin/splunk cmd openssl genrsa -out \ $(hostname).key 2048 ln -s $(hostname).key privkey.key /opt/splunk/bin/splunk cmd openssl req -new -out \ $(hostname).csr -key $(hostname).key -config openssl.conf
The defaults should be appropriate but check them anyway. (Note, you may soon need to use a different algorithm as RSA is being deprecated in favor of post-quantum cryptographic algorithms, but those require newer operating system levels than you may have.)
Validate the request
As splunk
/opt/splunk/bin/splunk cmd openssl req \ -in $(hostname).csr -text -noout
Read the result and ensure the DNS entries are in there to create the Subject Alternative Name entries for the request. Don't forget to check for the Subj of the CSR. You want to find in it a CN entry with the FQDN.
There is a technical reason why both the FQDN and the CNAME are listed as SANs. Don't forget to list both.
Now, collect the CSR and send that to request the certificate by attaching to a ticket to your certificate management group. The process outlined in “Renewal” in this document also validates and installs the resulting certificate.
Create Key for SAML
SAML has its own oddities. To set that up right, you want to create a unique SAML server.pem. Why? Because IAM teams don’t want to rework the SAML setup every time your Splunk regular certificate expires.
As splunk
splunk cmd openssl x509 -req -days 1825 \ -in $(hostname).csr -signkey privkey.key \ -out saml.crt -extfile openssl.conf \ -extensions v3_req cat privkey.key saml.crt > saml.pem
Provide the saml.crt as part of your Metadata to the IAM team setting up SAML for your instance.
Update the CA chain
To ensure update checks and Splunk Base checks work, append the appsCA.pem to the ca-chain.pem:
cd /opt/splunk/etc/apps/ORG_base/certs cat /opt/splunk/etc/auth/appsCA.pem >> ca-chain.pem
When working on the SHC, we are using a virtual hostname of "splunkshc" as a naming convention for purposes of documentation, and the directory is of course on the search head deployer (SD) at /opt/splunk/etc/shcluster/apps/ORG_base/certs
Make Splunk use the new certs
To tell Splunk to use the certificate, you must now update (or create) the appropriate configuration files in the server.conf or web.conf (or potentially other configuration files) appropriately. For this example, we are using a different base directory to emphasize the flexibility here
server.conf snippet
[sslConfig] sslRootCAPath = $SPLUNK_HOME/etc/apps/ORG_mc/certs/ca-chain.pem serverCert = $SPLUNK_HOME/etc/apps/ORG_mc/certs/server.pem cliVerifyServerName = true ...
A snippet only of the web.conf: (Only needed if Splunk Web is enabled, don't enable it if it isn't required)
web.conf
[settings] startwebserver = 1 enableSplunkWebSSL = true privKeyPath = $SPLUNK_HOME/etc/apps/ORG_mc/certs/privkey.key serverCert = $SPLUNK_HOME/etc/apps/ORG_mc/certs/$(hostname).pem ...
Finally, a snippet of the authentication.conf for systems using SAML auth:
authentication.conf
[saml] clientCert = /opt/splunk/etc/apps/ORG_mc/certs/saml.pem
Remember, SAML uses its own oddities in certificates, and you need to provide the crt file referenced above to the identity team. Also, after testing, authentication.conf doesn't honor $SPLUNK_HOME. You must fully qualify the path. Notice, it is a unique certificate for SAML. In this case, it’s self-signed.
The Search Head Cluster, the configurations are in $SPLUNK_HOME/etc/shcluster/apps/ORG_base with the certs in the certs subdirectory. They get pushed to all cluster members, so all work is done there, not on individual search heads.
Make sure to change the $(hostname) above to the listed filename you chose.
Remember to finish off with a btool check before applying changes (or a bundle validation).
Finally, restart splunk via splunk restart. Login after the restart is complete and examine the certificate to ensure it is taking the cert you intended.
Consider trying to replace the restart of Splunk with a series of REST calls referenced at https://help.splunk.com/en/splunk-enterprise/administer/manage-users-and-security/9.4/secure-splunk-platform-communications-with-transport-layer-security-certificates/renew-existing-tls-certificates
Renew a certificate
When you renew your certificates, start here.
When you get the certificate in pem form back, install it into the above directory (e.g., /opt/splunk/etc/apps/ORG_base/certs/$(hostname).pem). Validate the PEM is correct. Also ensure that the certificate chain (intermediate then root) is uploaded to ca-chain.pem in the same directory.
Validate the cert
Copy the received certificate back to the system (e.g. $(hostname).pem) and check it.
As splunk
cd /opt/splunk/etc/apps/ORG_base/certs /opt/splunk/bin/splunk cmd openssl x509 \ -in $(hostname).pem -text -noout
Ensure that the certificate decodes correctly, Subject is as expected and that the X509v3 Subject Alternative Name section is present with the expected DNS entries. (Subject Alternative Name is in the X509v3 extensions section.)
You can also do a more extensive test
splunk cmd openssl verify -verbose \ -CAfile ca-chain.pem $(hostname).pem
Set up the various files
Then create symbolic links back to privkey.key from the original key filename. Now, create the server.pem by combining the certificate and private key:
cat $(hostname).pem $(hostname).key > server.pem
Order matters, always the certificate then the private key
The CSR is not required after the PEM is provided but keeping it makes renewal easier and it hurts nothing to retain.
Verify the certificate one last time to ensure all DNS entries in the request are present:
/opt/splunk/bin/splunk cmd openssl x509 \ -in server.pem -text -noout
You also need to install the certificate chain if not already there. Download the file (or copy from another splunk server) and install into the same certs directory as ca-chain.pem e.g. /opt/splunk/etc/apps/ORG_base/certs/ca-chain.pem.
... View more