Overview Creating a TLS cert for Splunk usage is pretty much standard openssl.  To make life better, use an openssl.conf file.  Since I have to look up the syntax and waste a few hours every time I generate a certificate for figuring out how to generate one with a Subject Alternative Name (SAN) correctly, it's documented here.  Splunk Trust now recommends creating certs inside an app in the certs directory of the app, e.g. $SPLUNK_HOME/etc/apps/org_base/certs. Remember, if openssl complains about library issues, as user splunk, use splunk cmd openssl ...  instead of vanilla openssl ... and it will work. See https://hurricanelabs.com/splunk-tutorials/splunk-certificates-master-guide/#:~:text=The%20same%20certificate%20material%20can%20(and%20should)%20be,using%20the%20same%20allows%20for%20ease%20of%20management. for additional information on this procedure See https://help.splunk.com/en/splunk-enterprise/administer/manage-users-and-security/9.4/secure-splunk-platform-communications-with-transport-layer-security-certificates/configure-tls-certificate-host-name-validation-for-secured-connections-between-splunk-software-components for a lot of key details, bad settings can break even starting Splunk. Procedure When renewing, you will be issued a PEM and already have the key.  Go directly to the section below that applies, do not recreate your key and CSR. For purposes of continuity/sanity, we are using filenames for the cert and key of the hostname ending in .pem.  For example, for host clpv003, use the filename clpv003.pem for the certificate and clpv003.key for the key. Generate the certificate There's several steps to this process. openssl.conf Create or update the openssl.conf file.  It is a Good Idea® to store this file in the below referenced /opt/splunk/etc/appname/certs directory. openssl.conf [req] distinguished_name = req_distinguished_name req_extensions = v3_req [req_distinguished_name] countryName = Country Name (2 letter code) countryName_default = US stateOrProvinceName = State or Province Name (full name) stateOrProvinceName_default = CA localityName = Locality Name (eg, city) localityName_default = MyCity organizationalUnitName  = Organizational Unit Name (eg, section) organizationalUnitName_default  = ORG commonName                     = Common Name (eg, YOUR name) commonName_max                 = 64 commonName_default = splunk-test.example.com [ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = clpv003.example.com DNS.2 = splunk-dev.example.com For the commonName_default use the CNAME pointing to the application alias for the role the host in question will serve.  For DNS.1, use the FQDN of the host itself.  For DNS.2, put in the CNAME for the application role again.  It is an expectation and requirement of newer browsers that if a SAN is used, that it takes priority over the subject, so the subject must be listed as one of the SAN entries. ⚠️We may also need the short host name as an alternative name in the SAN for some use cases. Make sure to update the openssl.conf as above, or it will not be useful. While RFC 5280 section 4.2.1.6 explicitly states a Subject Alternative Name (SAN) may be a DNS (FQDN) entry, an IP, a RFC822 address, etc., some CAs do not permit anything but FQDNs.  Also, while contrary to the RFC, some CAs expect the subject field to be filled out, even though if the SAN is filled out, the Subject is optional per the RFC. Create the CSR and key Now, generate the cert almost as expected.  For documentation, we've placed the openssl.conf at /opt/splunk/etc/app/ORG_base/certs/openssl.conf.  Change the app name to the base app for the Splunk instance type. If certificates or keys already exist, alter the below script to ensure unique filenames.  For purposes of this code snippet, we are working on the app named ORG_base and that app is to be deployed by the search deployer (SD) to the search head cluster.  What's important is that the cert work is all done inside an app. As splunk install -d /opt/splunk/etc/apps/ORG_base/certs cd /opt/splunk/etc/apps/ORG_base/certs /opt/splunk/bin/splunk cmd openssl genrsa -out \   $(hostname).key 2048 ln -s $(hostname).key privkey.key /opt/splunk/bin/splunk cmd openssl req -new -out \   $(hostname).csr -key $(hostname).key -config openssl.conf The defaults should  be appropriate but check them anyway.  (Note, you may soon need to use a different algorithm as RSA is being deprecated in favor of post-quantum cryptographic algorithms, but those require newer operating system levels than you may have.) Validate the request As splunk /opt/splunk/bin/splunk cmd openssl req \   -in $(hostname).csr -text -noout Read the result and ensure the DNS entries are in there to create the Subject Alternative Name entries for the request.  Don't forget to check for the Subj of the CSR.  You want to find in it a CN entry with the FQDN. There is a technical reason why both the FQDN and the CNAME are listed as SANs.  Don't forget to list both. Now, collect the CSR and send that to request the certificate by attaching to a ticket to your certificate management group.  The process outlined in “Renewal” in this document also validates and installs the resulting certificate. Create Key for SAML SAML has its own oddities.  To set that up right, you want to create a unique SAML server.pem.  Why?  Because IAM teams don’t want to rework the SAML setup every time your Splunk regular certificate expires. As splunk splunk cmd openssl x509 -req -days 1825 \   -in $(hostname).csr -signkey privkey.key \   -out saml.crt -extfile openssl.conf \   -extensions v3_req cat privkey.key saml.crt > saml.pem Provide the saml.crt as part of your Metadata to the IAM team setting up SAML for your instance. Update the CA chain To ensure update checks and Splunk Base checks work, append the appsCA.pem to the ca-chain.pem: cd /opt/splunk/etc/apps/ORG_base/certs cat /opt/splunk/etc/auth/appsCA.pem >> ca-chain.pem When working on the SHC, we are using a virtual hostname of "splunkshc" as a naming convention for purposes of documentation, and the directory is of course on the search head deployer (SD) at /opt/splunk/etc/shcluster/apps/ORG_base/certs Make Splunk use the new certs To tell Splunk to use the certificate, you must now update (or create) the appropriate configuration files in the server.conf or web.conf (or potentially other configuration files) appropriately.  For this example, we are using a different base directory to emphasize the flexibility here server.conf snippet [sslConfig] sslRootCAPath = $SPLUNK_HOME/etc/apps/ORG_mc/certs/ca-chain.pem serverCert = $SPLUNK_HOME/etc/apps/ORG_mc/certs/server.pem cliVerifyServerName = true ... A snippet only of the web.conf:  (Only needed if Splunk Web is enabled, don't enable it if it isn't required) web.conf [settings] startwebserver = 1 enableSplunkWebSSL = true privKeyPath = $SPLUNK_HOME/etc/apps/ORG_mc/certs/privkey.key serverCert = $SPLUNK_HOME/etc/apps/ORG_mc/certs/$(hostname).pem ... Finally, a snippet of the authentication.conf for systems using SAML auth: authentication.conf [saml] clientCert = /opt/splunk/etc/apps/ORG_mc/certs/saml.pem Remember, SAML uses its own oddities in certificates, and you need to provide the crt file referenced above to the identity team.  Also, after testing, authentication.conf doesn't honor $SPLUNK_HOME.  You must fully qualify the path.  Notice, it is a unique certificate for SAML.  In this case, it’s self-signed. The Search Head Cluster, the configurations are in $SPLUNK_HOME/etc/shcluster/apps/ORG_base with the certs in the certs subdirectory.  They get pushed to all cluster members, so all work is done there, not on individual search heads. Make sure to change the $(hostname) above to the listed filename you chose.  Remember to finish off with a btool check before applying changes (or a bundle validation). Finally, restart splunk via splunk restart.  Login after the restart is complete and examine the certificate to ensure it is taking the cert you intended. Consider trying to replace the restart of Splunk with a series of REST calls referenced at https://help.splunk.com/en/splunk-enterprise/administer/manage-users-and-security/9.4/secure-splunk-platform-communications-with-transport-layer-security-certificates/renew-existing-tls-certificates Renew a certificate When you renew your certificates, start here. When you get the certificate in pem form back, install it into the above directory (e.g., /opt/splunk/etc/apps/ORG_base/certs/$(hostname).pem).  Validate the PEM is correct.  Also ensure that the certificate chain (intermediate then root) is uploaded to ca-chain.pem in the same directory. Validate the cert Copy the received certificate back to the system (e.g. $(hostname).pem) and check it. As splunk cd /opt/splunk/etc/apps/ORG_base/certs /opt/splunk/bin/splunk cmd openssl x509 \   -in $(hostname).pem -text -noout Ensure that the certificate decodes correctly, Subject is as expected and that the X509v3 Subject Alternative Name section is present with the expected DNS entries.  (Subject Alternative Name is in the X509v3 extensions section.) You can also do a more extensive test splunk cmd openssl verify -verbose \   -CAfile ca-chain.pem $(hostname).pem Set up the various files Then create symbolic links back to privkey.key from the original key filename.  Now, create the server.pem by combining the certificate and private key: cat $(hostname).pem $(hostname).key > server.pem Order matters, always the certificate then the private key The CSR is not required after the PEM is provided but keeping it makes renewal easier and it hurts nothing to retain. Verify the certificate one last time to ensure all DNS entries in the request are present: /opt/splunk/bin/splunk cmd openssl x509 \   -in server.pem -text -noout You also need to install the certificate chain if not already there.  Download the file (or copy from another splunk server) and install into the same certs directory as ca-chain.pem e.g. /opt/splunk/etc/apps/ORG_base/certs/ca-chain.pem.   ... View more

When it comes to getting data in, one of the earliest decisions made is what to use as a sourcetype. Often, this choice gets less thought, less consideration than any other detail in log onboarding. That's a mistake. Picking the right sourcetype often drives the right behavior throughout the log onboarding process. Part of the problem with picking a good sourcetype is people often don't have a good understanding of what a sourcetype is. The best comparison I've seen is a sourcetype is the "data type definition" of the data. For those who don't live in the world of XML, that means the list of all the fields that are required or optional, and what kind of data can live in each field. Two bad habits are immediately eliminated by this concept. First, the habit of bringing in different CSV files with different columns with the same sourcetype. It's a lot easier to specify the fields if they have different sourcetypes. Second is use of the _json sourcetype. Stay away from it. Don't use it, ever. It doesn't do what you want. You may be thinking, why not use _json? Well, there's two problems. The first problem is that _json indexes all fields. Contrary to popular myth, indexed fields are not inherently faster, and may actually be slower if you index every field. The second problem is that it makes adding additional props or transforms a lot harder. You now have to link to the source rather than the sourcetype in order to target your transformations or index time adjustments. Rather than only tell you the wrong way, what's a good way to do sourcetypes? A good example of sourcetypes done well is zeek. Each unique sourcetype in the family of zeek sourcetypes has a unique set of fields that are expected in the data. Example sourcetypes are "zeek:conn:json", "zeek:notice:json", "zeek:dns:json" or "zeek:quic:json". One can look at any of the various sourcetypes and know what fields will be present. The second thing to note is that the naming structure is hierarchical. The tradition is to use a single colon to denote the levels from least specific to most specific. In the above examples, The software product is listed first, then the specific component of the product. The last section is a type notation to distinguish the format. While "json" is the most commonly seen format specifier using this notation, you may have other formats as well, such as "csv" or even "kv" (key/value pairs). At this point, you're probably thinking that this issue seems too complex and nitpicky. There's a number of reasons why good sourcetype naming matters. First, a good sourcetype name makes it easy to know what kind of data is in the sourcetype, which improves the ease of referencing the sourceetype explicitly in searches. Second, by using a hierarchical structure for sourcetype naming, when one needs to reference multiple sourcetypes, it's easy to do a `sourcetype=zeek:*` or similar query, even if those sourcetypes span multiple indexes. What about more common sourcetypes, like access_combined? That sourcetype only emphasizes the point. The access_combined sourcetype is very strict and defines exactly what information is stored in the log and how it is stored. Try sending data that isn't perfectly fitting the access_combined definition and all your field extractions break. Some incoming JSON data requires special handling, such as extra large JSON objects sent as a single event. Good sourcetype naming doesn't have to be hard, but it is one of those choices that is hard to change later on. Taking the time to define your sourcetypes well also makes it much easier to understand how to organize your data in indexes and how to arrange your field extractions. Because in the end, what is useful is how you are searching your data. ... View more