How do I Secure SplunkWeb on a Search Head Cluster

[sniderwj]

I am working on securing our Splunk environment. I'm starting with the Splunk Web parts. I have a few non-clustered Search Heads and they went fine with no issues. I'm adding the SSL settings to the $SPLUNK_HOME/etc/system/local/web.conf.

When I do the same thing to my Search Head Cluster the Splunk Webs do not respond. Looking in the splunkd.log I can see issues with connecting to each other. I can't find any of the documentation for securing the Search Head Cluster.

I am following the Splunk SSL Best Practices .conf session for most of this (can't post links yet). They don't really discuss the SHC side of things.

I was building certificates for each SHC member with a common Subject Alternative Name as well as the host name. Figuring on the Splunk Web parts would use common name and then I could set them up to use the host name part of the certificate to use other Splunk parts (splunkd, kvstore).

Does anyone have any good documentation or guidance on how to do secure Search Head Clusters?

With it being a Cluster can I only do this through apps pushed out through the Deployer?

Should I be using a common certificate for Splunk Web?

[goodsellt]

This is definitely a challenge due to the lack of documentation on it. We've had good results by following the standard instructions, however make sure you have the following:
1. If you're not using a wildcard cert (such as *.splunk.org.com), make sure all the certs share the same CA, and that CA is valid/trusted within Splunk's CA store.
2. It's also a really good idea to make sure the servers are able to chat over port 8089 (or whatever management port you use) prior to turning on the certs.
3. If you're doing individual certs, make sure all settings related to SSL enablement and certs are put into $SPLUNK_HOME/system/local

[horsefez]

Hi sniderwj,

---

Lets start with securing splunk-web

First step is generating a private key:
openssl genrsa -out /opt/splunk/etc/auth/splunkweb/webserverprivate.key 2048

Then generate a certificate-request:
openssl req -new -subj "/C=something/ST=something/L=something/O=something/OU=something/CN=$(hostname)" -key /opt/splunk/etc/auth/splunkweb/webserverprivate.key -out /opt/splunk/etc/auth/splunkweb/webservercertrequest.csr

Here at my company I send this request to our company owned CA were I get a certificate in return in x509 format.
This is a "server" certificate.
I then save that certificate into the following file:
cat > /opt/splunk/etc/auth/splunkweb/webservercertificate.crt

Now I have to convert the x509 format into PEM format: (you will not see any difference, but trust me it's different)
openssl x509 -in "/opt/splunk/etc/auth/splunkweb/webservercertificate.crt" -out "/opt/splunk/etc/auth/splunkweb/webservercertificate.pem" -outform PEM

I then post the following stanza into the web.conf:

[settings]
enableSplunkWebSSL = 1
updateCheckerBaseURL = 0
privKeyPath = /opt/splunk/etc/auth/splunkweb/webserverprivate.key
caCertPath = /opt/splunk/etc/auth/splunkweb/webservercertificate.pem

---

Lets continue with securing inter-splunk-communication

I start with creating a new directory:
mkdir /opt/splunk/etc/auth/serverclientcert/

Then generating a new privatekey now with keyphrase:
openssl genrsa -des3 -out /opt/splunk/etc/auth/serverclientcert/private.key 2048

Generating a certificate request:
openssl req -new -subj "/C=something/ST=something/L=something/O=something/OU=something/CN=$(hostname)" -key /opt/splunk/etc/auth/serverclientcert/private.key -out /opt/splunk/etc/auth/serverclientcert/certificaterequest.csr

Sending Request to CA - IMPORTANT - This needs to be a "server" and also a "client" certificate.
cat > /opt/splunk/etc/auth/serverclientcert/serverclientcert.crt

My company has a SubCA certificate and a RootCA certificate:
cat > /opt/splunk/etc/auth/serverclientcert/subcacertificate.crt
cat > /opt/splunk/etc/auth/serverclientcert/rootcacertificate.crt

Then convert x509 into pem (won't go into details, because it's already up there)

Then... the not so well documented happens:
Splunk wants me to put all those files into one file in a certain order:
cat /opt/splunk/etc/auth/serverclientcert/serverclientcert.pem /opt/splunk/etc/auth/serverclientcert/private.key /opt/splunk/etc/auth/serverclientcert/subcacertificate.pem /opt/splunk/etc/auth/serverclientcert/rootcacertificate.pem > /opt/splunk/etc/auth/serverclientcert/completeserverclientcert.pem

Now there should be a file with following order:
- server-cert
- private-key (encrypted)
- subca-cert
- rootca-cert

Then we need also a chain-file:
cat /opt/splunk/etc/auth/serverclientcert/subcacertificate.pem /opt/splunk/etc/auth/serverclientcert/rootcacertificate.pem > /opt/splunk/etc/auth/serverclientcert/completecachain.pem

Then I copy all needed files into /auth:
cp /opt/splunk/etc/auth/serverclientcert/completecachain.pem /opt/splunk/etc/auth/serverclientcert/completeserverclientcert.pem /opt/splunk/etc/auth/

In inputs.conf on the indexers now goes the following stanza:

[SSL]
rootCA = $SPLUNK_HOME/etc/auth/completecachain.pem
serverCert = $SPLUNK_HOME/etc/auth/completeserverclientcert.pem
password = your-private-key-keyphrase
requireClientCert = true


[splunktcp-ssl:9997]
compressed=true
disabled=0

Then there goes this into the outputs.conf on every SH or Master/Deployer:

[indexAndForward]
index = false

[tcpout]
defaultGroup = Indexers
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:Indexers]
server = indexer1:9997,indexer2:9997


[tcpout-server://indexer1:9997]
sslRootCAPath = $SPLUNK_HOME/etc/auth/completecachain.pem
sslCertPath = $SPLUNK_HOME/etc/auth/completeserverclientcert.pem
sslPassword = your-private-key-keyphrase
sslVerifyServerCert = true
useClientSSLCompression = true
sslCommonNameToCheck = indexer1

[tcpout-server://indexer2:9997]
sslRootCAPath = $SPLUNK_HOME/etc/auth/completecachain.pem
sslCertPath = $SPLUNK_HOME/etc/auth/completeserverclientcert.pem
sslPassword = your-private-key-keyphrase
sslVerifyServerCert = true
useClientSSLCompression = true
sslCommonNameToCheck = indexer2

---

Voila!
Hope this helps and works!

Best regards,
pyro_wood

[sniderwj]

That is some great info! Thanks.

I haven't started tackling the inter-splunk communications yet. I'll definitely use this to help me out.

The Splunk Web steps that you describe are what I have been doing on my non-clustered Search Heads without a problem. I am assuming that I would need to put these entries in the $SPLUNK_HOME/system/local/web.conf for each of my Clustered Search Heads if I'm not using a common certificate for each of the hosts of the cluster. Right now I'm building certificates with Subject Alternative Names (SANs) for audit.foo.com and hostname.foo.com.

Do I need to do the Inter-Splunk communications as well before I can get the SHC working with the Splunk Web being secured?

[horsefez]

Hi,

the inter-splunk-communication is independent from the splunk-web configuration.

Yes, you need to put that stanza onto every Search-Head or every instance you want to access via secure web.

The configs (outputs/inputs) are for "forwarding-internal-data-to-the-peers"

[horsefez]

Hi there sniderwj,

first of all let me tell you my story with Inter-Splunk-Communication aswell as secure-web (https) access and securing them with company signed certificates.
Unfortunately the splunk-docs are a bit sloppy around that matter. When I was configuring those certificates I ran into several difficult problems and the worst problem took me almost 1 1/2 weeks to resolve. All in all I will always remember setting up non-splunk certificates for securing splunk was the most frustating time I had with administrating splunk.

Anyway... I finally resolved it and now have a pretty good understanding of it aswell as a good documentation and never had anymore issues with it. Sadly its Friday in the evening here in germany, so I'm not at work to access my files there. If you would love to hear about my experience with that subject more, you could contact me via PM and we can use "discord" or so for the talking.

If you are not able to resolve this till monday, I will be able to post you my docs and experiences about this.

[brreeves_splunk]

I downvoted this post because this is not a helpful story to the community. it did not actually answer any questions.

[sniderwj]

Morning pyro_wood,

Thanks for trying to help out. I'm not in a good spot to jump on a chat or anything. If you could post any of the docs you have or at something to point me in the right direction.

Thanks!