How to verify splunk Enterprise Management port 8089 is indeed secured with own certificates

[vtalanki]

Hi All,

We want to enable ssl in our aws splunk Enterprise cluster on management port 8089 with own certs(provided by my company)
I followed all the required steps from various documents and enabled splunkd ssl in server.conf in all splunk components - cm, indexer cluster, sh cluster and deployer

How can I verify that the ssl is correctly enabled and is using our own certs?
I don't see any errors in any of splunkd logs. But I do not know how to prove that splunk instances are communicating with own certs.
Also how is the secure communication happening without client certs? Don't we need both client and server certs in all splunk instances to securely communicate on port 8089(for that matter any port)
Any help is highly appreciated

[PavelP · Accepted solution]

Hello @vtalanki,

you can use wireshark or openssl to verify it.

openssl s_client -connect your-aws-server:8089

[vtalanki]

Thanks @PavelP , I see my cert being displayed by using openssl.

However when I use curl to connect to indexer from master ( for that matter between any two instances ) I'm seeing this

$curl 'https://idxr-ip:8089/services/cluster/config' -v
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.

When I provided --cacert and --capath options to my curl I get
* NSS error -12276 (SSL_ERROR_BAD_CERT_DOMAIN)
* Unable to communicate securely with peer: requested domain name does not match the server's certificate.
I guess this is because the CN or SAN in my certs doesn't match the indexer dns I'm using. Any idea how this can be achieved ?

For that matter even if splunk self-signed certs I should be seeing these errors. But I dont see any errors in Splunkd.log . Is this because splunk doesn't verify certs when communicating on 8089 with ssl? If yes, what's the point in enabling ssl then?

[PavelP]

if you can connect by adding -k switch (ignore cert errors), then your question "How can I verify that the ssl is correctly enabled and is using our own certs?" can be answered positively - the ssl is enabled and the server is using your certs. You have to provide --cacert and --capath unless you include your company's CA and SubCA certificates in the root storage, so this error is OK if the certificate was issued by the company's PKI CA.

BAD_CERT_DOMAIN means the domain which you requesting doesn't match the SAN entry. You have to used domain exactly as it written in the SAN entry.

so if your certificate looks like this:

subject=.....CN=id.example.com
            X509v3 Subject Alternative Name:
            DNS:idxr.example.com

you have to use idxr.example.com only. Not an IP address any kind, not a short domain name like idxr (even if can be connected).

If you cannot connect using -k switch, then the whole SSL setup is wrong.

[vtalanki]

Thanks again @PavelP . I checked with -k switch at it gave me the required output. And seems like the

 requireClientCert = false

config in server.conf is equivalent to -k switch (correct me if I'm wrong).
However going to production, we do want to enable certificate verification to happen on management port 8089. So I made requireClientCert = true , restarted splunkd and I am seeing below errors/warning

cluster master splund.log

    ERROR X509Verify - X509 certificate (CN=XXXXXX) failed validation; error=26, reason="unsupported certificate purpose"
WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client certificate B', alert_description='unsupported certificate'.
WARN  HttpListener - Socket error from <master-ip>:36874 while idling: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.

Indexer splund.log

WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server session ticket A', alert_description='unsupported certificate'.

Any ideas about this? Do I need to have client certificate aswell for this? If yes, how should I refer to client cert(where should I set it)

And thanks for clarifying on BAD_CERT_DOMAIN. Got your point here.

[PavelP]

this called mutual TLS - not only the client verifies the server's cert, but also the server checks the client's cert. This part is not set correctly, so read this presentation by @dwaddle : https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr... , this is exactly information you need.

[vtalanki]

By 'This part is not set correctly' , do you mean that Splunk currently doesn't support mTLS?

I have gone thru both the slides and recording of this conf session but didn't understand exactly why requireClientCert is disabled for 'Splunk to Splunk' communication(All it talks about is why it is disabled between DS and DC).

[PavelP]

Hello @vtalanki ,

the talk is 5 year old, it was ahead of time (most people just wanted to make splunk "work") and is still great as an overview.

here is a link about securing inter splunk communication (port 8089): https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/Securingyourdeploymentserverandclients

Edit server.conf to authenticate against your certificates by adding the following attribute to the [sslConfig] stanza in previous step:

requireClientCert = true

Important: This requireClientCert is set to "false" by default. If you change it to true to force Splunk to check your client's certificates, Splunk Web and the CLI will also be checked for certificates. Your CLI connection will no longer work because your CLI is unable to present a certificate as a client.

Let me know if mTLS works for you.

[vtalanki]

Hello @pavelP,
I have followed this and made requireClientCert = true in CM and restarted splunk. Below is what I see

Cluster Master splunkd.log

ERROR X509Verify - X509 certificate (CN=XXXX,OU=XXX....) failed validation; error=26, reason="unsupported certificate purpose"
WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client certificate B', alert_description='unsupported certificate'.
WARN  HttpListener - Socket error from <idxr1_ip>:36874 while idling: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.

Indexer splunkd.log

WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server session ticket A', alert_description='unsupported certificate'.

In both instances, server.conf is something like

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
sslVersions = tls1.2
serverCert = <path_to_pem>/server-cert-with-key.pem
sslRootCAPath =  <path_to_pem>/ca-chain.pem
sslVerifyServerCert = true
requireClientCert = true

Except that requireClientCert = false in indexer server.conf

Is it correct to have server.pem in both CM and indexers for serverCert? Sorry, really confused.

BTW, here is the output of openssl verify

$ /opt/splunk/bin/splunk cmd openssl verify -CAfile ca-chain.pem server-chain-with-key.pem
server-chain-with-key.pem: OK

[PavelP]

I think there is something wrong with your certificates, the error message mentioned: "note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name."

[vtalanki]

@PavelP , This is my current certificate structure. Please correct me if I am not having them in correct structure.

server-cert-with-key.pem

[ server cert ]
[ server cert key ]
[ intermediate A cert ]
[ intermediate B cert ]
[ Root cert ]

ca-chain.pem

[ intermediate A cert ]
[ intermediate B cert ]
[ Root cert ]

And in my server.conf

 serverCert = <path_to_pem>/server-cert-with-key.pem
 sslRootCAPath =  <path_to_pem>/ca-chain.pem

[PavelP]

not sure if ca-chain.pem requires a server cert at the top too, check https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/HowtoprepareyoursignedcertificatesforSpl...

[ server certificate]
[ intermediate certificate]
[ root certificate (if required) ]

[vtalanki]

@PavelP , firstly thanks for your continuous guidance in this regard.
secondly, I tried making the ca-chain.pem to include [server cert] at. the top. but ended up getting the same error.
Also I dont think [server cert] need to be in ca-chain.pem - I have gone thru multiple splunk answers and even in the 2015 .conf, they use cacert.crt for ca chain which is CA root certificate chain

[vtalanki]

@PavelP , this issue has been resolved after I have replaced server certs(server only) certs with multi-purpose certs

server cert

$ openssl x509 -noout -in server.pem -purpose
Certificate purposes:
SSL client : No
SSL server : Yes

Multi-purpose cert

$ openssl x509 -noout -in multi-purpose.pem -purpose
Certificate purposes:
SSL client : Yes
SSL server : Yes

I dont think has been documented anywhere.

Thanks alot for your overall help on this