confused about X509Verify default certificate warning

[mfrost8]

Hi,

On all my Splunk Enterprise instances, I regularly see

04-10-2017 16:50:03.290 -0500 WARN  X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>

I have setup certificates using our internal CA (not self-signed) for universal forwarders to talk to indexers, for each web interface we use anywhere for Splunk. From my perspective "SSL everywhere". This has worked fine for years now. However, I still see this rather dire sounding warning.

I'm guessing this is complaining that I'm somehow using $SPLUNK_HOME/etc/auth/server.pem at the very least. When I dug around to see what that might be about, it seems that in server.conf, the serverCert is defined as server.pem. I don't set that and leave it as the defaults. I can't really find anything that quite spells this out for me, but I'm guessing that that server.pem cert might be used for say, the local management port. Maybe?

The warning seems to say that I really should change it, but it's not clear to me what to change and how (why?). I can't find any of the SSL documentation that addresses this directly. Can I just point the "serverCert =" definition to that local server's pem file? Something else?

I don't want to be insecure if I don't have to be, which is why I have secured all those other Splunk-to-Splunk channels, but ultimately, I just want to see this warning go away :-).

Thanks

[jkat54 · Accepted solution]

Looks like your replication between peers/indexers isnt SSL.

You'd have something like this in server.conf on the peers:

[replication_port-ssl://8060]

Here's how you test all of this using openssl:

openssl s_client -connect {server}:{port}

#Try all the ports applicable: 8000, 8060, 8089, 9998, 9997, 443, etc.  Your ports can be different, so only you know what ports to test, this is just an example list.  To be more generic: test all of your Splunk ports; management, web, forwarding, replication, etc. as applicable to your environment.

# should end with something like this:
    Verify return code: 0 (ok)
    ---
# if there are any errors, the ssl cert will not validate/verify and the certificate will not be trusted
# you will also see what certificate chain is being used with this test (close to top of results)

When in doubt, refer to this document:
https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr...

Sometimes requires a few reads... to sort it all out, but if you start top to bottom it works almost every time 😉

[mfrost8]

I see this on all hosts everywhere when using Splunk, not just on indexers.

Per my comment with mmodestino above, I'm focusin on a simple heavy forwarder that really isn't doing anything right now except forwarding its own internal events to some clustered indexers.

Referencing Splunk network ports

8000 - shows our certificate
8088 - shows Splunk's certificate (index replication port, but not using it on this server)
8089 - shows Splunk's certificate (as I suspected)
9997 - not listening

As it turns out, I did use Duane's doc to make a final pass at the setup (I'd already figured out and setup most of it, but used it to fill in some cracks). After doing a quick re-read, I can see there's some stuff in there that I did not do as I didn't think they were applicable in our network. I see a references to securing the REST interface and indexer replication port. However, unlike the other port types, I don't quite see that mechanism spelled out. I'm guessing that's slide 26?

It looks like I need to do some experimentation...

Thanks

[jkat54]

You said on all your Splunk instances. Surely some of them listen on different ports according to their roles. The message you are seeing will happen if you use the Splunk cert on any of those ports (to my knowledge).

[mfrost8]

While I'd secured the data port and splunkweb with SSL, I had done nothing to change the default not-so-secure key used for the management port and anywhere else it was used (replication, kvstore, etc). Somehow I missed that part of it. We have a rather extensive environment and I'll need to move carefully to fix this everywhere. Would have been much easier had I spotted it earlier :-.

Have the issues Duane references in his PDF presentation regarding using SSL for clustered indexer replication gone away now? I see the doc is several years old now.

We currently do not use SSL with the deployment server. I've gone back and forth as to why this is even necessary as compared to the other ports. I suppose if I'm passing an unencrypted password within a deployment that could represent an issue. This will mean I have to pass a certificate to each initial install of a UF, I guess. It's possible that this is difficult enough to do that "that ship has sailed", but I'll investigate further.

Anyway, I've now got a server that has an encrypted private key and the "missing" ports are now using it without issue. (It still really sucks that you can't use the same password-protected private key that you use in other configuration with Splunkweb so you have to generate a password-less version of it just for splunkweb).

Thanks very much, jkat54 and mmodestino!

[mattymo]

Hey mfrost8!

You are on the right track with suspecting the mgmt port, but lets start by focusing on a particular host and we can walk through validating that. Filter down to a host and examine the ports splunk is running and we can go from there.

Based on what you have advised I would suggest you don't have too much to worry about. It is likely the communication to the Deployment server it is complaining about...but lets prove it!

[mfrost8]

OK, so let's take a heavy forwarder with no event activity passing through it, that I was playing with yesterday. That is, no other host has this server in its outputs.conf anywhere. It does have the best-practices outputs.conf mechanism in place to forward its own logs to an indexer, but that would be the only event traffic involving this server.

This server does have its own internal-CA-generated certificate as is common here.

Note that in the output below, names and encrypted passwords have been changed to protect the innocent. "hostA" refers to the local server's physical hostname, however we use CNAMEs for our servers and in this case, this server is "fwd01" below. Our certificates have SANs that allow both names plus the IP of the server as valid names. Hence, physical hostname, alias or IP would all work fine through Splunkweb.

inputs.conf

[default]
host = hostA

[SSL]
sslPassword = zzz
requireClientCert = false
serverCert = $SPLUNK_HOME/etc/auth/mycerts/fwd01.pem

outputs.conf (the local event forwarding is in another file...)

[tcpout]
clientCert = $SPLUNK_HOME/etc/auth/mycerts/fwd01.pem
defaultGroup = prod_default
sslRootCAPath = $SPLUNK_HOME/etc/auth/mycerts/cacerts.pem
sslVerifyServerCert = true
useClientSSLCompression = true

server.conf

[general]
serverName = fwd01
pass4SymmKey = xxx

[sslConfig]
sslPassword = yyy
sslRootCAPath = $SPLUNK_HOME/etc/auth/mycerts/cacerts.pem

web.conf

[settings]
enableSplunkWebSSL = true
privKeyPath = $SPLUNK_HOME/etc/auth/mycerts/fwd01.privatekey.pem
serverCert = $SPLUNK_HOME/etc/auth/mycerts/fwd01.pem

lsof -p -P | grep IPv4

reports the following

splunkd 23350 splunk    5u  IPv4          481132186      0t0       TCP *:8089 (LISTEN)
splunkd 23350 splunk   31u  IPv4          481135551      0t0       TCP *:8088 (LISTEN)
splunkd 23350 splunk   79u  IPv4          481159127      0t0       TCP hostA:35646->idx2:9997 (ESTABLISHED)
splunkd 23350 splunk   84u  IPv4          481133325      0t0       TCP *:8000 (LISTEN)
splunkd 23350 splunk   87u  IPv4          481159129      0t0       TCP hostA:57190->idx1:9997 (ESTABLISHED)
splunkd 23350 splunk   93u  IPv4          481133327      0t0       TCP localhost:32931->localhost:8191 (ESTABLISHED)
splunkd 23350 splunk   94u  IPv4          481134009      0t0       TCP localhost:32932->localhost:8191 (ESTABLISHED)
splunkd 23350 splunk   95u  IPv4          481135617      0t0       TCP localhost:32933->localhost:8191 (ESTABLISHED)
splunkd 23350 splunk   96u  IPv4          481134011      0t0       TCP localhost:32934->localhost:8191 (ESTABLISHED)

Just to see, I removed the deploymentclient.conf entry here so this heavy forwarder doesn't have a deployment server to talk to -- all local config -- and restarted. The X509 warning doesn't go away, so I doubt it's the deployment client/server.

I assumed everyone got this warning in their logs... I can say for instance, that if I spin up a "virgin" Splunk Enterprise with no configuration at all, this warning shows up in the logs as well. If it's worth of displaying a warning but it isn't a problem, I'm not sure why it would show up in the logs? It's like saying there's an "OK error".

Thanks!