Getting Data In

Why does Splunk KVStore does not start with mongod error "please specify an sslCAFile parameter" after SSL'ing all servers? "

Path Finder

Windows 2008 R2 servers.

I have SSL/TLS enabled all comms between search head, deployment server, indexers and universal forwarders.

server.conf:

[sslConfig]
caCertFile = chain.pem
caPath = $SPLUNK_HOME\etc\auth\DOD
requireClientCert = false
sslKeysfile = password.pem
sslKeysfilePassword = **********

[kvstore]
caCertpath = $SPLUNK_HOME\etc\auth\DOD
sslKeysPassword = *********
sslKeysPath = $SPLUNK_HOME\etc\auth\DOD\private.pem

===============================================================================================
When I start Splunk servers I am getting errors in the mongod.log:

W CONTROL No SSL certificate validation can be performed since no CA file has been provided;  please specify an sslCAFile parameter
E NETWORK cannot read certificate file: d:/splunk/etc/auth/DOD/private.pem error:0906D06C:PEM routines:PEM_read_bio:no start line
F CONTROL Failed global initialization: Location16778 ssl initialization problem

===============================================================================================
sslCAFile appears to be a mongod command line variable.

Is Splunk suppose to start and pass the sslCAFile variable somewhere???

Robert

0 Karma

Motivator

On our servers the Old Splunk certs expired Oct 1 2015. We just upgraded to 6.3 and new certs were not generated. Recreating all certs with ./splunk createssl fixed it.

0 Karma

Path Finder

Still nothing. I am leaning towards the FIPS mode not being enabled when the server was built.

No matter what key I create or try to set in the [kvstore] area, I get the same errors about not being able to read the key that is set in the [sslconfig] for sslKeysfile. why does mongod/kvstore keep trying to read that password.pem key????

W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
E NETWORK cannot read certificate file: d:/splunk/etc/auth/DOD/password.pem error:0906D06C:PEM routines:PEM_read_bio:no start line
F CONTROL Failed global initialization: Location16778 ssl initialization problem

0 Karma

Path Finder

I deleted these lines and restarted and mongod starts without errors

[sslConfig]
caCertFile = chain.pem <--------------------------------deleted
caPath = $SPLUNK_HOME\etc\auth\DOD <--------------------------------deleted
requireClientCert = false <--------------------------------deleted
sslKeysfile = password.pem <--------------------------------deleted
sslKeysfilePassword = $1$**********

[kvstore] <--------------------------------deleted
caCertpath = $SPLUNK_HOME\etc\auth\cacert.pem <--------------------------------deleted
sslKeysPath = $SPLUNK_HOME\etc\auth\server.pem <--------------------------------deleted
sslKeysPassword = password <--------------------------------deleted

There's something wrong with my certs.... .I'll look into them more.

0 Karma

Path Finder

what was the resolution here?

0 Karma

Path Finder

command line: I did:
splunk stop
splunk createssl server-cert -d d:\splunk\etc\auth -n server
server.pem was created

my \splunk\etc\system\local\server.conf [sslconfig] and [kvstore] look like this:

[sslConfig]
caCertFile = chain.pem
caPath = $SPLUNK_HOME\etc\auth\DOD
requireClientCert = false
sslKeysfile = password.pem
sslKeysfilePassword = $1$**********

[kvstore]
caCertpath = $SPLUNK_HOME\etc\auth\cacert.pem
sslKeysPath = $SPLUNK_HOME\etc\auth\server.pem
sslKeysPassword = password

I did splunk start. there are no errors.

the sslKeysPassword stanza under kvstore, is now equal to $1$********** (matches the sslconfig hash)

mongod.log still giving same errors:

W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
E NETWORK cannot read certificate file: d:/splunk/etc/auth/DOD/password.pem error:0906D06C:PEM routines:PEM_read_bio:no start line
F CONTROL Failed global initialization: Location16778 ssl initialization problem

mongod/kvstore still keeps looking at the sslconfig area.....

Robert

0 Karma

Motivator

You don't need the caCertpath and sslKeysPath for the kvstore stanza. Also make sure you recreate the web certs as I'm not sure which ones apply here. As for names of the certs I just went for "server" for the server certs to replace the defaults.

0 Karma

Motivator

Btw your ca path may need to be changed to the default. Just uncomment that parameter

0 Karma

Communicator

It looks like you need to add the actual file to the value specified in caCertpath in the [kvstore] stanza (e.g., caCertpath = $SPLUNK_HOME\etc\auth\DOD\chain.pem).

From the server.conf spec:
caCertPath =
* Public key of the signing authority.
* If specified, it will be used in KV Store SSL connections and
authentication.
* Must be specified if FIPS is enabled (i.e. SPLUNK_FIPS=1), otherwise, KV
Store will not be available.
* Only used when FIPS is enable

You could use btool to see what is actually being applied and what file it is reading it from:
$SPLUNK_HOME\bin\splunk btool server list --debug

0 Karma

Path Finder

I typo'd this in my original post: E NETWORK cannot read certificate file: d:/splunk/etc/auth/DOD/private.pem error:0906D06C:PEM routines:PEM_read_bio:no start line <--- shud be password.pem, not private.pem

I made these two new files root1 and root2 and changed the server.conf file:
caCertPath = $SPLUNK_HOME\etc\auth\DOD\root1.pem
sslKeysPassword = *********
sslKeysPath = $SPLUNK_HOME\etc\auth\DOD\root2.pem

root1.pem contains the public key for the root authority
root2.pem contains my password protected private key

Restarted splunk services. Still no good. mongod.log has same error. It's like the [kvstore] configs are not even being recognized:

W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
E NETWORK cannot read certificate file: d:/splunk/etc/auth/DOD/password.pem error:0906D06C:PEM routines:PEM_read_bio:no start line
F CONTROL Failed global initialization: Location16778 ssl initialization problem

What's this about FIPS ???? If we did not enable SPLUNK_FIPS = 1 in our initial server build, will all of this stuff not work????????

0 Karma