Archive

After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Contributor

I had to make a non clean upgrade to 6.3.0.
The disk Splunk has its indexes on had to be expanded, and the way it happened was, that a new disk was allocated, the indexes copied to the new disk over a period of time, and then in the service window, a final copy, and then rename the disks. The only problem was, it was a copy, not a mirror, so Splunk was unable to start due to duplicate buckets (hot buckets moved to warm - but not removed in the copy)

After the upgrade I have troubles with Mongod.
In the Splunkd.log I get the following:

10-04-2015 03:20:44.705 +0200 ERROR MongodRunner - mongod exited abnormally (exit code 2, status: exited with code 2) - look at mongod.log to investigate.
10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not get pint from mongod.
10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.

In Mongod.log I get nothing.

Anybody having any experiences with this?

kind regards

1 Solution

Contributor

As dmr195 I upgraded to Splunk 6.4.0, and regenerated the server certificate, and that solved tho problem for me.

View solution in original post

Engager

cd splunk/etc/auth
for i in *.pem; do echo $i && openssl x509 -in $i -text |grep -A 2 Validity; done
server.pem
Validity
Not Before: Mar 25 17:03:32 2013 GMT
Not After : Mar 24 17:03:32 2016 GMT

regenerate server.pem
$SPLUNK_HOME/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n server
....
subject=/CN=server/O=SplunkUser
Getting CA Private Key
unable to write 'random state'
subject= /CN=server/O=SplunkUser
issuer= /C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
notBefore=Apr 6 15:43:00 2017 GMT
notAfter=Apr 5 15:43:00 2020 GMT

new file
ls -ltr
-rw------- 1 2747 Apr 6 10:43 server.pem
-rw------- 1 17 Apr 6 10:43 ca.srl

verify
for i in *.pem; do echo $i && openssl x509 -in $i -text |grep -A 2 Validity; done
server.pem
Validity
Not Before: Apr 6 15:43:00 2017 GMT
Not After : Apr 5 15:43:00 2020 GMT

restart, issue fixed

Contributor

As dmr195 I upgraded to Splunk 6.4.0, and regenerated the server certificate, and that solved tho problem for me.

View solution in original post

Communicator

Ran into the same errors referencing a KB article that is installed on windows: The exact syntax referenced above was giving me grief but a quick call to support squared me away:

Specific to windows:
in $splun_home ... in my case e:\splunk

e:\splunk\bin> splunk createssl server-cert -d e:\splunk\etc\auth -n server -c cn.domain.com -l 2048

.\splunk.exe restart

GTG

Path Finder

Solved my problem on 6.5.2, as well as the detailed explanation here: link text

0 Karma

Communicator

19859 2016-03-23T14:14:52.408Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
19860 2016-03-23T14:14:52.423Z I CONTROL Hotfix KB2731284 or later update is not installed, will zero-out data files
19861 2016-03-23T14:14:52.876Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

I appear to be getting the exact same error in mongod.log and also on a windows 2008 R2 machine. Does anyone have exact syntax required to recreate those certs on windows? I'm coming up empty... OR can someone tell me when 6.3.4 will be out to hopefully just resolve itself?

0 Karma

Splunk Employee
Splunk Employee

We hit similar situation after upgrade from Splunk Version 6.2.0 to 6.3.
Post Upgrade KVstore is unable to start and failed with the following error messages

tail -f mo*.log

Heading

2015-10-28T00:56:25.328Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-28T00:56:25.334Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
2015-10-28T00:59:56.176Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-28T00:59:56.182Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
2015-10-28T01:41:30.882Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-28T01:41:30.888Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
2015-10-28T01:43:25.746Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-28T01:43:25.752Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
2015-10-28T02:13:30.655Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-28T02:13:30.661Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

Heading

Certificate related Information:

splunkd always connects to mongod using SSL. mongod replicas also communicate with each other using SSL. By default, the Splunk server certificate, etc/auth/server.pem, is used as the SSL server cert. In this case, SSL is only used to encrypt communication, not for client authentication or authentications between replicas, those authentications are done using an auto generated and protected password and key files. In SHC, the password and key files share same contents among cluster members.

In Dash, mongo password auth is not FIPS compliant. For this reason, we introduced X509 cert based auth. If you turn on FIPS, you must configure certificates in server.conf for KV store.

In Ember, we upgraded to a new version of mongo. It's FIPS compliant. Thus, X509 cert base auth becomes optional.

As mentioned earlier, by default, we use SSL only to encrypt communication. We don't use it to either to verify server identity or authenticate client. For this reason, you will see a warning message in mongod log:

2015-10-28T17:50:00.648Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

Here are the step that resolved our issue :

1) First we ran the following command to check teh expiry data of the Current certificate

Could you please run the following command in directory $SPLUNK_HOME/etc/auth and provide me the output of it.

for i in *.pem; do echo $i && openssl x509 -in $i -text | grep -A 2 Validity ; done

In our case we could see that certificate server.pem is valid and have not expired- as shown below :


opt/splunk/etc/auth$ for i in *.pem; do echo $i && openssl x509 -in $i -text | grep -A 2 Validity ; done
appsCA.pem
Validity
Not Before: Sep 1 12:00:00 1998 GMT
Not After : Jan 28 12:00:00 2028 GMT
cacert.pem
Validity
Not Before: May 11 19:51:37 2015 GMT
Not After : May 8 19:51:37 2025 GMT
ca.pem
Validity
Not Before: May 11 19:51:37 2015 GMT
Not After : May 8 19:51:37 2025 GMT
privKeySecure.pem
unable to load certificate
2018:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: TRUSTED CERTIFICATE
req.pem
unable to load certificate
2022:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: TRUSTED CERTIFICATE
server.pem
Validity
Not Before: Oct 30 01:20:02 2015 GMT
Not After : Oct 29 01:20:02 2018 GMT

2) Although the certificate was valid we re-generated it using command ---followed by restart of splunk and that seems to be resolved the issue.

$SPLUNk_HOME/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n server ( assuming that auth directory is /opt/splunk/etc/auth )

Explorer

Create new certificate worked for me in 6.5.0. Thanks

0 Karma

Builder

It worked also for a Customer of our who had the same problem migrating from Splunk 6.2.4 to Splunk 6.3.2 on Windows 2008R2.

In mongod.log we had:

2016-02-05T13:55:47.766Z I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
2016-02-05T13:55:48.359Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
2016-02-10T11:45:16.558Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

The "splunk createssl" command refreshed the certificate and everithing is working fine again.

Thanks!

0 Karma

Communicator

I had the same errors as in the original post after upgrading to Splunk 6.4, and regenerating the certificates using the splunk createssl command fixed it.

0 Karma

Path Finder

I have exactly the same problem and an open ticket at Splunk Support.

I tried to upgrade from version 6.2.5 and 6.3 (OS: Windows Server 2008R2)

After the upgrade I get the following errors in mongod.log
2015-09-23T19:30:11.390Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-09-23T19:30:11.452Z I CONTROL Hotfix KB2731284 or later update is installed, no need to zero-out data files
2015-09-23T19:30:11.764Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

What I found is a similar known issue with Mongodb Version 3.0.4
https://jira.mongodb.org/browse/SERVER-19538
It was fixed in Version 3.0.6.
I don't get this error on my Deployment Server (same operating system: Windows Server 2008R2)

kind regards

Splunk Employee
Splunk Employee

Could you ping them and ask them to ping me (dgladkikh)? I would like to look on the diag.

0 Karma

Path Finder

@dgladkikh
I discovered the problem directly after the upgrade and went back to version 6.2 (thanks to Vmware snapshot)
I didn't create a diag and could only sent the log files (splunkd.log, mongod.log, etc)
The problem appears only on our production system, so I have to wait for a maintenance window (probably Wednesday next week) to upgrade again and create a diag (Splunk support is waiting for that)

0 Karma

Motivator

Any official response from Splunk yet?

0 Karma

Path Finder

It will be fixed in 6.3.4 (bug SPL-109512)

0 Karma

New Member

Maybe you can try to disable ssl in the server.conf. Put it in the following path /opt/splunk/etc/system/local and write the following entrys in it:
[sslConfig]
enableSplunkdSSL = false

0 Karma

Contributor

I don't really want to disable SSL for Splunk, and I don't think that is the issue.
If I stop Splunkd, and delete the kvstore folder, I don't get the SSL errors, but mongod is stil unable to start. The only difference is, that it doesn't start with anything in mongod.log - absolutely no entries in that log.

0 Karma

New Member

SSL for the LoginWebsite is still working when you set this override. But with this Setting the kvstore is working for me. From the upgrade documentation:

enableSplunkdSSL = true|false
* Enables/disables SSL on the splunkd management port (8089) and KV store
port (8191).

Maybe you can replace the certificate in the following path with trusted ones: /opt/splunk/etc/auth

Greets
Jörg

0 Karma

New Member

Same problem here, only difference is that we are using a linux splunk instance:
[user@server bin]#splunk validate files
Validating installed files against hashes from '/opt/splunk/splunk-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64-manifest'
All installed files intact.
[user@server bin]# splunk cmd mongod --version
db version v3.0.3-splunk
git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f
OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015

Maybe it's usefull

0 Karma

Splunk Employee
Splunk Employee

Problem can look similar, but dont think that it os the same problem. Do you have anything in mongod.log? Any errors in splunkd.log?

0 Karma

New Member

After the update only this event appears in the mongod.log:
2015-10-07T11:46:43.020Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-07T11:46:43.025Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

And the splunkd.log shows:
10-07-2015 13:46:43.034 +0200 ERROR MongodRunner - mongod exited abnormally (exit code 2, status: exited with code 2) - look at mongod.log to investigate.
10-07-2015 13:46:43.157 +0200 ERROR KVStoreConfigurationProvider - Could not get pint from mongod.
10-07-2015 13:46:44.159 +0200 ERROR MongodRunner - Did not get EOF from mongod after 1 second(s).
10-07-2015 13:46:57.445 +0200 INFO WatchedFile - Will begin reading at offset=15701880 for file='/opt/splunk/var/log/splunk/mongod.log'.

the splunkd.log hast GMT+2 Timezone, so there is a difference in the timestamps.

Greets
Jörg

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!