topic Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors? in Installation

After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

las — Mon, 05 Oct 2015 06:50:57 GMT

I had to make a non clean upgrade to 6.3.0.
The disk Splunk has its indexes on had to be expanded, and the way it happened was, that a new disk was allocated, the indexes copied to the new disk over a period of time, and then in the service window, a final copy, and then rename the disks. The only problem was, it was a copy, not a mirror, so Splunk was unable to start due to duplicate buckets (hot buckets moved to warm - but not removed in the copy)

After the upgrade I have troubles with Mongod.
In the Splunkd.log I get the following:

10-04-2015 03:20:44.705 +0200 ERROR MongodRunner - mongod exited abnormally (exit code 2, status: exited with code 2) - look at mongod.log to investigate.
10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not get pint from mongod.
10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.

In Mongod.log I get nothing.

Anybody having any experiences with this?

kind regards

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

dgladkikh_splun — Mon, 05 Oct 2015 18:36:05 GMT

Please take a look inside #SPLUNK_HOME/var/log/mongod.log to see what is wrong. It should give you more details about this problem.

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

las — Tue, 06 Oct 2015 06:47:00 GMT

That is part of the problem, There are no entries in mongod.log.
If I copy the original kvstore into the current repository, I get some expired certificate errors:

2015-10-04T01:20:44.330Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-04T01:20:44.330Z I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
2015-10-04T01:20:44.673Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

But if I clear the kvstore, I get no entries what so ever.

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

dgladkikh_splun — Tue, 06 Oct 2015 13:59:37 GMT

What do you mean by copying original kvstore in current repository? What is original and what is repository?
Also, could you describe what was your steps to upgrade to 6.3? Is it Linux/Windows?

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

las — Tue, 06 Oct 2015 14:15:37 GMT

Sorry, I should have mentioned, it is on Windows 2008 R2 64 bit.
So I just ran the MSI file to upgrade.
As I mentioned, there were problems due to a diskexpansion, where the procedure was to make a copy of the folder containing the live indexes, in the service window, Splunk was stopped, and a last copy was made, the live disk was then renamed, and the new disk renamed to the old disks name.
That gives me a copy off the folder kvstore\mongod on the old live disk.
As there were problems with the copy being a copy, not a mirror (duplicate buckets), I had a cleaning job to do, and since Mongod did not start, I assumed it had similar problems.
We do not really use KVstore at the moment, so I deleted the folder, and let Splunk recreate it.
Now there are no entried in mongod.log, but it still fails to start.
So I tried to copy the "backup" back to the live folder, and I get the SSL errors, and it still wont start.

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

dgladkikh_splun — Tue, 06 Oct 2015 18:34:47 GMT

Could you try to run

$SPLUNK_HOME/splunk cmd mongod --version

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

dgladkikh_splun — Tue, 06 Oct 2015 19:06:51 GMT

Also could you validate all installed files with

splunk validate files

And show the output from both commands (including mongod version above)

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

las — Wed, 07 Oct 2015 06:27:17 GMT

Of course.
D:\Splunk\bin>splunk cmd mongod --version
db version v3.0.3-splunk
git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f
OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015

D:\Splunk\bin>splunk validate files
Validating installed files against hashes from 'D:\Splunk\splunk-6.3.0-a
a7d4b1ccb80-windows-64-manifest'
Could not open 'D:/Splunk/bin/mongod_legacy.exe': The system cannot find the fil
e specified.

File 'etc/splunk-launch.conf.default' changed or missing.

D:\Splunk\bin>

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

joerg_vonderohe — Wed, 07 Oct 2015 11:43:53 GMT

Same problem here, only difference is that we are using a linux splunk instance:
[user@server bin]#splunk validate files
Validating installed files against hashes from '/opt/splunk/splunk-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64-manifest'
All installed files intact.
[user@server bin]# splunk cmd mongod --version
db version v3.0.3-splunk
git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f
OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015

Maybe it's usefull

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

dgladkikh_splun — Wed, 07 Oct 2015 15:23:48 GMT

Ok. Missing files is a problem. Can you try to reinstall splunk to fix missing files? Or maybe get them from other instance?
Btw. Have you modified splunk launch conf file or is it really missed?

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

dgladkikh_splun — Wed, 07 Oct 2015 15:25:09 GMT

Problem can look similar, but dont think that it os the same problem. Do you have anything in mongod.log? Any errors in splunkd.log?

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

las — Wed, 07 Oct 2015 18:50:10 GMT

I think this might be a wrong tree to look at.
I tried running the same two commands on a working installation, on windows 2008 r8, with Splunk 6.3.0, where KVstore is working:

   D:\Splunk\bin>splunk validate files
            Validating installed files against hashes from 'D:\Splunk\splunk-6.3.0-aa7d4b1ccb80-windows-64-manifest'
    Could not open 'D:/Splunk/bin/mongod_legacy.exe': The system cannot find the file specified.



File 'etc/splunk-launch.conf.default' changed or missing.

D:\Splunk\bin>splunk cmd mongod --version
db version v3.0.3-splunk
git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f
OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015

why etc/splunk-launch.conf.default has changed, I have no idea, but it is an exact match to etc/splunk-launch.conf, and the change from "default" is a relocation of SPLUNK_DB.

I am not really comfortable wit reinstalling Splunk, as far as I remember, it involves uninstalling, and then installing Splunk again - and then make sure all configurations are back in order - yes, I can just make a backup of etc, but still... After all this is a production installation.

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

joerg_vonderohe — Thu, 08 Oct 2015 07:51:09 GMT

After the update only this event appears in the mongod.log:
2015-10-07T11:46:43.020Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-07T11:46:43.025Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

And the splunkd.log shows:
10-07-2015 13:46:43.034 +0200 ERROR MongodRunner - mongod exited abnormally (exit code 2, status: exited with code 2) - look at mongod.log to investigate.
10-07-2015 13:46:43.157 +0200 ERROR KVStoreConfigurationProvider - Could not get pint from mongod.
10-07-2015 13:46:44.159 +0200 ERROR MongodRunner - Did not get EOF from mongod after 1 second(s).
10-07-2015 13:46:57.445 +0200 INFO WatchedFile - Will begin reading at offset=15701880 for file='/opt/splunk/var/log/splunk/mongod.log'.

the splunkd.log hast GMT+2 Timezone, so there is a difference in the timestamps.

Greets
Jörg

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

joerg_vonderohe — Mon, 12 Oct 2015 07:34:54 GMT

Maybe you can try to disable ssl in the server.conf. Put it in the following path /opt/splunk/etc/system/local and write the following entrys in it:
[sslConfig]
enableSplunkdSSL = false

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

andreasz — Thu, 15 Oct 2015 08:50:22 GMT

I have exactly the same problem and an open ticket at Splunk Support.

I tried to upgrade from version 6.2.5 and 6.3 (OS: Windows Server 2008R2)

After the upgrade I get the following errors in mongod.log
2015-09-23T19:30:11.390Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-09-23T19:30:11.452Z I CONTROL Hotfix KB2731284 or later update is installed, no need to zero-out data files
2015-09-23T19:30:11.764Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

What I found is a similar known issue with Mongodb Version 3.0.4
https://jira.mongodb.org/browse/SERVER-19538
It was fixed in Version 3.0.6.
I don't get this error on my Deployment Server (same operating system: Windows Server 2008R2)

kind regards

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

las — Mon, 19 Oct 2015 12:45:56 GMT

I don't really want to disable SSL for Splunk, and I don't think that is the issue.
If I stop Splunkd, and delete the kvstore folder, I don't get the SSL errors, but mongod is stil unable to start. The only difference is, that it doesn't start with anything in mongod.log - absolutely no entries in that log.

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

joerg_vonderohe — Mon, 19 Oct 2015 13:01:28 GMT

SSL for the LoginWebsite is still working when you set this override. But with this Setting the kvstore is working for me. From the upgrade documentation:

enableSplunkdSSL = true|false
* Enables/disables SSL on the splunkd management port (8089) and KV store
port (8191).

Maybe you can replace the certificate in the following path with trusted ones: /opt/splunk/etc/auth

Greets
Jörg

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

mikaelbje — Tue, 20 Oct 2015 13:56:46 GMT

Any official response from Splunk yet?

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

dgladkikh_splun — Tue, 20 Oct 2015 23:49:04 GMT

Could you ping them and ask them to ping me (dgladkikh)? I would like to look on the diag.

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

andreasz — Wed, 21 Oct 2015 08:28:36 GMT

@dgladkikh
I discovered the problem directly after the upgrade and went back to version 6.2 (thanks to Vmware snapshot)
I didn't create a diag and could only sent the log files (splunkd.log, mongod.log, etc)
The problem appears only on our production system, so I have to wait for a maintenance window (probably Wednesday next week) to upgrade again and create a diag (Splunk support is waiting for that)