Archive
Highlighted

After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Contributor

I had to make a non clean upgrade to 6.3.0.
The disk Splunk has its indexes on had to be expanded, and the way it happened was, that a new disk was allocated, the indexes copied to the new disk over a period of time, and then in the service window, a final copy, and then rename the disks. The only problem was, it was a copy, not a mirror, so Splunk was unable to start due to duplicate buckets (hot buckets moved to warm - but not removed in the copy)

After the upgrade I have troubles with Mongod.
In the Splunkd.log I get the following:

10-04-2015 03:20:44.705 +0200 ERROR MongodRunner - mongod exited abnormally (exit code 2, status: exited with code 2) - look at mongod.log to investigate.
10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not get pint from mongod.
10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.

In Mongod.log I get nothing.

Anybody having any experiences with this?

kind regards

Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Splunk Employee
Splunk Employee

Please take a look inside #SPLUNK_HOME/var/log/mongod.log to see what is wrong. It should give you more details about this problem.

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Contributor

That is part of the problem, There are no entries in mongod.log.
If I copy the original kvstore into the current repository, I get some expired certificate errors:

2015-10-04T01:20:44.330Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-10-04T01:20:44.330Z I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
2015-10-04T01:20:44.673Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

But if I clear the kvstore, I get no entries what so ever.

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Splunk Employee
Splunk Employee

What do you mean by copying original kvstore in current repository? What is original and what is repository?
Also, could you describe what was your steps to upgrade to 6.3? Is it Linux/Windows?

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Contributor

Sorry, I should have mentioned, it is on Windows 2008 R2 64 bit.
So I just ran the MSI file to upgrade.
As I mentioned, there were problems due to a diskexpansion, where the procedure was to make a copy of the folder containing the live indexes, in the service window, Splunk was stopped, and a last copy was made, the live disk was then renamed, and the new disk renamed to the old disks name.
That gives me a copy off the folder kvstore\mongod on the old live disk.
As there were problems with the copy being a copy, not a mirror (duplicate buckets), I had a cleaning job to do, and since Mongod did not start, I assumed it had similar problems.
We do not really use KVstore at the moment, so I deleted the folder, and let Splunk recreate it.
Now there are no entried in mongod.log, but it still fails to start.
So I tried to copy the "backup" back to the live folder, and I get the SSL errors, and it still wont start.

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Splunk Employee
Splunk Employee

Could you try to run


$SPLUNK_HOME/splunk cmd mongod --version

?

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Splunk Employee
Splunk Employee

Also could you validate all installed files with


splunk validate files

And show the output from both commands (including mongod version above)

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Contributor

Of course.
D:\Splunk\bin>splunk cmd mongod --version
db version v3.0.3-splunk
git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f
OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015

D:\Splunk\bin>splunk validate files
Validating installed files against hashes from 'D:\Splunk\splunk-6.3.0-a
a7d4b1ccb80-windows-64-manifest'
Could not open 'D:/Splunk/bin/mongod_legacy.exe': The system cannot find the fil
e specified.

File 'etc/splunk-launch.conf.default' changed or missing.

D:\Splunk\bin>

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Splunk Employee
Splunk Employee

Ok. Missing files is a problem. Can you try to reinstall splunk to fix missing files? Or maybe get them from other instance?
Btw. Have you modified splunk launch conf file or is it really missed?

0 Karma
Highlighted

Re: After upgrading to Splunk 6.3.0, why am I getting the following Mongod errors?

Contributor

I think this might be a wrong tree to look at.
I tried running the same two commands on a working installation, on windows 2008 r8, with Splunk 6.3.0, where KVstore is working:

   D:\Splunk\bin>splunk validate files
            Validating installed files against hashes from 'D:\Splunk\splunk-6.3.0-aa7d4b1ccb80-windows-64-manifest'
    Could not open 'D:/Splunk/bin/mongod_legacy.exe': The system cannot find the file specified.



File 'etc/splunk-launch.conf.default' changed or missing.

D:\Splunk\bin>splunk cmd mongod --version
db version v3.0.3-splunk
git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f
OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015

why etc/splunk-launch.conf.default has changed, I have no idea, but it is an exact match to etc/splunk-launch.conf, and the change from "default" is a relocation of SPLUNK_DB.

I am not really comfortable wit reinstalling Splunk, as far as I remember, it involves uninstalling, and then installing Splunk again - and then make sure all configurations are back in order - yes, I can just make a backup of etc, but still... After all this is a production installation.

0 Karma