I had to make a non clean upgrade to 6.3.0.
The disk Splunk has its indexes on had to be expanded, and the way it happened was, that a new disk was allocated, the indexes copied to the new disk over a period of time, and then in the service window, a final copy, and then rename the disks. The only problem was, it was a copy, not a mirror, so Splunk was unable to start due to duplicate buckets (hot buckets moved to warm - but not removed in the copy)
After the upgrade I have troubles with Mongod.
In the Splunkd.log I get the following:
10-04-2015 03:20:44.705 +0200 ERROR MongodRunner - mongod exited abnormally (exit code 2, status: exited with code 2) - look at mongod.log to investigate. 10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not get pint from mongod. 10-04-2015 03:20:55.328 +0200 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.
In Mongod.log I get nothing.
Anybody having any experiences with this?
Please take a look inside #SPLUNK_HOME/var/log/mongod.log to see what is wrong. It should give you more details about this problem.
That is part of the problem, There are no entries in mongod.log.
If I copy the original kvstore into the current repository, I get some expired certificate errors:
2015-10-04T01:20:44.330Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter 2015-10-04T01:20:44.330Z I CONTROL Hotfix KB2731284 or later update is not installed, will zero-out data files 2015-10-04T01:20:44.673Z I CONTROL dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
But if I clear the kvstore, I get no entries what so ever.
What do you mean by copying original kvstore in current repository? What is original and what is repository?
Also, could you describe what was your steps to upgrade to 6.3? Is it Linux/Windows?
Sorry, I should have mentioned, it is on Windows 2008 R2 64 bit.
So I just ran the MSI file to upgrade.
As I mentioned, there were problems due to a diskexpansion, where the procedure was to make a copy of the folder containing the live indexes, in the service window, Splunk was stopped, and a last copy was made, the live disk was then renamed, and the new disk renamed to the old disks name.
That gives me a copy off the folder kvstore\mongod on the old live disk.
As there were problems with the copy being a copy, not a mirror (duplicate buckets), I had a cleaning job to do, and since Mongod did not start, I assumed it had similar problems.
We do not really use KVstore at the moment, so I deleted the folder, and let Splunk recreate it.
Now there are no entried in mongod.log, but it still fails to start.
So I tried to copy the "backup" back to the live folder, and I get the SSL errors, and it still wont start.
Also could you validate all installed files with
splunk validate files
And show the output from both commands (including mongod version above)
D:\Splunk\bin>splunk cmd mongod --version
db version v3.0.3-splunk
git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f
OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015
D:\Splunk\bin>splunk validate files
Validating installed files against hashes from 'D:\Splunk\splunk-6.3.0-a
Could not open 'D:/Splunk/bin/mongod_legacy.exe': The system cannot find the fil
File 'etc/splunk-launch.conf.default' changed or missing.
Ok. Missing files is a problem. Can you try to reinstall splunk to fix missing files? Or maybe get them from other instance?
Btw. Have you modified splunk launch conf file or is it really missed?
I think this might be a wrong tree to look at.
I tried running the same two commands on a working installation, on windows 2008 r8, with Splunk 6.3.0, where KVstore is working:
D:\Splunk\bin>splunk validate files Validating installed files against hashes from 'D:\Splunk\splunk-6.3.0-aa7d4b1ccb80-windows-64-manifest' Could not open 'D:/Splunk/bin/mongod_legacy.exe': The system cannot find the file specified. File 'etc/splunk-launch.conf.default' changed or missing. D:\Splunk\bin>splunk cmd mongod --version db version v3.0.3-splunk git version: f0547c3bbc1fc8eb4325ce6c158edfe4ef45733f OpenSSL version: OpenSSL 1.0.2d-fips 9 Jul 2015
why etc/splunk-launch.conf.default has changed, I have no idea, but it is an exact match to etc/splunk-launch.conf, and the change from "default" is a relocation of SPLUNK_DB.
I am not really comfortable wit reinstalling Splunk, as far as I remember, it involves uninstalling, and then installing Splunk again - and then make sure all configurations are back in order - yes, I can just make a backup of etc, but still... After all this is a production installation.