After upgrading Splunk, I see the following error (in bold) when Splunk is started:
Checking http port : open Checking mgmt port : open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port : open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _thefishbucket windows wineventlog winevents Done
New certs have been generated in '/opt/splunk/etc/auth'. Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Cannot find any source of hashes. Manifest file '(null)' not present? Problems were found, please review your files and move customizations to local All preliminary checks passed.
Starting splunk server daemon (splunkd)...
What is the source of this error and how can it be corrected?
This can happen if during the upgrade process, the existing manifest file was not replaced by the new one for the version you are now on. You will want to be sure that the version of splunk that you are running and is listed in $SPLUNKHOME/etc/splunk.version corresponds with the manifest file in $SPLUNKHOME. For example:
-rw-r--r--. 1 root root 1737038 Dec 1 14:22 splunk-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64-manifest
As you can see, the Splunk manifest file is for Splunk 6.3.0 but the version of Splunk on this system is 6.3.1. You will want to replace the manifest file with the correct one by copying it from the downloaded Splunk installation file.