After upgrading Splunk Enterprise Security 3.3.0 to 4.0 for Splunk 6.3.1 on Windows 2012, why are we getting "...KV Store initialization has failed'?

[belka]

We were upgrading Splunk Enterprise Security 3.3.0 to ES 4.0 on Windows 2012 running Splunk 6.3.1. We ran into errors with Splunk ES 4.0 hanging up during the configuration phase. We did the manual work around contained in the article here: https://answers.splunk.com/answers/323361/upgrade-enterprise-security-from-33x-to-40-hangs-o.html

We got ES 4.0 to come up. The manual process allowed us to skip the ES 4.0 APPs that returned the "JSON not serializable" errors. Now that ES 4.0 is up and running, we are getting multiple errors of this sort: "External command based lookup 'es_notable_events' is not available because KV Store initialization has failed. "

We are trying to troubleshoot how we can over come this error by correcting what ever missing or altered configuration is causing this error.

[ChrisG]

Check this previous answer: https://answers.splunk.com/answers/215816/after-upgrading-splunk-app-for-microsoft-exchange.html. It might be that you need to restart the mongod process manually, or that there is a permissions issue on a file somewhere.

[belka]

I found that reference. We are working through it and trying to see if that will solve the issue.