Splunk 6.6 upgrade seems to have permissions issues

[earlhelms]

After doing an rpm upgrade to 6.6 I'm having some pretty big issues that appear to be permission related.
Examples:
Failed to start KV Store process. See mongod.log and splunkd.log for details.

When I tried to search the splunkd log I received more errors...
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:config' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:hipmatch' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:system' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:threat' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:traffic' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'pan:threat' and lookup table 'threat_lookup'.

Fortunately, this is on my dev box.

[earlhelms · Accepted solution]

I resolved my own issue...
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n server -c zzzz-zzzz.zzz.zzzz.com -l 2048
Note: I edited the FQDN in the above example

[damode]

This will just recreate the default cert. In my case, we are using our own cert. Still getting the same error.
I checked file permission issue on Splunk.key file and that also seems fine.

[earlhelms]

A related link: https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html

Which led me to this...
[root@zzz]# tail /opt/splunk/var/log/splunk/mongod.log
2017-05-08T14:35:11.400Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2017-05-08T14:35:11.416Z F NETWORK The provided SSL certificate is expired or not yet valid.
2017-05-08T14:35:11.416Z I - Fatal Assertion 28652
2017-05-08T14:35:11.416Z I -
***aborting after fassert() failure

and this...
[root@zzzzz]# openssl x509 -enddate -noout -in ./server.pem
notAfter=Apr 13 20:57:57 2017 GMT

Still not entirely sure what to do