Trying to run the Distributed Management Console, getting "ERROR KVStoreIntrospection - failed to get introspection data"

[howyagoin]

Hi,

When trying to Run DMC (ha ha), I've noticed that it's failing to get the introspection data from the kvstore.

05-22-2016 17:52:08.996 +1000 ERROR KVStorageProvider - An error occurred during the last operation ('serverStatus', domain: '2', code: '5'): Failed to connect to target host: 127.0.0.1:8191
05-22-2016 17:52:08.996 +1000 ERROR KVStoreIntrospection - failed to get introspection data
05-22-2016 17:52:09.072 +1000 ERROR KVStorageProvider - An error occurred during the last operation ('serverStatus', domain: '2', code: '5'): Failed to connect to target host: 127.0.0.1:8191
05-22-2016 17:52:09.072 +1000 ERROR KVStoreIntrospection - failed to get introspection data
05-22-2016 17:52:09.142 +1000 ERROR KVStorageProvider - An error occurred during the last operation ('serverStatus', domain: '2', code: '5'): Failed to connect to target host: 127.0.0.1:8191
05-22-2016 17:52:09.142 +1000 ERROR KVStoreIntrospection - failed to get introspection data

There must be something incredibly trivial that I'm missing here - I've set the Splunk Enterprise box to have a KV Store, but haven't created anything in a collections.conf (as it's not clear what needs to be set up for the distributed management console to work).

There's likely something obvious here...hints appreciated.

[howyagoin · Accepted solution]

I'm an idiot.

Problem was that mongod copied the server.pem which expired last week; as per another note in Answers I found:

$SPLUNK_HOME/bin/splunk createssl server-cert -d . -n server

And that created another server.pem; likely this is related to the tech-note which Splunk sent out talking about expiring certificates.

Restarted splunk after making a new server.pem, all good.