What does this mean? ERROR IntrospectionGenerator:resource_usage - KVStorageProvider - Internal read failed with error code '13436' and message 'not master or secondary; cannot currently read from this replSet member'

[jgoddard]

I have been having issues with the KV Store under 6.2.0 and 6.2.1, mostly in my search head cluster. The first issue I have is that the KV store doesn't always get stopped by a splunk restart, resulting in the mongod process being left running and the KV port bound. This prevents splunk from restarting, which has resulted in a couple of cluster failures/issues. I see an existing unanswered question about this one (http://answers.splunk.com/answers/191451/splunk-62-mongod-does-not-seem-t).

The error in the subject line i am only seeing on one of three search head cluster members. Has anyone seen this before and even better, what do I do to resolve it?

[amirofmn]

Curious if a solution was ever found for this particular error message.

[bpaul_splunk]

One way this message can be displayed is if the number of search head cluster members is greater than the KVStore instances. Please check to make sure KVStore is enabled on all the search head cluster members.