- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The log is repeating at sub-second intervals:
2017-10-27T20:44:53.389Z I REPL [ReplicationExecutor] Error in heartbeat request to shccaptain:8191; InvalidReplicaSetConfig Our replica set configuration is invalid or does not include us
The kvstore appears to be healthy otherwise.
> curl -sku admin:password https://shcmember:8089/services/server/info | grep -i kv
<s:key name="kvStoreStatus">ready</s:key>
> curl -sku admin:password https://shccaptain:8089/services/server/info | grep -i kv
<s:key name="kvStoreStatus">ready</s:key>
What's the problem? What's the fix?
EDIT: The error started after creating a collection by running this curl command:
curl -ku admin -d name=userid https://shcmember:8089/servicesNS/nobody/alpha_search/storage/collections/config
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk Support provided this solution:
on the SHC captain, splunk stop
, then splunk clean kvstore --local
, accept the scary warning message, and finally restart splunk. The errors have subsided and as far as I can tell, no damage was done.
Hope this helps a future admin.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk Support provided this solution:
on the SHC captain, splunk stop
, then splunk clean kvstore --local
, accept the scary warning message, and finally restart splunk. The errors have subsided and as far as I can tell, no damage was done.
Hope this helps a future admin.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks!
Has the same problem today and changing the captain did not help.
But running this on the problematic host, solved it.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
A slightly safer way to do this is just transfer the captaincy to another member of the cluster so that it rebuilds the replicaSet configuration for mongod. No stops or cleans required.
https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Transfercaptain#Change_the_captain
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


HI
can you share collection configuration?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The collection was configured as above. No other settings applied. These are the defaults as far as I can tell:
[userid]
profilingEnabled = false
profilingThresholdMs = 1000
replicate = false
replication_dump_maximum_file_size = 10240
replication_dump_strategy = auto
type = undefined
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


Hi
Can you please try below stanza?
[userid]
replicate = true
enforceTypes = false
I have removed other configuration bcoz those have assigned to default values. Make sure about configuration would same in entire cluster.
Thanks
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I should say that i have many other collections running, with exactly the same creation procedure. None triggered this error repeating. That said, I added the configs you suggested above to this single collection via the REST API. Confirmed via btool. The errors continue.
