Why is an indexer in a cluster reporting "CMMessages - got genid thats invalid or out of range, setting to INVALID_GENID"?

[gpaks]

Hello,

I have an indexer node running Splunk Version 6.3.2 (build aaff59bb082c) that constantly outputs the following "WARN" to the "/apps/splunk/splunk/var/log/splunk/splunkd.log" log file:

CMSlave - handleHeartbeatDone: successful heartbeat and re-add not received but proxy is in disconnected state. Forcing re-add.

CMMessages - got genid thats invalid or out of range, setting to INVALID_GENID, jn=18446744073709551616.000000

In addition:
1. The same indexer node indexes the data pushed to it, and it is searchable in Splunk Web by browsing to http://{indexer_host}:8000
2. The same indexer node can't search data that was indexed into other indexers in the cluster. And the other Indexers can't search the data that was indexed on the the problematic node mentioned above.

Before the issue appeared, no configuration changes were done.

I'd be very grateful if anyone could provide assistance, explain what causing the above WARN messages, or just point me into the right direction to investigate the cause.

Thank you in advance,

Greg

[dshakespeare_sp]

If you are seeing errror in your Clustered Indexers splunkd.log:

WARN  CMMessages - got genid thats invalid or out of range, setting to INVALID_GENID, jn=18446744073709551616.000000

This typicaly means that the Cluster Master has been restarted, but cannot bring the cluster online because Replication Factor has not been met due to the required number of Index Peers being unavailable.

There should be a corrosponding error in splunkd.log on the Cluster Master:

INFO  CMMaster - event=commitGenerationFailure pendingGen=13 requesterReason=addPeerSuccess guid= <GUID> failureReason='Cluster has only 'x' peers (waiting for 'y' peers to join the cluster).

The errors will stop appearing when the required number of peers to meet Replication factor is online

[SierraX]

Hi,
also haven't hear about this ClusterMaster (CM) error Message. Could it be a Custom one from a 3rd party App?

But for the additional 2. 😞 The Indexers in a indexing Cluster typically can only search on own data... maybe Its changeable when you connect the cluster as search peer : But I'm pretty sure you should not do this
For searching on the cluster: The Cluster Master is always a Searchhead for his own cluster too

Kind Regards
SierraX

[gpaks]

Hi SierraX,

Thank you for your replay.

In addition, I've noticed the following message in the "Messages" menu in the Splunk WebUI (Upper right corner):

One or more replicated indexes may not be fully searchable. Some search results may be incomplete or duplicated as we try to fix up your cluster. For more information, check the cluster manager page on the master - splunkd URI: https://{splunk_master_node}:8089.

What resulted the above message? and what should I look for in the "https://{splunk_master_node}:8089" URL?

A bit confused..