We recently upgraded from 9.0.2 to 9.2.1 and started seeing some new errors on all indexer peer nodes as shown below.
--------
05-17-2024 14:35:07.225 +0000 ERROR DispatchCommandProcessor [949840 TcpChannelThread] - Search results may be incomplete, peer <indexer peer ip>'s search ended prematurely. Error = Peer <indexer peer hostname> will not return any results for this search, because the search head is using an outdated generation (search head gen_id=4626; peer gen_id=4969). This can be caused by the peer re-registering and the search head not yet updating to the latest generation. This should resolve itself shortly.
--------
The master has logs like below.
--------
splunkd.log.1:05-17-2024 12:06:59.491 +0000 WARN CMMaster [950487 CMMasterServiceThread] - got a large jump in gen_id suggestion=4921 current pending=1 reason=event=addPeerParallel Success guid=xxx adding_peers=7
--------
I tried suggestion actions from below discussion but no luck so far and ERROR is continuing for days now.
https://community.splunk.com/t5/Splunk-Enterprise/Why-am-I-receiving-this-error-quot-The-search-head...
It looks like the problem is with the primary master as we could see that when switching to the standby master, the error goes away.
Can anyone advise on this? What is a generation/gen_id and if there is a way to reset this to fix the issue?
