Splunk Enterprise

search head is using an outdated generation ERROR

jpillai
Path Finder

We recently upgraded from 9.0.2 to 9.2.1 and started seeing some new errors on all indexer peer nodes as shown below.

--------
05-17-2024 14:35:07.225 +0000 ERROR DispatchCommandProcessor [949840 TcpChannelThread] - Search results may be incomplete, peer <indexer peer ip>'s search ended prematurely. Error = Peer <indexer peer hostname> will not return any results for this search, because the search head is using an outdated generation (search head gen_id=4626; peer gen_id=4969). This can be caused by the peer re-registering and the search head not yet updating to the latest generation. This should resolve itself shortly.
--------

The master has logs like below.
--------

splunkd.log.1:05-17-2024 12:06:59.491 +0000 WARN CMMaster [950487 CMMasterServiceThread] - got a large jump in gen_id suggestion=4921 current pending=1 reason=event=addPeerParallel Success guid=xxx adding_peers=7
--------

I tried suggestion actions from below discussion but no luck so far and ERROR is continuing for days now.
https://community.splunk.com/t5/Splunk-Enterprise/Why-am-I-receiving-this-error-quot-The-search-head...

It looks like the problem is with the primary master as we could see that when switching to the standby master, the error goes away.

Can anyone advise on this? What is a generation/gen_id and if there is a way to reset this to fix the issue?

Labels (2)
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...