WARN StreamedSearch - Could not find bundles for search head provided checksum=

[rphillips_splk]

The message: WARN StreamedSearch - Could not find bundles for search head provided checksum=xxx keeps popping up on one search head in a search head pool from all search peers anytime a search (basic or complex) from that search head is initiated. Verified that all search heads, nfs server and indexers are synchronized to the same ntp server.

SPLUNK VERSION:
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

[rsimmons]

This was a known issue (SPL-97601) in bundle replication where skewed modtimes on temporary bundle files cause premature reaping and errors in distributed search. The workaround is to fix clock skew between indexers and NFS server hosting $SPLUNK_HOME/var/run/searchpeers. The issue has also been resolve in the latest release of Splunk.

[rphillips_splk · Accepted solution]

I recently encountered this issue and did not see any answers on how to resolve this message other than check ntp sync. but what if ntp is already in sync between search heads, nfs and indexers?

-check status of your search peers from the search head throwing the WARN message and see if any are in a failed state.
settings> distributed search> Search peers

(in this case one of the indexers had replication status of failed)

Although not a sophisticated answer this is what was done to correct / stop the message from occurring:

• restart splunk on the indexer that had replication status of failed
• restart splunk on the search head throwing the WARN message
• verify the indexer replication status is successful via the Search Head GUI >settings> distributed search> Search peers

to restart splunk from command line:
$SPLUNK_HOME/bin
./splunk restart

subsequently another set of messages (below) suspected to be related to the problem we were seeing also cleared:

ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle

WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxx: Directory not empty