Deployment Architecture

WARN StreamedSearch - Could not find bundles for search head provided checksum=

Splunk Employee
Splunk Employee

The message: WARN StreamedSearch - Could not find bundles for search head provided checksum=xxx keeps popping up on one search head in a search head pool from all search peers anytime a search (basic or complex) from that search head is initiated. Verified that all search heads, nfs server and indexers are synchronized to the same ntp server.

SPLUNK VERSION:
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

1 Solution

Splunk Employee
Splunk Employee

I recently encountered this issue and did not see any answers on how to resolve this message other than check ntp sync. but what if ntp is already in sync between search heads, nfs and indexers?

-check status of your search peers from the search head throwing the WARN message and see if any are in a failed state.
settings> distributed search> Search peers

(in this case one of the indexers had replication status of failed)

Although not a sophisticated answer this is what was done to correct / stop the message from occurring:

  • restart splunk on the indexer that had replication status of failed
  • restart splunk on the search head throwing the WARN message
  • verify the indexer replication status is successful via the Search Head GUI >settings> distributed search> Search peers

to restart splunk from command line:
$SPLUNK_HOME/bin
./splunk restart

subsequently another set of messages (below) suspected to be related to the problem we were seeing also cleared:

ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle

WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxx: Directory not empty

View solution in original post

Splunk Employee
Splunk Employee

This was a known issue (SPL-97601) in bundle replication where skewed modtimes on temporary bundle files cause premature reaping and errors in distributed search. The workaround is to fix clock skew between indexers and NFS server hosting $SPLUNK_HOME/var/run/searchpeers. The issue has also been resolve in the latest release of Splunk.

Splunk Employee
Splunk Employee

I recently encountered this issue and did not see any answers on how to resolve this message other than check ntp sync. but what if ntp is already in sync between search heads, nfs and indexers?

-check status of your search peers from the search head throwing the WARN message and see if any are in a failed state.
settings> distributed search> Search peers

(in this case one of the indexers had replication status of failed)

Although not a sophisticated answer this is what was done to correct / stop the message from occurring:

  • restart splunk on the indexer that had replication status of failed
  • restart splunk on the search head throwing the WARN message
  • verify the indexer replication status is successful via the Search Head GUI >settings> distributed search> Search peers

to restart splunk from command line:
$SPLUNK_HOME/bin
./splunk restart

subsequently another set of messages (below) suspected to be related to the problem we were seeing also cleared:

ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle

WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxx: Directory not empty

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!