Deployment Architecture

WARN StreamedSearch - Could not find bundles for search head provided checksum=

rphillips_splk
Splunk Employee
Splunk Employee

The message: WARN StreamedSearch - Could not find bundles for search head provided checksum=xxx keeps popping up on one search head in a search head pool from all search peers anytime a search (basic or complex) from that search head is initiated. Verified that all search heads, nfs server and indexers are synchronized to the same ntp server.

SPLUNK VERSION:
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

I recently encountered this issue and did not see any answers on how to resolve this message other than check ntp sync. but what if ntp is already in sync between search heads, nfs and indexers?

-check status of your search peers from the search head throwing the WARN message and see if any are in a failed state.
settings> distributed search> Search peers

(in this case one of the indexers had replication status of failed)

Although not a sophisticated answer this is what was done to correct / stop the message from occurring:

  • restart splunk on the indexer that had replication status of failed
  • restart splunk on the search head throwing the WARN message
  • verify the indexer replication status is successful via the Search Head GUI >settings> distributed search> Search peers

to restart splunk from command line:
$SPLUNK_HOME/bin
./splunk restart

subsequently another set of messages (below) suspected to be related to the problem we were seeing also cleared:

ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle

WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxx: Directory not empty

View solution in original post

rsimmons
Splunk Employee
Splunk Employee

This was a known issue (SPL-97601) in bundle replication where skewed modtimes on temporary bundle files cause premature reaping and errors in distributed search. The workaround is to fix clock skew between indexers and NFS server hosting $SPLUNK_HOME/var/run/searchpeers. The issue has also been resolve in the latest release of Splunk.

rphillips_splk
Splunk Employee
Splunk Employee

I recently encountered this issue and did not see any answers on how to resolve this message other than check ntp sync. but what if ntp is already in sync between search heads, nfs and indexers?

-check status of your search peers from the search head throwing the WARN message and see if any are in a failed state.
settings> distributed search> Search peers

(in this case one of the indexers had replication status of failed)

Although not a sophisticated answer this is what was done to correct / stop the message from occurring:

  • restart splunk on the indexer that had replication status of failed
  • restart splunk on the search head throwing the WARN message
  • verify the indexer replication status is successful via the Search Head GUI >settings> distributed search> Search peers

to restart splunk from command line:
$SPLUNK_HOME/bin
./splunk restart

subsequently another set of messages (below) suspected to be related to the problem we were seeing also cleared:

ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle

WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxx: Directory not empty

Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Spunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...