Deployment Architecture

Why am I getting repeated message "WARN DispatchThread - Can not download search.log from peer '' because no remote sid was received."?

ncsantucci
Path Finder

WARN DispatchThread - Can not download search.log from peer '<FQDN>' because no remote sid was received.

This warning shows up on a single ad hoc search head, once for each indexer <FQDN> which is configured as a search peer, repeating every 20-30 seconds.

This is a lot of noise in splunkd.log and I would like to get to the bottom of this because I believe this degrades performance even being a physical search head due to error frequency.

0 Karma

ncsantucci
Path Finder

It turns out that this particular search head was on a different VLAN than all my other ad-hoc search heads and search head pool members. Furthermore, there was a network issue (router issues) between the search head in question and the target indexers.
When the network connectivity was restored, I re-entered the password for the search peers (indexers) and then everything was fine.

0 Karma

ncsantucci
Path Finder

Correction: I saw the error re-appear, so the exact root cause it unknown!

0 Karma
Get Updates on the Splunk Community!

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...

Modern way of developing distributed application using OTel

Recently, I had the opportunity to work on a complex microservice using Spring boot and Quarkus to develop a ...