Deployment Architecture

Search Peer status keeps changing every minute. Why?

DerekB
Splunk Employee
Splunk Employee

The search head server is giving various errors communicating with search peers/indexers. It keeps bouncing various error notifications every other minute from search peer to search peer.

  • Unable to distribute to peer named SH1 at uri https://SH1:8089 because peer has status = "Down". Unable to get bundle list -Unable to distribute to peer named SH2 at uri https://SH2:8089 because peer has status = "Down". Unable to get bundle list

Search Peer status keeps changing from minute to minute

SH1:8089 sh1 Up Successful
SH3:8089 sh3 Up Successful
SH4:8089 SH4 Down Initial
SH5:8089 sh5 Up Successful

sh2:8089 SH2 Up Successful

SH1:8089 SH1:8089 Down Initial
SH2:8089 sh2 Down Initial
SH3:8089 SH3 Up In progress
SH1:8089 SH1 Up In progress
SH2:8089 SH3 Authentication Failed Initial
sh4:8089 SH4 Up In progress

What the heck is going on?

Tags (3)
1 Solution

DerekB
Splunk Employee
Splunk Employee

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.

View solution in original post

DerekB
Splunk Employee
Splunk Employee

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...