Deployment Architecture

Search Peer status keeps changing every minute. Why?

DerekB
Splunk Employee
Splunk Employee

The search head server is giving various errors communicating with search peers/indexers. It keeps bouncing various error notifications every other minute from search peer to search peer.

  • Unable to distribute to peer named SH1 at uri https://SH1:8089 because peer has status = "Down". Unable to get bundle list -Unable to distribute to peer named SH2 at uri https://SH2:8089 because peer has status = "Down". Unable to get bundle list

Search Peer status keeps changing from minute to minute

SH1:8089 sh1 Up Successful
SH3:8089 sh3 Up Successful
SH4:8089 SH4 Down Initial
SH5:8089 sh5 Up Successful

sh2:8089 SH2 Up Successful

SH1:8089 SH1:8089 Down Initial
SH2:8089 sh2 Down Initial
SH3:8089 SH3 Up In progress
SH1:8089 SH1 Up In progress
SH2:8089 SH3 Authentication Failed Initial
sh4:8089 SH4 Up In progress

What the heck is going on?

Tags (3)
1 Solution

DerekB
Splunk Employee
Splunk Employee

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.

View solution in original post

DerekB
Splunk Employee
Splunk Employee

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.

Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...