Deployment Architecture

Search Peer status keeps changing every minute. Why?

DerekB
Splunk Employee
Splunk Employee

The search head server is giving various errors communicating with search peers/indexers. It keeps bouncing various error notifications every other minute from search peer to search peer.

  • Unable to distribute to peer named SH1 at uri https://SH1:8089 because peer has status = "Down". Unable to get bundle list -Unable to distribute to peer named SH2 at uri https://SH2:8089 because peer has status = "Down". Unable to get bundle list

Search Peer status keeps changing from minute to minute

SH1:8089 sh1 Up Successful
SH3:8089 sh3 Up Successful
SH4:8089 SH4 Down Initial
SH5:8089 sh5 Up Successful

sh2:8089 SH2 Up Successful

SH1:8089 SH1:8089 Down Initial
SH2:8089 sh2 Down Initial
SH3:8089 SH3 Up In progress
SH1:8089 SH1 Up In progress
SH2:8089 SH3 Authentication Failed Initial
sh4:8089 SH4 Up In progress

What the heck is going on?

Tags (3)
1 Solution

DerekB
Splunk Employee
Splunk Employee

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.

View solution in original post

DerekB
Splunk Employee
Splunk Employee

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...