The search head server is giving various errors communicating with search peers/indexers. It keeps bouncing various error notifications every other minute from search peer to search peer.
SH1:8089 sh1 Up Successful
SH3:8089 sh3 Up Successful
SH4:8089 SH4 Down Initial
SH5:8089 sh5 Up Successful
SH1:8089 SH1:8089 Down Initial
SH2:8089 sh2 Down Initial
SH3:8089 SH3 Up In progress
SH1:8089 SH1 Up In progress
SH2:8089 SH3 Authentication Failed Initial
sh4:8089 SH4 Up In progress
What the heck is going on?
There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.
There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.