topic Re: Why does my search head only sometimes see a peer? in Deployment Architecture

Why does my search head only sometimes see a peer?

duke_splunk_adm — Tue, 22 Aug 2017 15:49:16 GMT

I have an index cluster where one peer is down (as in we're talking to Cisco). When I run a Splunk list search-server on one of our search-heads, sometimes it shows the IP of the down peer as Down, and other times it just doesn't list the downed peer at all.
I have an alert set up which monitors the results of the command that is sent to my SOC so they know when there is a connection problem, so I'm getting a lot of tickets. Why does the search-head see the peer sometimes and not others? How do I stop it? The box is entirely unreachable.
The setup is standard, where the search-head connects to the cluster master, and the indexer is still listed on the master because it hasn't been long enough for it to age out.

Re: Why does my search head only sometimes see a peer?

vasanthmss — Tue, 29 Sep 2020 15:25:03 GMT

Hi Duke_Splunk_admins,

During each search the list of search peers will be involved in the search action. Since you are aware of one peer is down you can remove from your indexer cluster till the time server backup.

Follow the below steps to remove the peer, (Read the notes before perform any, make sure master is in maintenance mode )

https://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Removepeerfrommasterlist

Best practice to take the search peer offline,

http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Takeapeeroffline

Hope this will helps you....

Re: Why does my search head only sometimes see a peer?

duke_splunk_adm — Wed, 23 Aug 2017 16:47:24 GMT

That's not exactly what I'm asking, although it is very helpful.
The behaviour is is: I have 4 peers, one of which is down. When I do a splunk list search-server sometimes I see 3 peers, all up, and sometimes I see 4, one of which is marked Down.
This throws off the monitoring system, as it appears to be flapping.

Re: Why does my search head only sometimes see a peer?

vasanthmss — Tue, 29 Aug 2017 20:55:23 GMT

Have you check your distsearch.conf?

Re: Why does my search head only sometimes see a peer?

duke_splunk_adm — Tue, 29 Aug 2017 21:11:24 GMT

It wouldn't show up in there, as that only has static search-servers. This is an index cluster member, which means that the search-servers are added dynamically by the cluster master. The cluster master, in turn, doesn't poll for indexers, but receives indexers as they say they are available.

Re: Why does my search head only sometimes see a peer?

duke_splunk_adm — Tue, 29 Aug 2017 21:11:55 GMT

And yes, I did check my distsearch.conf, thanks.

Re: Why does my search head only sometimes see a peer?

maraman_splunk — Tue, 29 Aug 2017 21:34:19 GMT

One possible scenario to explain your problem :
- specifications or configuration is different on that indexer -> he has more work to do OR it is less powerfull than others
- at some times, you are creating more search load that the indexer can handle -> load spike on this indexer
- this indexer is now slow to respond to sh -> the search head thinks it's down

So I would :
- investigate why that indexer is different than the others
- check search concurrency limit on sh and if needed lower it (lower shedule perc may be enough)
- spread your search load over time (schedule window, continous scheduling, skew, ..)