I have encountered the exact same issue with one of our SHC.
We have 2 SHCs connected to our indexers. For one of the SHCs, the SHC members keep flickering between 'Up' and 'Down' status on the 'Indexer Clustering' page.
For members of both SHCs, 'generation_poll_interval' defaults to 5. The flickering status only happens for one SHC, not the other.
Any further inputs on this behavior would be appreciated.
try increasing generation_poll_interval on the SHs in server.conf
[clustering] generation_poll_interval = <positive integer> * Only valid if mode=master or mode=searchhead * Determines how often the searchhead polls the master for generation information. * Defaults to 60s.
some older docs state default is 60s but actual config is set to 5 sec.
./splunk btool server list --debug | grep generation_poll_interval
/opt/splunk/etc/system/default/server.conf generation_poll_interval = 5
you can increase generation_poll_interval to 60 sec and see if it stabilizes the SH status from cluster master UI.
Hi rphillips [Splunk],
Thanks for your answer..!!
I've tried with the above option , but its ended up with an error when i restarted the splunkd after the config change in the server.conf in the SH.
I've set the mode as searchhead and poll_interval value was 5.
Invalid key in stanza [clustering] in /pkg/splunk/etc/system/default/server.conf, line 397: only valid for mode (value: master or mode=searchhead).
Would you please check and let me know if there are changes needs to be done.