Why is the search head status is flickering in the indexer clustering?`

[AbilashSe]

Search heads are up and healthy, but there is a fluctuation in the Search head status in the indexer clustering.

Can anyone please let me know what the problem is?

Regards,
Abi

[rahul_bhatia]

I have encountered the exact same issue with one of our SHC.

We have 2 SHCs connected to our indexers. For one of the SHCs, the SHC members keep flickering between 'Up' and 'Down' status on the 'Indexer Clustering' page.

For members of both SHCs, 'generation_poll_interval' defaults to 5. The flickering status only happens for one SHC, not the other.

Any further inputs on this behavior would be appreciated.

Thanks

[richgalloway]

@rahul_bhatia This thread is nearly two years old with little activity. Consider posting a new question.

[rphillips_splk]

try increasing generation_poll_interval on the SHs in server.conf

[clustering]
generation_poll_interval = <positive integer>
    * Only valid if mode=master or mode=searchhead
    * Determines how often the searchhead polls the master for generation
      information.
    * Defaults to 60s.

some older docs state default is 60s but actual config is set to 5 sec.

[root@rplinux bin]#
./splunk btool server list --debug | grep generation_poll_interval

/opt/splunk/etc/system/default/server.conf generation_poll_interval = 5

you can increase generation_poll_interval to 60 sec and see if it stabilizes the SH status from cluster master UI.

[AbilashSe]

Hi rphillips [Splunk],

Thanks for your answer..!!

I've tried with the above option , but its ended up with an error when i restarted the splunkd after the config change in the server.conf in the SH.
I've set the mode as searchhead and poll_interval value was 5.

Invalid key in stanza [clustering] in /pkg/splunk/etc/system/default/server.conf, line 397: only valid for mode (value: master or mode=searchhead).

Would you please check and let me know if there are changes needs to be done.

Thank you,
Abi

[rphillips_splk]

Hi Abi,
- what version are you on?
- can you send the output after running this command on your SH
$SPLUNK_HOME/bin
./splunk btool server list --debug clustering