Deployment Architecture

Why is the search head status is flickering in the indexer clustering?`

AbilashSe
Explorer

Search heads are up and healthy, but there is a fluctuation in the Search head status in the indexer clustering.

Can anyone please let me know what the problem is?

Regards,
Abi

0 Karma

rahul_bhatia
New Member

I have encountered the exact same issue with one of our SHC.

We have 2 SHCs connected to our indexers. For one of the SHCs, the SHC members keep flickering between 'Up' and 'Down' status on the 'Indexer Clustering' page.

For members of both SHCs, 'generation_poll_interval' defaults to 5. The flickering status only happens for one SHC, not the other.

Any further inputs on this behavior would be appreciated.

Thanks

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@rahul_bhatia This thread is nearly two years old with little activity. Consider posting a new question.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

rphillips_splun
Splunk Employee
Splunk Employee

try increasing generation_poll_interval on the SHs in server.conf

[clustering]
generation_poll_interval = <positive integer>
    * Only valid if mode=master or mode=searchhead
    * Determines how often the searchhead polls the master for generation
      information.
    * Defaults to 60s.

some older docs state default is 60s but actual config is set to 5 sec.

[root@rplinux bin]#
./splunk btool server list --debug | grep generation_poll_interval

/opt/splunk/etc/system/default/server.conf generation_poll_interval = 5

you can increase generation_poll_interval to 60 sec and see if it stabilizes the SH status from cluster master UI.

0 Karma

AbilashSe
Explorer

Hi rphillips [Splunk],

Thanks for your answer..!!

I've tried with the above option , but its ended up with an error when i restarted the splunkd after the config change in the server.conf in the SH.
I've set the mode as searchhead and poll_interval value was 5.

Invalid key in stanza [clustering] in /pkg/splunk/etc/system/default/server.conf, line 397: only valid for mode (value: master or mode=searchhead).

Would you please check and let me know if there are changes needs to be done.

Thank you,
Abi

0 Karma

rphillips_splun
Splunk Employee
Splunk Employee

Hi Abi,
- what version are you on?
- can you send the output after running this command on your SH
$SPLUNK_HOME/bin
./splunk btool server list --debug clustering

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!