Hi,
Indexer cluster peers status are fluctuating few of the times from up to pending. I have verified, there is no resources shortage problems (memory/cpu) when the indexer cluster peers fluctuated. what could be the reason for fluctuating indexer peers?
splunk/Health.logs are showing below messages.
01-02-2022 02:00:11.893 +0000 INFO PeriodicHealthReporter - feature="Indexers" color=yellow indicator="missing_peers" due_to_threshold_value=1 measured_value=3 reason="The following peers are in transition: Indexer1(Pending), Indexer2(Pending), Indexer3(Pending).
" node_type=indicator node_path=splunkd.indexer_clustering.indexers.missing_peers
Source watchdog/watchdog.log are showing below messages
01-02-2022 02:15:19.01 +0000 ERROR Watchdog - No response received from IMonitoredThread=0x7fe9c63f70 within 8000 ms. Looks like thread name='CMMasterRemoteStorageThread' tid=28158 is busy !? Starting to trace with 8000 ms interval.
01-02-2022 02:16:23.12 +0000 INFO Watchdog - Stopping trace. Response for IMonitoredThread ptr=0x7fefb70 - thread name='CMMasterRemoteStorageThread' tid=28158 - finally received after 72049 ms (estimation only).
