Indexer cluster peers status are fluctuating?

[90522prakash]

Hi,

Indexer cluster peers status are fluctuating few of the times from up to pending. I have verified, there is no resources shortage problems (memory/cpu) when the indexer cluster peers fluctuated. what could be the reason for fluctuating indexer peers?
 
splunk/Health.logs are showing below messages.

01-02-2022 02:00:11.893 +0000 INFO PeriodicHealthReporter - feature="Indexers" color=yellow indicator="missing_peers" due_to_threshold_value=1 measured_value=3 reason="The following peers are in transition: Indexer1(Pending), Indexer2(Pending), Indexer3(Pending).
" node_type=indicator node_path=splunkd.indexer_clustering.indexers.missing_peers

Source watchdog/watchdog.log are showing below messages

01-02-2022 02:15:19.01 +0000 ERROR Watchdog - No response received from IMonitoredThread=0x7fe9c63f70 within 8000 ms. Looks like thread name='CMMasterRemoteStorageThread' tid=28158 is busy !? Starting to trace with 8000 ms interval.

01-02-2022 02:16:23.12 +0000 INFO Watchdog - Stopping trace. Response for IMonitoredThread ptr=0x7fefb70 - thread name='CMMasterRemoteStorageThread' tid=28158 - finally received after 72049 ms (estimation only).

[askargbo]

I am following up on this issue. Was it resolved? If so, what was the solution, as I am experiencing similar issue. We know it is not a networking issue on our end after going through some network testing between indexers and CM. All ports that need to be opened between these components are opened and communications is between 0.5 to 1ms.

[isoutamo]

I think that this time for creating a support case to splunk.