Why is Cluster Master logging strange errors?

[AntoineDRN]

Hello Splunkers,

I'm here to ask you for a bit or your wisdom.

Context : This happens since the upgrade from 8.2.x to 9.0.3. The issue does not impact the platform (which is a dev platform). 

For a few weeks, I receive some errors in my interna logs that are really bizarre : 

03-02-2023 17:25:30.518 +0100 ERROR CMRepJob [49280 CMExecutorWorker-1] - job=CMSyncP2PJob bid=firewall_juniper~219~91483FBC-75D0-4410-9205-DE9DB070C3F3 my_guid=B3309FA8-4903-40B3-9E5D-B7BD712F6F70 my_rawport=xxxx my_usessl=1 ot_guid=91483FBC-75D0-4410-9205-DE9DB070C3F3 ot_hp=xxx.xxx.xxx.xxx :xxxx ot_rawport=xxxx ot_usessl=1 relative_path= custact=p2p_syncup getHttpReply failed; err: Connect Timeout

I thought it was a problem with few buckets that have been replicated wrong or not at all.

But, first, I don't have any warning on the Replcation Factor / Search Factor and, it is always the same 17 bids. So I guess it is not what I thought.

I don't have any other logs like this, every event that get in is correctly indexed / replicated.

Have you any idea of what's happening here ? And if it might be a problem, have you any idea of how I can fix that? 

Thanks for your time.

Best regards ! 

[richgalloway]

Looks like network connectivity problems between the indexer with GUID B3309FA8-4903-40B3-9E5D-B7BD712F6F70 and the one with GUID 91483FBC-75D0-4410-9205-DE9DB070C3F3

[AntoineDRN]

Hello @richgalloway ,

Yeah, it makes sense but I don't understand why it only raise this error for the 17 same things. 

I'll investigate further for the connectivity but I don't have any error other than this so I'm a bit confused. 

Thanks for your answer !