This condition can occur when a customer replaces an indexer in the cluster with another indexer. The bug filed for this issue is SPL-235527.
Workaround:
a.) restart CM to clear msg
or
b.) retain GUID of old indexer onto new indexer replacing old indexer
This condition can occur when a customer replaces an indexer in the cluster with another node.
The error indicates stale freeze notifications for buckets on decommissioned indexers are stuck in the frozen notification queue of the CM. The sequence of events would be the CM marks the bucket to be frozen prior to the indexer being decomm'ed. However it isn't sent out from the CM until after the indexer is decomm'ed.
This causes the log message with the old GUID:
101-30-2023 19:43:07.561 +0000 ERROR CMMaster [9370 CMMasterServiceThread] - sendQueuedFrozenNotifications: GUID not found in PeerMap, guid=6B12B9AC-E5AD-49F0-98F6-DA3DA4B9BA37
This can happen when upgrading indexers without putting the cluster in maintenance mode (which would stop bucket freezing) and the indexer being adding has a different GUIDs than the one being removed. Upping the frozen_notifications_per_batch (undocumented config in server.conf/[clustering]/frozen_notifications_per_batch=25 from the default 10 may help clear out this queue faster so that this is less likely to happen. However even with this the window where freeze notifications can be generated and the indexer removed prior to it being sent would still exist for this to happen.
This message can only be cleared by a restart of the CM currently. The message is more of an annoyance than an actual issue. Since the bucket has been moved to a new indexer, it will get a freeze notification and be frozen on the new indexer. So no functionality is lost, just a log message lingering.
Since the bucket will have already moved to a new indexer/GUID, it will tell the CM it owns that bucket and when the CM sees its new location it will issue another freeze notification for the new guid and the bucket will get properly frozen.
The bug filed for this issue is SPL-235527.
In the meantime, if a CM restart is undesirable you can use the same GUID of the old indexer and give that same GUID to the new indexer. GUID config location is at:
$SPLUNK_HOME/etc/instance.cfg
[general]
guid = xxx
