Why are the mask changes failing?

[tkw03]

Hello

We recently migrated our CM to a new clean host. After migration almost everything is good but I have a few errors relating to masking changes and it seems to only be on a single host:

04-26-2022 08:46:36.578 -0400 INFO  CMRepJob - running job=CMChangeMasksJob guid=4769183B-D1C2-4906-AFE5-7A799E2A3B5D number-of-changes=30 genid=91958
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~220~4C96BDC6-0710-452F-9514-50C631A94286 transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 WARN  CMMaster - mask change failed, reverting back bid=server_oracle~239~FBCEF346-BB10-406D-BD87-087A1DC6F5BF mask=0 searchState=Searchable status=Complete
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~239~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 WARN  CMMaster - mask change failed, reverting back bid=windows~29197~FBCEF346-BB10-406D-BD87-087A1DC6F5BF mask=0 searchState=Searchable status=Complete
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows~29197~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 WARN  CMMaster - mask change failed, reverting back bid=windows_events~3932~7AD6523B-A4F1-4B0E-8C77-511FD5FA2286 mask=0 searchState=Searchable status=Complete
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows_events~3932~7AD6523B-A4F1-4B0E-8C77-511FD5FA2286 transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 INFO  CMMaster - event=commitGenerationFailure pendingGen=91958 requesterReason=changeBucketMasks failureReason='event=checkDirtyBuckets first unmet bid=cim_modactions~196~5678E53C-8999-4C69-8032-C55BDB86745E'
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows_events~3932~7AD6523B-A4F1-4B0E-8C77-511FD5FA2286 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows~29197~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~239~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~220~4C96BDC6-0710-452F-9514-50C631A94286 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~215~F448A588-91B9-47D9-99DB-B1CE27CA51AA transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~203~7423238A-3907-4BA1-A8A6-8A9A126A6B21 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~49~BB39BC9E-D7DA-4934-8D4A-FC7DD9C982B4 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~2~A98C1984-B48A-4B58-8D83-B6D1FAA01F08 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_ad~20~AC4D9A8B-995F-4043-BBE1-1FD61BFA3BEB transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:45.718 -0400 INFO  CMMaster - event=commitGenerationFailure pendingGen=91958 requesterReason=changeBucketMasks failureReason='event=checkDirtyBuckets first unmet bid=cim_modactions~196~5678E53C-8999-4C69-8032-C55BDB86745E'

these issues are the only thing keeping our cluster from being completely migrated/fixed. Thanks for the help!

Todd Waller

[tkw03 · Accepted solution]

…….As usual fixed it myself.

After some exploration I fund that the master saw the bucket/copies as successfully existing in 2 places (rep/search factor 2/2). So I had a hunch that these may be copies and if I deleted the copy from the host that was throwing the error, even if the bucket did delete it would resync anyway. So I deleted one and it reduced the issue count and made the test index fully searchable, so I continued.

All data is now searchable and search and rep factors are met. Issue resolved.