Deployment Architecture

Why are the mask changes failing?

tkw03
Communicator

Hello

We recently migrated our CM to a new clean host. After migration almost everything is good but I have a few errors relating to masking changes and it seems to only be on a single host:

 

 

04-26-2022 08:46:36.578 -0400 INFO  CMRepJob - running job=CMChangeMasksJob guid=4769183B-D1C2-4906-AFE5-7A799E2A3B5D number-of-changes=30 genid=91958
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~220~4C96BDC6-0710-452F-9514-50C631A94286 transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 WARN  CMMaster - mask change failed, reverting back bid=server_oracle~239~FBCEF346-BB10-406D-BD87-087A1DC6F5BF mask=0 searchState=Searchable status=Complete
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~239~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 WARN  CMMaster - mask change failed, reverting back bid=windows~29197~FBCEF346-BB10-406D-BD87-087A1DC6F5BF mask=0 searchState=Searchable status=Complete
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows~29197~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 WARN  CMMaster - mask change failed, reverting back bid=windows_events~3932~7AD6523B-A4F1-4B0E-8C77-511FD5FA2286 mask=0 searchState=Searchable status=Complete
04-26-2022 08:46:36.726 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows_events~3932~7AD6523B-A4F1-4B0E-8C77-511FD5FA2286 transitioning from=SearchablePendingMask to=Searchable oldmask=0xffffffffffffffff newmask=0x0 reason="mask change failed, reverting back"
04-26-2022 08:46:36.726 -0400 INFO  CMMaster - event=commitGenerationFailure pendingGen=91958 requesterReason=changeBucketMasks failureReason='event=checkDirtyBuckets first unmet bid=cim_modactions~196~5678E53C-8999-4C69-8032-C55BDB86745E'
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows_events~3932~7AD6523B-A4F1-4B0E-8C77-511FD5FA2286 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=windows~29197~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~239~FBCEF346-BB10-406D-BD87-087A1DC6F5BF transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~220~4C96BDC6-0710-452F-9514-50C631A94286 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~215~F448A588-91B9-47D9-99DB-B1CE27CA51AA transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~203~7423238A-3907-4BA1-A8A6-8A9A126A6B21 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~49~BB39BC9E-D7DA-4934-8D4A-FC7DD9C982B4 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_oracle~2~A98C1984-B48A-4B58-8D83-B6D1FAA01F08 transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:37.724 -0400 INFO  CMPeer - peer=4769183B-D1C2-4906-AFE5-7A799E2A3B5D peer_name=host01 bid=server_ad~20~AC4D9A8B-995F-4043-BBE1-1FD61BFA3BEB transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0xffffffffffffffff reason="fixup searchable mask"
04-26-2022 08:46:45.718 -0400 INFO  CMMaster - event=commitGenerationFailure pendingGen=91958 requesterReason=changeBucketMasks failureReason='event=checkDirtyBuckets first unmet bid=cim_modactions~196~5678E53C-8999-4C69-8032-C55BDB86745E'

 

 

 

 

these issues are the only thing keeping our cluster from being completely migrated/fixed. Thanks for the help!

Todd Waller

Labels (2)
0 Karma
1 Solution

tkw03
Communicator

…….As usual fixed it myself.

After some exploration I fund that the master saw the bucket/copies as successfully existing in 2 places (rep/search factor 2/2). So I had a hunch that these may be copies and if I deleted the copy from the host that was throwing the error, even if the bucket did delete it would resync anyway. So I deleted one and it reduced the issue count and made the test index fully searchable, so I continued.

All data is now searchable and search and rep factors are met. Issue resolved.

 

View solution in original post

tkw03
Communicator

…….As usual fixed it myself.

After some exploration I fund that the master saw the bucket/copies as successfully existing in 2 places (rep/search factor 2/2). So I had a hunch that these may be copies and if I deleted the copy from the host that was throwing the error, even if the bucket did delete it would resync anyway. So I deleted one and it reduced the issue count and made the test index fully searchable, so I continued.

All data is now searchable and search and rep factors are met. Issue resolved.

 

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...