Fixup tasks over 600 after Upgrade to 8.2

[Stefanie]

After upgrading to 8.2 it seems that there are over 600 tasks to fix up in the Cluster Master.

We have a Cluster Master and four indexers. Two sites with two indexers per.

The Cluster master was placed into maintenance mode during the upgrade and we upgraded one site at a time while pointing the forwarders to the indexers that were not getting upgraded, and then swapped it for the upgrades.

There are 633 fixup tasks and have been there for about a day now.

Manually selecting roll bucket and trying to roll the bucket by the splunk _internal call command on the indexers to roll did nothing either.

What can I do to resolve this?

[somesoni2]

What are the different fixup reasons that you see on CM ?

[Stefanie]

Reasons being: Received shutdown notification from peer

Current Status: Cannot fix search count as the bucket hasn't rolled yet

[isoutamo]

Hi

can it be this issue https://community.splunk.com/t5/Splunk-Enterprise/bucket-status-an-change-masks-failed/m-p/553149 ?

r. Ismo

[Stefanie]

Hmm, no... but I'm sure it's related.

I guess I might have to put in a support ticket to get it looked at.

Restarting the cluster master seems to have some sort of effect on the number though.

After I restarted it for updates the number disappeared but it is slowly rising back up 😞 

[gjanders]

If the reason is bucket has not rolled yet that can happen during an upgrade...

You could roll buckets via UI or something like https://github.com/gjanders/Splunk/blob/master/bin/roll_and_resync_buckets.py

I use splunk cmd python to run it but it might need to be tweaked if you are not using a multi-site cluster