topic Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor? in Deployment Architecture

Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

LiquidTension — Tue, 24 Feb 2015 13:58:01 GMT

My cluster master is currently reporting there are 18 fixup tasks pending which is preventing me from meeting my Search Factor

8 with the category of Search Factor:

Cannot fix search count as the bucket hasn't rolled yet.

10 under replication factor:

Cannot replicate as bucket hasn't rolled yet.

Do people typically wait something like this out?

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

LiquidTension — Tue, 24 Feb 2015 17:01:11 GMT

The cluster certainly resolved the replication and search count events.

All Data is Searchable

Search Factor is Met
Replication Factor is Met

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

BP9906 — Tue, 21 Jul 2015 16:04:33 GMT

I keep restarting my cluster (2 peers) and I keep ending up in the same boat with the above error.

How did you resolve it?

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

LiquidTension — Tue, 21 Jul 2015 16:43:03 GMT

The cluster actually resolved the issue itself once the bucket rolled (within a few hours). In my case I just had to be patient and resist the urge that something had to be done by me. We originally started utilizing clustering in 5.0.3 and the feature is light years ahead now.

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

BP9906 — Tue, 21 Jul 2015 17:56:34 GMT

So my problem is resolved. The issue was that is restarted both cluster peers and still had the issue. Soon as I restarted the cluster master, the issue was resolved (after several minutes of waiting for the remaining items to complete).

Seems to me that when seeing this issue, restarting the cluster master resolves it.

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

dxu_splunk — Tue, 21 Jul 2015 19:28:17 GMT

normally it'll resolve itself after some time. if you don't mind restarts, try:

stop cluster master
restart peers
start cluster master

(and make sure your rf/sf <= 2)

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

lguinn2 — Mon, 17 Aug 2015 23:57:34 GMT

Use the cluster master to restart the peers. Restart the cluster master, and then do

splunk rolling-restart cluster-peers

See the docs here. It is not good practice to manually restart peers...

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

fulldanad — Tue, 19 Jan 2016 10:47:07 GMT

I downvoted this post because not a reply

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

rbal_splunk — Sat, 26 Mar 2016 00:56:38 GMT

In an indexer clustering environment, when the data is beginning to get ingested on one indexer (as hot Bucket) and indexed locally, it’s also streamed to other indexers depending on the Replication Factor. If due to some reason the stream of the bucket breaks, the cluster master will throw the error like above, which means data is starting to get indexed on originating indexer, but it is not being streamed to other, and as a result, out of compliance from Clusters RF and SF.

In Splunk Web, you will see messages for these buckets when you navigate to Setting>Indexer clustering>Indexes and click on button “Bucket Status” Look at the bucket name and you will see name like:
summary_forwarders~36~A1688691-A0AE-493C-A8C8-5300DEB73388

where
summary_forwarders > name of index
36 > bucket Id
A1688691-A0AE-493C-A8C8-5300DEB73388> GUID of the Peer where bucket originated

To get the Peer name from the GUID please run the search below on the C;luster Master

| rest /services/cluster/master/peers |table label as Peer  id as “last_string_is_guid”

=====output====

2p2262      https://127.0.0.1:24501/services/cluster/master/peers/A1688691-A0AE-493C-A8C8-5300DEB73388   
2p1262      https://127.0.0.1:24501/services/cluster/master/peers/E26613B0-8146-466F-B33B-37370B2C7197

Normally Such error messages will eventually clear out when the hot bucket rolls to warm, but that may take some time as Splunk buckets can grow up to 8Gb to 12Gb based on the index configuration.

To resolve this issue right away, you could Force a hot-to-warm roll of l buckets of the impacted index:

/splunk _internal call /data/indexes/YOUR_INDEX_HERE/roll-hot-buckets -auth admin:password

or curl:

curl -k -u admin:changeme https://INDEXER:MGMT_PORT/services/data/indexes/YOUR_INDEX_HERE/roll-hot-buckets -X POST

This need to be done of the index where bucket Originated.

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

fabiocaldas — Thu, 16 Jun 2016 15:48:06 GMT

I downvoted this post because it's not a real response

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

miyamaet — Mon, 25 Jul 2016 04:55:38 GMT

I downvoted this post because this post isn't a reply

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

anilyelmar — Mon, 30 Jan 2017 07:29:13 GMT

Splunk 6.4+ comes with easy option to roll the buckets remotely. You can ask your cluster master to identify cluster peer(remote indexer) and roll the buckets automatically.

Execute below curl
curl -k -u username:password https://localhost:/services/cluster/master/control/control/roll-hot-buckets -X POST -d "bucket_id=<...>"

Just make sure to put correct port of your cluster master and bucket_id as you see in error messages.

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

anilyelmar — Mon, 30 Jan 2017 07:32:12 GMT

Splunk 6.4+ comes with easy option to roll the buckets remotely. You can ask your cluster master to identify cluster peer(remote indexer) and roll the buckets automatically.

Execute below curl
curl -k -u username:password https://localhost:/services/cluster/master/control/control/roll-hot-buckets -X POST -d "bucket_id=<...>"

Just make sure to put correct port of your cluster master and bucket_id as you see in error messages.

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

jbarlow_splunk — Thu, 17 May 2018 08:19:41 GMT

In 6.5.3 and higher , buckets can also be rolled in the UI

Settings > the Distributed Environment group, click Indexer clustering. This takes you to the Master dashboard.
select the Indexes tab.
Click the Bucket Status
Click Bucket, Action > Roll

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

kishor_pinjark2 — Fri, 03 Aug 2018 09:13:50 GMT

On CM if I see there are more than 10 indexes having this issue, then I have to run below command for all indexes ?

curl -k -u admin:changeme https://HOST:PORT/services/data/indexes/YOUR_INDEX/roll-hot-buckets -X POST

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me from meeting my Search Factor?

gjanders — Fri, 03 Aug 2018 22:56:34 GMT

kishor_pinjarkar it looks like your asking the question on multiple old posts.

That will work but you will have to run it per index or 10 times.
There is also a roll bucket option under the bucket status section of the indexer clustering webpage in the GUI

Alternatively you can wait until the bucket rolls to warm at which point the bucket will fix itself

Re: Why is cluster master reporting "Cannot fix search count as the bucket hasn't rolled yet.", preventing me

Karthik — Tue, 28 Jul 2020 11:24:56 GMT

how can you fix this issue when you have thousands of buckets in pending? I have around 4800 buckets and i cannot go and execute roll command on each and every Index right? Can you give me any solution for this please?

Thanks for the answer.

@jbarlow_splunk