maxWarmDBCount limit exceeded

[chrisduimstra]

I am using different storage drives for hot/warm and cold storage. The Fire Brigade app was reporting a total of 524 buckets for index A with a limit of 300. I verified on the storage drive that there are 524 buckets. The indexes.conf file has the following settings which should trigger the rotation policy, maxWarmDBCount = 300 and rotatePeriodInSecs = 60. Why are the buckets not rolling from warm to cold?

[lycollicott]

Is the cold location correct? Accessible? Have the correct permissions and ownership?

[chrisduimstra]

I ran a search for bucketmover and discovered splunk did not have permissions to remove inflight-db... folders in the cold storage that had my user account permissions only and not the permissions for the account which splunk was running under. I enabled inheritance on the storage drive and splunk was able to move the buckets afterwards.

[lycollicott]

Cool. We have had the exact same issue, so I created an alert for this :

index=_internal sourcetype=splunkd log_level=ERROR component=BucketMover "inflight-" "access is denied earliest=-1m latest=now"

We run it every minute instead of realtime and then we act on it.