Determine Host of Bucket Involved in Cluster Fixup Task

[muebel]

If you go to /manager/system/clustering_bucket_details in the Web Interface of the Cluster Master you will get a nice view on all the fixup tasks involved in the Indexer Cluster. One thing that is missing from this view is the name of the indexer peer that is hosting the troubled bucket.

If you take the name of a bucket, and append it to the rest interface /services/cluster/master/buckets/, you do get the indexer peer involved with that bucket.

Is there any way to get this association for every fixup in progress? Essentially, I want to be able to generate a list of indexer peers that have active fixup tasks for some bucket they contain.

[muebel · Accepted solution]

Ended up figuring this out in a not-exactly-streamlined-but-works sort of way:

curl -s -k -u user:pass 'https://splunk-clusterMaster:8089/services/cluster/master/fixup?level=replication_factor&count=-1' | grep '<title>' | grep -v '<title>clustermasterfixup</title>' | sed 's/    <title>//g' | sed 's@</title>@@g' > fixup.list
for i in $(cat fixup.list); do curl -s -k -u user:pass "https://splunk-clusterMaster:8089/services/cluster/master/buckets/$i" | grep server_name >> fixup_peers.list; done
cat fixup_peers.list | sort -u

This might be easier using Splunk's 'rest' command as it seems easier to parse the results using SPL rather than the haphazard grep and sed here.

[jaime_ramirez]

Oustanding solution.

We ended up using it for deleting buckets that were causing issues in our cluster.

[dxu_splunk]

Just a note - a single bucket may live on multiple indexers. When the fixup jobs get scheduled by the Cluster Master, it may not involve the source peer at all - for example the Cluster Master may tell a Non-Source peer to replicate the bucket to another indexer.