Deployment Architecture

Determine Host of Bucket Involved in Cluster Fixup Task

muebel
SplunkTrust
SplunkTrust

If you go to /manager/system/clustering_bucket_details in the Web Interface of the Cluster Master you will get a nice view on all the fixup tasks involved in the Indexer Cluster. One thing that is missing from this view is the name of the indexer peer that is hosting the troubled bucket.

If you take the name of a bucket, and append it to the rest interface /services/cluster/master/buckets/, you do get the indexer peer involved with that bucket.

Is there any way to get this association for every fixup in progress? Essentially, I want to be able to generate a list of indexer peers that have active fixup tasks for some bucket they contain.

1 Solution

muebel
SplunkTrust
SplunkTrust

Ended up figuring this out in a not-exactly-streamlined-but-works sort of way:

curl -s -k -u user:pass 'https://splunk-clusterMaster:8089/services/cluster/master/fixup?level=replication_factor&count=-1' | grep '<title>' | grep -v '<title>clustermasterfixup</title>' | sed 's/    <title>//g' | sed 's@</title>@@g' > fixup.list
for i in $(cat fixup.list); do curl -s -k -u user:pass "https://splunk-clusterMaster:8089/services/cluster/master/buckets/$i" | grep server_name >> fixup_peers.list; done
cat fixup_peers.list | sort -u

This might be easier using Splunk's 'rest' command as it seems easier to parse the results using SPL rather than the haphazard grep and sed here.

View solution in original post

muebel
SplunkTrust
SplunkTrust

Ended up figuring this out in a not-exactly-streamlined-but-works sort of way:

curl -s -k -u user:pass 'https://splunk-clusterMaster:8089/services/cluster/master/fixup?level=replication_factor&count=-1' | grep '<title>' | grep -v '<title>clustermasterfixup</title>' | sed 's/    <title>//g' | sed 's@</title>@@g' > fixup.list
for i in $(cat fixup.list); do curl -s -k -u user:pass "https://splunk-clusterMaster:8089/services/cluster/master/buckets/$i" | grep server_name >> fixup_peers.list; done
cat fixup_peers.list | sort -u

This might be easier using Splunk's 'rest' command as it seems easier to parse the results using SPL rather than the haphazard grep and sed here.

View solution in original post

jaime_ramirez
Communicator

Oustanding solution.

We ended up using it for deleting buckets that were causing issues in our cluster.

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

Just a note - a single bucket may live on multiple indexers. When the fixup jobs get scheduled by the Cluster Master, it may not involve the source peer at all - for example the Cluster Master may tell a Non-Source peer to replicate the bucket to another indexer.

.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!