topic Removed index from indexes.conf but still getting errors about index in Splunk Search

Removed index from indexes.conf but still getting errors about index

wsanderstii — Wed, 17 Jan 2018 17:58:01 GMT

I tried removing an index from /opt/splunk/etc/master-apps/_cluster/local/indexes.conf as per https://answers.splunk.com/answers/471105/deleting-an-index-in-a-distributed-splunk-deployme.html and https://answers.splunk.com/answers/387161/official-way-to-clean-indexed-data-from-index-clus.html, restarted the cluster, but the index is still present in the cluster. There are errors in the web GUI that read "cannot replicate as bucket is not serviceable", and splunkd is continuously scrolling:

01-17-2018 09:54:33.389 -0800 INFO  CMReplicationRegistry - Finished replication: bid=akamailogs~220~A3DE1411-41D8-44A2-811A-B30A7284FAB2 src=A12B1B95-3FA8-459A-BD3A-357F88B6B4EC target=A3DE1411-41D8-44A2-811A-B30A7284FAB2
01-17-2018 09:54:33.389 -0800 INFO  CMMaster - event=handleReplicationError bid=akamailogs~220~A3DE1411-41D8-44A2-811A-B30A7284FAB2 tgt=A3DE1411-41D8-44A2-811A-B30A7284FAB2 peer_name=splunk03.s2prod msg='target doesn't have bucket now. ignoring'

Any ideas how to get splunk to completey forget about this index?

Re: Removed index from indexes.conf but still getting errors about index

nickhills — Wed, 17 Jan 2018 18:04:09 GMT

was your cluster consistent before you removed the index?

Re: Removed index from indexes.conf but still getting errors about index

nickhills — Wed, 17 Jan 2018 18:34:30 GMT

Pre-empting your reply - this should only be likely to occur if your cluster was inconsistent when you removed the index, if your cluster was full sf/rf then you may have other underlying issues.

If this is a production cluster, you may wish to consult with Splunk support - but if your cluster is expendable (and with all the normal caveats about taking advice from some random guy on the internet)

Run this query to get a list of buckets which have failed to replicate properly:

index=_internal event=handleReplicationError bid=akamilogs*|dedup bid|table bid

Export that as a csv (confirming all the buckets are ones you want to remove)

Depending on the total number of buckets you can either remove them one by one using:

curl -k -u admin:changeme -X POST "https://clusterMAaterAddress:8089/services/cluster/master/buckets/akamilogs~xxxx~xxxxxxxxx-xxxx-xxxxx-xxxxx-xxxxxxxxx/remove_all"

Or this script will delete the lot (run the script in the same folder as the csv)

#!/bin/bash

cat bad_buckets.csv | while read bucket
do
    curl -k -u admin:changeme -X POST "https://clusterMAaterAddress:8089/services/cluster/master/buckets/$bucket/remove_all"
done

You would be wise to test this by removing one or two buckets, and confirming that the errors stop for those bids. Also, I have typed this on a train with no access to splunk, so it has not been tested, so please take every precaution - peer review / backups / mother on speed dial...