Deployment Architecture

After upgrading to Splunk 6.6.0, why are 5 indexes marked as not searchable?

sylim_splunk
Splunk Employee
Splunk Employee

After an upgrade of our small cluster (two indexers, one search head, RF=SF=2) to 6.6.0, 5 indexes are marked as not searchable.
There are 0 "fix up" type tasks. I have tried gracefully restarting both indexers again (one at a time) and no luck there. Any help is appreciated. Thanks.alt text

no Fix-up tasks
alt text

0 Karma
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

OK the situation appears OK now. This may not be a commonly happening case. I ran the commands below to find the buckets status ;

i) REST call https://ClusterMaster:8089/services/cluster/master/buckets?filter=replication_count%3C2&filter=froze...
List of buckets with "frozen=false" & "replication_count < 2" (RF=SF=2 deployment)

ii) After I get the list of buckets from above, checked individual buckets using https://ClusterMaster:8089/services/cluster/master/buckets/{BID OF BUCKET IN ABOVE LIST}

Strangely there were some buckets without PEER information, which appears that the cluster master has some buckets information that its peers do not have.
Then ran remove_all which worked around the issue :

curl -k -u admin:password https://:8089/services/cluster/master/buckets//remove_all -X POST

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

OK the situation appears OK now. This may not be a commonly happening case. I ran the commands below to find the buckets status ;

i) REST call https://ClusterMaster:8089/services/cluster/master/buckets?filter=replication_count%3C2&filter=froze...
List of buckets with "frozen=false" & "replication_count < 2" (RF=SF=2 deployment)

ii) After I get the list of buckets from above, checked individual buckets using https://ClusterMaster:8089/services/cluster/master/buckets/{BID OF BUCKET IN ABOVE LIST}

Strangely there were some buckets without PEER information, which appears that the cluster master has some buckets information that its peers do not have.
Then ran remove_all which worked around the issue :

curl -k -u admin:password https://:8089/services/cluster/master/buckets//remove_all -X POST

Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...