Deployment Architecture
Highlighted

Why can't I see indexes from the search head after enabling Index Clustering?

Explorer

Details:
- 3 x Indexers
- 1 x Cluster Master
- 1 x Search head
- 1 x Deployment server
- All running 6.2.0

I created the cluster master, then added each of my indexers as cluster peers. After copying my defaultIndexes app from the deployment server to the cluster master, I pushed the bundle out to the peers, then removed the defaultIndexes app from each indexer, as well as the deployment server (and reloaded). I then enabled index clustering on the search head (as a search head node) and disabled the indexers as search peers in the distributed search panel.

As soon as I disable the search peers, I am unable to search indexes in my cluster. I have additional search peers that are stand-alone (syslog servers) and I can search those indexes just fine, but I get 0 results when searching an index that is in the cluster. I found a similar topic on here that mentioned to add the indexes deployed in the config bundle as real indexes on the Cluster Master, but this gave the same result.

I'm not sure where else to look for this. Any ideas?

Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Splunk Employee
Splunk Employee

Does your cluster master report the cluster's search factor and replication factor as being met? Does it show the indexes with data in them, and buckets as having been replicated?

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Explorer

Search factor and replication factor are met, and if I run the search query from the cluster master for content in the cluster, I get full results returned. From the cluster master dashboard, everything is green.

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Splunk Employee
Splunk Employee

You should be adding your search heads to the cluster master, and not individual search peers. I would remove them from the cluster, remove / disable any distributed search configuration, and then rejoin them to the cluster and see if that resolves the issue.

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Explorer

That's been done. I added the existing search head to the cluster, and then disabled the duplicate search peers as described in the documentation.

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Splunk Employee
Splunk Employee

Try removing all SH and distributed search configuration. Then rejoining the SH to the cluster. Sounds like you have a configuration that is still left in there somewhere.

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Explorer

I get the same results. Even a search of "index=_internal" get's me 0 results. I think I'm just going to abandon this SH and provision a new one to add to the index cluster.

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Explorer

Just thought of this. Is the master supposed to send the configuration bundle to the search head?

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Splunk Employee
Splunk Employee

No.. just the indexers.

0 Karma
Highlighted

Re: Why can't I see indexes from the search head after enabling Index Clustering?

Splunk Employee
Splunk Employee

I'm wondering why you disabled the indexers as search peers in the distributed search panel.

0 Karma