Why can't I see indexes from the search head after enabling Index Clustering?

[TerrySoucy]

Details:
- 3 x Indexers
- 1 x Cluster Master
- 1 x Search head
- 1 x Deployment server
- All running 6.2.0

I created the cluster master, then added each of my indexers as cluster peers. After copying my defaultIndexes app from the deployment server to the cluster master, I pushed the bundle out to the peers, then removed the defaultIndexes app from each indexer, as well as the deployment server (and reloaded). I then enabled index clustering on the search head (as a search head node) and disabled the indexers as search peers in the distributed search panel.

As soon as I disable the search peers, I am unable to search indexes in my cluster. I have additional search peers that are stand-alone (syslog servers) and I can search those indexes just fine, but I get 0 results when searching an index that is in the cluster. I found a similar topic on here that mentioned to add the indexes deployed in the config bundle as real indexes on the Cluster Master, but this gave the same result.

I'm not sure where else to look for this. Any ideas?

[mahamed_splunk]

I'm wondering why you disabled the indexers as search peers in the distributed search panel.

[TerrySoucy]

The indexers were previously search peers on the search head. Changing the stand-alone search head to a cluster search head adds all clustered indexers as search peers automatically, which duplicated them. The original search peers needed to be disabled.

[esix_splunk]

Does your cluster master report the cluster's search factor and replication factor as being met? Does it show the indexes with data in them, and buckets as having been replicated?

[TerrySoucy]

Search factor and replication factor are met, and if I run the search query from the cluster master for content in the cluster, I get full results returned. From the cluster master dashboard, everything is green.

[TerrySoucy]

Just thought of this. Is the master supposed to send the configuration bundle to the search head?

[esix_splunk]

No.. just the indexers.

[esix_splunk]

You should be adding your search heads to the cluster master, and not individual search peers. I would remove them from the cluster, remove / disable any distributed search configuration, and then rejoin them to the cluster and see if that resolves the issue.

[TerrySoucy]

That's been done. I added the existing search head to the cluster, and then disabled the duplicate search peers as described in the documentation.

[esix_splunk]

Try removing all SH and distributed search configuration. Then rejoining the SH to the cluster. Sounds like you have a configuration that is still left in there somewhere.

[TerrySoucy]

I get the same results. Even a search of "index=_internal" get's me 0 results. I think I'm just going to abandon this SH and provision a new one to add to the index cluster.