Deployment Architecture

Why can't I see indexes from the search head after enabling Index Clustering?

TerrySoucy
Explorer

Details:
- 3 x Indexers
- 1 x Cluster Master
- 1 x Search head
- 1 x Deployment server
- All running 6.2.0

I created the cluster master, then added each of my indexers as cluster peers. After copying my defaultIndexes app from the deployment server to the cluster master, I pushed the bundle out to the peers, then removed the defaultIndexes app from each indexer, as well as the deployment server (and reloaded). I then enabled index clustering on the search head (as a search head node) and disabled the indexers as search peers in the distributed search panel.

As soon as I disable the search peers, I am unable to search indexes in my cluster. I have additional search peers that are stand-alone (syslog servers) and I can search those indexes just fine, but I get 0 results when searching an index that is in the cluster. I found a similar topic on here that mentioned to add the indexes deployed in the config bundle as real indexes on the Cluster Master, but this gave the same result.

I'm not sure where else to look for this. Any ideas?

mahamed_splunk
Splunk Employee
Splunk Employee

I'm wondering why you disabled the indexers as search peers in the distributed search panel.

0 Karma

TerrySoucy
Explorer

The indexers were previously search peers on the search head. Changing the stand-alone search head to a cluster search head adds all clustered indexers as search peers automatically, which duplicated them. The original search peers needed to be disabled.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Does your cluster master report the cluster's search factor and replication factor as being met? Does it show the indexes with data in them, and buckets as having been replicated?

0 Karma

TerrySoucy
Explorer

Search factor and replication factor are met, and if I run the search query from the cluster master for content in the cluster, I get full results returned. From the cluster master dashboard, everything is green.

0 Karma

TerrySoucy
Explorer

Just thought of this. Is the master supposed to send the configuration bundle to the search head?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

No.. just the indexers.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You should be adding your search heads to the cluster master, and not individual search peers. I would remove them from the cluster, remove / disable any distributed search configuration, and then rejoin them to the cluster and see if that resolves the issue.

0 Karma

TerrySoucy
Explorer

That's been done. I added the existing search head to the cluster, and then disabled the duplicate search peers as described in the documentation.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Try removing all SH and distributed search configuration. Then rejoining the SH to the cluster. Sounds like you have a configuration that is still left in there somewhere.

0 Karma

TerrySoucy
Explorer

I get the same results. Even a search of "index=_internal" get's me 0 results. I think I'm just going to abandon this SH and provision a new one to add to the index cluster.

0 Karma