Splunk search heads "waiting for data..."

[scottj1y]

I have a indexer cluster up and running receiving data from forwarders. I can confirm the indexer cluster is set up correctly both from the master dashboard (which shows all peer indexers as healthy and all indexes as searchable) and from running the various CLI commands to check cluster health.

I've integrated my search head cluster with the clustered indexers.  I can confirm this by checking the dashboard on the seach head captain and it shows the indexers as present. The command line also shows all the search heads are part of the search head cluster and the host intended to be the captain as the elected captain.

So the forwarders are working and sending data. The indexers are working receiving that data and are clustered correctly. And the search heads are clustered together, identify the correct search head captain, and the captain shows the 12 indexers.

However, every one of the search heads say they are "Awaiting on data..." from the search page.

I seem to have forgotten some step in the set up.  Anyone know what that might be?

[dustymehul]

were you able to resolve this issue.

[amitm05]

is this a working setup that has stopped for you ? OR you've just set it up and have not got it working yet ?
The answer can help to a better troubleshooting approach.

[dustymehul]

For me its a new setup.
i have a stand alone Search Head node running in parallel with SH-Cluster(new) over Indexer Cluster(old).

Stand Alone SH Node has proper configurations and roles-index mapping. I am not sure, if i am facing the issue because of incorrect role mapping.

Here are some observations -
"sourcetype="mySourceType" " does not returns any event from new SH-Clustered node but,
"index=* sourcetype="mySourceType" " returns correct events.

Now my idea is to push the apps and user configurations from StandAlone SH Node to SH-Cluster nodes via Deployer.

Although i am not sure if this will work

[Lucas_K]

All search heads need to be able to access the indexers not just the captain. Make sure they can all access the index cluster master also. This allows search heads to know where primary buckets are located.

The captain is responsible for distributing the search head cluster bundle but individual search heads talk to indexers.

Check the search head cluster status and that it is stable.

[scottj1y]

Each of the search heads report the correct Master IP and the correct number of peers, 12, from the Indexer Clustering menu item of the Settings menu.

Here's the text of that page (and it's identical across the Captain and the three cluster peers):

Cluster Master Searchable Search Factor Replication Factor Status

https://10.x.x.219:5500 All Data is Searchable Met Met Up

Generation id
154
Peers
12