How to troubleshoot why a Splunk search head is stuck on "Waiting for data" trying to run a search?

[OMohi]

Hi

I have an issue with a Splunk search head unable to return any search results. It is stuck on "waiting for data" page. I have deleted the dispatch folder and restarted Splunk, but to no relief. Also, l checked on the distributed search and this search head is able to establish connections with all its peers.

Please let me know any troubleshooting guidelines that'll assist me.

Thanks

Mohammed

[Raghav2384]

Have you checked splunkd.log? Any errors or clues in there?

Can you search local _internal logs from that search head(Assuming you are the admin/have access to search for _* indexes)?

When you look under distributed search settings, do all the peers show up?
1. If index=_internal returns for the search head, i would focus on the 8089 communication between search head and indexers. Bounce the splunkd on indexers and see.

I have seen this problem before but i had something or the other warn me though....Example, process hung on xyz indexer. My Splunk did not detect it but our other monitoring tools caught it etc.

Hope this helps!

Thanks,
Raghav

[OMohi]

I am unable to query for any searches on the search head, l am getting error exiting code=255.

[Raghav2384]

Can you list the Splunk version on Search head as well as Indexers/Search peers?

Please see this answer: https://answers.splunk.com/answers/170240/why-am-i-getting-error-search-process-did-not-exit.html

Thanks,
Raghav