Splunk Search

How to troubleshoot why a Splunk search head is stuck on "Waiting for data" trying to run a search?

OMohi
Path Finder

Hi

I have an issue with a Splunk search head unable to return any search results. It is stuck on "waiting for data" page. I have deleted the dispatch folder and restarted Splunk, but to no relief. Also, l checked on the distributed search and this search head is able to establish connections with all its peers.

Please let me know any troubleshooting guidelines that'll assist me.

Thanks

Mohammed

0 Karma

Raghav2384
Motivator

Have you checked splunkd.log? Any errors or clues in there?

Can you search local _internal logs from that search head(Assuming you are the admin/have access to search for _* indexes)?

When you look under distributed search settings, do all the peers show up?
1. If index=_internal returns for the search head, i would focus on the 8089 communication between search head and indexers. Bounce the splunkd on indexers and see.

I have seen this problem before but i had something or the other warn me though....Example, process hung on xyz indexer. My Splunk did not detect it but our other monitoring tools caught it etc.

Hope this helps!

Thanks,
Raghav

0 Karma

OMohi
Path Finder

I am unable to query for any searches on the search head, l am getting error exiting code=255.

0 Karma

Raghav2384
Motivator

Can you list the Splunk version on Search head as well as Indexers/Search peers?

Please see this answer: https://answers.splunk.com/answers/170240/why-am-i-getting-error-search-process-did-not-exit.html

Thanks,
Raghav

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...