Splunk Admin searches being queued, but not running.

[djreschke]

Good afternoon everyone, 

I am the Splunk admin for our instance of Splunk, and yesterday later in the afternoon, I noticed that my searches were not running anymore, and that everything was being sent to the queue? In my troubleshooting efforts I signed into another searchhead, and ran a similar search with successful results. 

Today, I thought what could be causing the issue was, i have a lot of saved searches that run with my username as the owner, so reassigned those knowledge objects, which did not fix the issue. I can see other users running searches, with no issue, so it appears to be my account. 

Any advice of where to look next or has anyone experienced this issue before? 

I am running 7.2.8 for on my searchheads and using my account not the local admin account. 

Thank you.

[djreschke]

Thanks Everyone for your feedback on this topic, I did have a few jobs that need purged, and I did perform a full restart, but it was a combination of the above comments/suggestions that help and have yet to experience the issue again. 

[djreschke]

Good morning everyone, 

Sorry for the late update, but a full restart did not work and I checked the running jobs and I don't have any running, just 5 in queue, which they have been stopped. Checking internal logs this morning. 

[isoutamo]

And how many finalised/failed etc. jobs you have there waiting for purge?

[bobd32]

Full restart! 

[isoutamo]

Check if your quotas has full. You could open jobs and remove some old finalized jobs and if your new jobs starts after this you know the reason. You could also check the situation from internal logs on node.