Splunk Search

Splunk Admin searches being queued, but not running.

djreschke
Explorer

Good afternoon everyone, 

I am the Splunk admin for our instance of Splunk, and yesterday later in the afternoon, I noticed that my searches were not running anymore, and that everything was being sent to the queue? In my troubleshooting efforts I signed into another searchhead, and ran a similar search with successful results. 

Today, I thought what could be causing the issue was, i have a lot of saved searches that run with my username as the owner, so reassigned those knowledge objects, which did not fix the issue. I can see other users running searches, with no issue, so it appears to be my account. 

Any advice of where to look next or has anyone experienced this issue before? 

I am running 7.2.8 for on my searchheads and using my account not the local admin account. 

 

Thank you.

Labels (1)
0 Karma

djreschke
Explorer

Thanks Everyone for your feedback on this topic, I did have a few jobs that need purged, and I did perform a full restart, but it was a combination of the above comments/suggestions that help and have yet to experience the issue again. 

0 Karma

djreschke
Explorer

Good morning everyone, 

Sorry for the late update, but a full restart did not work and I checked the running jobs and I don't have any running, just 5 in queue, which they have been stopped. Checking internal logs this morning. 

 

0 Karma

soutamo
SplunkTrust
SplunkTrust
And how many finalised/failed etc. jobs you have there waiting for purge?

bobd32
Engager

Full restart! 

soutamo
SplunkTrust
SplunkTrust
Check if your quotas has full. You could open jobs and remove some old finalized jobs and if your new jobs starts after this you know the reason. You could also check the situation from internal logs on node.