Saved search running from very long time and not finishing?


Hi Folks,
From last couple of  weeks we have observed an issue in our newly developed Splunk app(Radware Bot Risk Scanner ). our app schedules a saved search which runs every hour and extract some data from indices and forwards to custom search command which we developed and saves the result in result indices.

Flow:  Splunk Search -> Custom Search Command (which preforms REST API call for each record) -> save result to new indice.

Saved Searches got stuck in Running state. when I try to stop it manually, its going to Finalizing state not done state. Ideal time for this saved search to finish is ~2mins including all Rest API calls, yet you can see often its running from a very long time. please refer attached screenshot for the same
Wanted to attach search log as well but can't due to message restriction

Any help or idea over here is very much appreciated, thanks in advance 😊.

P.S: Very important thing to notice is if I run any job for any hour manually, I wasn't facing any issues at all 😁.

Labels (2)
0 Karma


Have a look at the job inspector for the running jobs. If that search scheduled to run hourly and has a 2 hour expiry, it looks like there are skipped searches.

Check if you do - also you have a yellow warning triangle - does that tell you anything about searches?

Do you have any logging from your custom search command that shows whether the REST commands are working or not?

0 Karma


Hi @bowesmana , thanks for responding back.

I have debugged search log for few of pending jobs, I didn't get any clue from it.

Yes, yellow warning triangle indicating about skipped searches as current running jobs were reached the threshold and new ones being skipped automatically.

I was actually debugging from my API side only as my custom search command uses multithreading to speedup the of REST API calls. I was checking for scenarios like Unbounded Queue/ Unhandled exceptions that might cause this.

Will add more info once I 'm done with debugging.

0 Karma
Get Updates on the Splunk Community!

Devesh Logendran, Splunk, and the Singapore Cyber Conquest

At this year’s Splunk University, I had the privilege of chatting with Devesh Logendran, one of the winners in ...

There's No Place Like Chrome and the Splunk Platform

WATCH NOW!Malware. Risky Extensions. Data Exfiltration. End-users are increasingly reliant on browsers to ...

Customer Experience | Join the Customer Advisory Board!

Are you ready to take your Splunk journey to the next level? 🚀 We invite you to join our elite squad ...