Splunk Search
Highlighted

Mysterious realtime search generating 100s of jobs

Builder

We recently upgraded to Splunk 6 and on multiple occasions a real-time search seems to magically appear and causes all other searches/dashboards to halt because the limit for searches has been reached. When I check all running jobs I see 100s of real-time searches "|" (a single pipe) with no start-time or end-time by the user Admin. Initially, I thought this may be related to work on a real-time dashboard and post-process searches, but the dashboard is in a different app than where these mysterious searches are reportedly running.

To resolve the issue, I restart Splunk and delete the searches out of the dispatch directory. Simply trying to stop/delete them from the job management app will not work.

Is there any way to determine what is causing this? Could this be a bug in version 6?

Tags (3)
0 Karma
Highlighted

Re: Mysterious realtime search generating 100s of jobs

Path Finder

Just curious on the job management app - whats it?

0 Karma
Highlighted

Re: Mysterious realtime search generating 100s of jobs

Builder

I was referring to the job manager; not really a separate app.

0 Karma
Highlighted

Re: Mysterious realtime search generating 100s of jobs

Champion

I would suspect that this could be one of a couple of things (that I can think of).
How tight is your control of searches on the box? Is it possible that users have created lots of subsearches or real time searches and used the map command? I can't recall of the top of my head how this would appear in the job manager but it might fit the pattern.

Another option is that you have a couple of dashboards which have some oddly created searches that are impacting in v6 but weren't in v5? Can you do a search through your audit/internal logs to see where these searches are firing from.

0 Karma
Highlighted

Re: Mysterious realtime search generating 100s of jobs

Builder

Good questions and suggestions. I'll dig into those deeper to see if I can isolate the issue. I'll follow up.

0 Karma