Splunk Search

Mysterious realtime search generating 100s of jobs

sc0tt
Builder

We recently upgraded to Splunk 6 and on multiple occasions a real-time search seems to magically appear and causes all other searches/dashboards to halt because the limit for searches has been reached. When I check all running jobs I see 100s of real-time searches "|" (a single pipe) with no start-time or end-time by the user Admin. Initially, I thought this may be related to work on a real-time dashboard and post-process searches, but the dashboard is in a different app than where these mysterious searches are reportedly running.

To resolve the issue, I restart Splunk and delete the searches out of the dispatch directory. Simply trying to stop/delete them from the job management app will not work.

Is there any way to determine what is causing this? Could this be a bug in version 6?

Tags (3)
0 Karma

Drainy
Champion

I would suspect that this could be one of a couple of things (that I can think of).
How tight is your control of searches on the box? Is it possible that users have created lots of subsearches or real time searches and used the map command? I can't recall of the top of my head how this would appear in the job manager but it might fit the pattern.

Another option is that you have a couple of dashboards which have some oddly created searches that are impacting in v6 but weren't in v5? Can you do a search through your audit/internal logs to see where these searches are firing from.

0 Karma

sc0tt
Builder

Good questions and suggestions. I'll dig into those deeper to see if I can isolate the issue. I'll follow up.

0 Karma

sc0tt
Builder

I was referring to the job manager; not really a separate app.

0 Karma

splunkears
Path Finder

Just curious on the job management app - whats it?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...