We recently upgraded to Splunk 6 and on multiple occasions a real-time search seems to magically appear and causes all other searches/dashboards to halt because the limit for searches has been reached. When I check all running jobs I see 100s of real-time searches "
|" (a single pipe) with no start-time or end-time by the user Admin. Initially, I thought this may be related to work on a real-time dashboard and post-process searches, but the dashboard is in a different app than where these mysterious searches are reportedly running.
To resolve the issue, I restart Splunk and delete the searches out of the dispatch directory. Simply trying to stop/delete them from the job management app will not work.
Is there any way to determine what is causing this? Could this be a bug in version 6?
I would suspect that this could be one of a couple of things (that I can think of).
How tight is your control of searches on the box? Is it possible that users have created lots of subsearches or real time searches and used the map command? I can't recall of the top of my head how this would appear in the job manager but it might fit the pattern.
Another option is that you have a couple of dashboards which have some oddly created searches that are impacting in v6 but weren't in v5? Can you do a search through your audit/internal logs to see where these searches are firing from.