Why realtime dashboard searches continue to run in the background after browser is closed?

[Lucas_K]

I noticed that one particular power user was taking up almost all the realtime searches on 2 of our search heads. The twist is that this particular user didn't actually have ANY dashboards open. Yes they have dashboards with about 7 realtime searches on it but none of these had been openned in the last 10-12 hours at the time I approached them (first thing in the morning when they just turned their pc on).

Checking s.o.s again I could see that they had maxed out their roles quota of realtime searches (20 per search head for a power user).

It appeared that these searches were STILL running after the user has closed their browser. Checking their simple xml dashboard in question I found that they had a combination of saved searches (non-scheduled) and inline queries. Once again, they don't have ANY scheduled rt searches!

Aren't these sorts of searches supposed to be stopped after some amount of time after a user closes the browser? This users searches run until the search head is restarted. This is taking up valuable searches and is a waste of search head and indexing capacity.

So ... how can i tell which real time searches are actually orphans or
how can I get splunk to cull these searches that aren't going to a client?

Environment details: linux x64 splunk v6.0.3. Distributed search using search head pools & mounted bundles.

[Lucas_K · Accepted solution]

ok this is a bug and quite a serious one so not sure why no one else experienced it! It eventually causes your search heads & indexers to grind to a halt.

It is currently only fixed in 6.0.5. 6.1.x fix coming soon.
SPL-83708 - http://docs.splunk.com/Documentation/Splunk/6.0.5/ReleaseNotes/6.0.5

[Lucas_K]

Confirmed that this is fixed in 6.0.5.