We recently had the situation, where the web interface was responding slowly and we had a high load on the system caused by splunk processes.
The following message was in splunkd.log:
WARN DispatchCommand - The system is approaching the maximum number of historical searches that can be run concurrently. current=55 maximum=68
We think this could be because the searches running concurrently cause the high load.
Is there a way to use the CLI to display how resource intensive jobs/searches are that are currently running and is there a way to kill/cancel/delete resource intensive jobs?