topic Re: Is it possible to kill or disable long running searches?? in Splunk Search

Is it possible to kill or disable long running searches??

dbashyam — Sat, 02 Feb 2019 08:00:12 GMT

Hi, is it possible to kill or disable long running searches automatically. For example whenever we hit performance issues we observe that people are running 30 day searches or 10 day searches like that, so we did educate the users not to run long running searches but few listen few don’t so wanted to know whether to it’s possible to kill or warn or disable long running searches?

Re: Is it possible to kill or disable long running searches??

chrisyounger — Sat, 02 Feb 2019 09:19:48 GMT

Hi @dbashyam

Yes it is possible to kill running searches. Click Activity (top right of screen) > Jobs and you can see the running jobs here. Click the square to stop any that you want to kill.

Hope this solves your problem.

Re: Is it possible to kill or disable long running searches??

dbashyam — Sat, 02 Feb 2019 12:05:05 GMT

Thanks for your reply. But I want to automate it. I don’t want to login and check who is running Long running searches and the disable, I don’t want to do this. I want to write a script or a search which should tell long running searches and if it running for a long time as per my condition just disable or delete that search. Is this possible?

Re: Is it possible to kill or disable long running searches??

chrisyounger — Sat, 02 Feb 2019 12:16:00 GMT

The best thing to do would be to enforce limits to stop users doing silly thing. You can set this setting in authorize.conf :

srchMaxTime = <number><unit>
* Maximum amount of time that searches of users from this role will be
  allowed to run.
* Once the search has been ran for this amount of time it will be auto
  finalized, If the role
* Inherits from other roles, the maximum srchMaxTime value specified in the
  included roles.
* This maximum does not apply to real-time searches.
* Examples: 1h, 10m, 2hours, 2h, 2hrs, 100s
* Defaults to 100days

If this isn't suitable then yes it is possible to do what you want with searches or a script that calls the API. It would be a lot of work though.

Re: Is it possible to kill or disable long running searches??

woodcock — Sat, 02 Feb 2019 18:55:46 GMT

Download this app and search through the alerts. There is nothing in Splunk that allows you to automate killing searches so you will have to build that part. With this app, though, you will have all the searches that you need to find the searches and maybe you can just email people (which splunk can do) and say STOP DOING THIS!

Alerts for Splunk Admins: https://splunkbase.splunk.com/app/3796/

Re: Is it possible to kill or disable long running searches??

gjanders — Sun, 03 Feb 2019 00:45:02 GMT

As per woodcock's post the app I have created has ways to detect them, but you won't be able to auto-stop jobs from within Splunk from what I know, you could build an external script.

However you might be better off using srchMaxTime as per chrisyoungerjds, just keep in mind that it's a subtle message when the search is auto-finalized and it appears in the dispatch/info CSV (and as an info-level message on screen that you have to look for, not a warning).
I have a search for this called SearchHeadLevel - Users with auto-finalized searches which is here in github or Alerts for SplunkAdmins on SplunkBase which requires some extra internal logs files to be indexed to find auto-finalization of jobs...

Re: Is it possible to kill or disable long running searches??

MuS — Mon, 04 Feb 2019 02:36:05 GMT

hi dbashyam,

It would indeed be possible to automate this using a custom alert action, see the docs for more https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModAlertsIntro

First your search would query the REST endpoint /services/search/jobs/ to get what ever you define as a kill reason. Than you need to make sure the sid of the search is available as the result of this search. The reason for this is the alert action can then use $job.resultCount$ and do a post against the REST API using curl like this:

curl -k -u admin:pass https://localhost:8089/services/search/jobs/<sid>/control -d action=cancel

More details about this can be found in the docs https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTsearch#search.2Fjobs.2F.7Bsearch_id.7D.2Fcontrol

You it is possible, but it takes some work to get there.

Hope this helps ...

cheers, MuS