topic How do i get a list of all these concurrent searches? in Splunk Search

How do i get a list of all these concurrent searches?

HattrickNZ — Mon, 03 Oct 2016 21:28:52 GMT

I am trying to setup a summary and schedule it to run daily at 03.05a.m. as a cron job. But I get this error Your maximum number of concurrent searches has been reached.

How do i get a list of all these searches? Do i have to go into each search I have and see which ones of them are scheduled? And what does it mean by concurrent searches, can I have so many running within a certain time frame?

Schedule and alert
Schedule this search

Schedule type *

Cron schedule

05 3 * * *
Enter a cron-style schedule.

...

Summary indexing
Enable
Enabling summary indexing will set the alert condition to 'always'.
Select the summary index
Failed to fetch data: Search not executed: Your maximum number of concurrent searches has been reached. usage=4 quota=4 user=user.name. SearchId=1475527164.1062544

Re: How do i get a list of all these concurrent searches?

jhupka — Mon, 03 Oct 2016 21:54:40 GMT

There's several ways to see information about these scheduled searches. In addition to looking through your scheduled search configurations like you mention (and they're also in all of the savedsearches.conf files under the hood), you can manually find these by clicking on the Activity->Jobs menu in the upper right of the Splunk UI. From there you can use the various drop-downs to show All searches, then limit to just the ones that are Running.

It is also possible to search for this information. Here is a simple search that will find any information about Splunk queuing your searches due to hitting these limits. Run this over the timeframe you're concerned about (+/- 15 minutes, for example) to see what could be going on (you'll typically need to be an admin to search the _internal index):

index=_internal sourcetype=splunkd component=DispatchManager queued

If you want to search for them, the following search will show currently running searches:

| rest /services/search/jobs | search dispatchState=RUNNING

BUT, you can end up in a Catch-22 if the Search Head is queuing your search and you're trying to run a search to see what is/was running.

Re: How do i get a list of all these concurrent searches?

sloshburch — Wed, 05 Oct 2016 14:41:34 GMT

@hattrickNZ - did this answer your question? If so, you can "accept" this answer or post clarifying comments.