Deployment Architecture

Stop concurrent scheduled summary update in clustered env

proylea
Contributor

I have a scheduled search that updates a summary index.
On our non clustered splunk search head it would execute every minute from cron and the next job would wait and not start until the last one had finished.

On our clustered environment the job seems to run concurrently on different search heads when the first job runs longer than a minute.
For reasons I won't go into here I don't want the jobs to run concurrently.

I have looked at the setting in savedsearches.conf called "max_concurrency"
I don't have access to play with config files in this clustered environment (and I don't have a test cluster)

Can someone tell me if I can use this setting or others to stop job concurrency
maybe:
max_concurrency=0

logloganathan
Motivator

sorry to say..

look like only admin can do this task.

we have only this document to refer

http://docs.splunk.com/Documentation/Splunk/6.1/Admin/Savedsearchesconf

Please honor me if you like this answer

0 Karma

damiensurat
Contributor

Hi there proylea, You may also have success by setting the quota via the authorize.conf file. I can't say that I'm an expert on the subject, but figured I'd provide any insight seeing as how you haven't had many responses. Hopefully this helps you get onto the path of succeeding in limiting concurrent searches:

http://docs.splunk.com/Documentation/Splunk/6.0.6/Admin/authorizeconf

From what I've read on the topic, you should be able to limit search concurrency to a role rather than across applying a limit to all searches across the cluster:

cumulativeRTSrchJobsQuota =
* Maximum number of concurrently running real-time searches in total
across all members of this role
* Requires enable_cumulative_quota = true in limits.conf to take effect.
* If a user belongs to multiple roles, the user's searches count against the role with
the largest cumulative search quota. Once the quota for that role is consumed, the
user's searches count against the role with the next largest quota, and so on.
* In search head clustering environments, this setting takes effect on a per-member basis.
There is no cluster-wide accounting.

cumulativeSrchJobsQuota =
* Maximum number of concurrently running historical searches in total
across all members of this role
* Requires enable_cumulative_quota = true in limits.conf to take effect.
* If a user belongs to multiple roles, the user's searches count against the role with
the largest cumulative search quota. Once the quota for that role is consumed, the
user's searches count against the role with the next largest quota, and so on.
* In search head clustering environments, this setting takes effect on a per-member basis.
There is no cluster-wide accounting.

My idea is that you can create a user and add the user to only this role to limit the amount of concurrent searches the user can perform. Then run the search as that user.

somesoni2
Revered Legend

Try settingrealtime_schedule = 0 for your summary index search.

realtime_schedule = [0|1]
* Controls the way the scheduler computes the next execution time of a
  scheduled search.
* If this value is set to 0, the scheduler bases its determination of the next
  scheduled search on the last search execution time. This is called continuous
  scheduling.
  * If set to 0, the scheduler never skips scheduled execution periods.
  * However, the execution
    of the saved search might fall behind depending on the scheduler's load.
    Use continuous scheduling whenever you enable the summary index option.
* The scheduler tries to execute searches that have realtime_schedule set to 1
  before it executes searches that have continuous scheduling
  (realtime_schedule = 0).
* Defaults to 1

proylea
Contributor

Thank you sir 🙂
I'll have a go, wasn't sure about that one, didn't really understand the doco on it.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!