Symptom : "SummaryDirector" searches running every 30 minutes on all instances with indexes defined (even empty).
and on the process explorer, and dispatch folder, a large number of "SummaryDirector" search jobs are visible.
On splunk 6.0 and 6.0.1 on the indexers/search-head/cluster-masters with a large number of indexes.
Because of the data model acceleration maintenance, a large number of SummaryDirector searches are triggered and the instances can become unresponsive eg. high CPU.
maintenance_period(in seconds) in limits.conf on the Search Head(s) and Cluster Master
maintenance_period = 43200
# changed from 30 minutes to 12 hours
disabledprocessors = LiveSplunks
Just 2 questions related to this workaround :
1. What are side-effects of the first workaround (increasing the maintenance_period interval) ? Does that mean that reports/dashboards based on Data Models won't include data of up to the last 12 hours (rather than 30min)?
2. To be sure whether our issue may be the same : what is a "large number of indexes" ?
Thanks for you contributions!
1 - Yes, the workaround will make the acceleration feature less useful, because only older events will be accelerated, and for recent events, they will be based on on a regular search results.
1- This is an approximation, I would say that large is more than 100. Of course it depends also of the number of accelerated searches, volume of data, and server capacity...