Indexers with large number of indexes becoming unresponsive because of acceleration

[yannK]

Known issue SPL-76956, http://docs.splunk.com/Documentation/Splunk/6.0/ReleaseNotes/KnownIssues#Data_model_and_Pivot_issues

Symptom : "SummaryDirector" searches running every 30 minutes on all instances with indexes defined (even empty).
and on the process explorer, and dispatch folder, a large number of "SummaryDirector" search jobs are visible.

On splunk 6.0 and 6.0.1 on the indexers/search-head/cluster-masters with a large number of indexes.
Because of the data model acceleration maintenance, a large number of SummaryDirector searches are triggered and the instances can become unresponsive eg. high CPU.

[yannK · Accepted solution]

Workarounds :

• the default for this SummaryDirector task is 30 minutes. Increase the interval for the maintenance_period (in seconds) in limits.conf on the Search Head(s) and Cluster Master


[auto_summarizer]
maintenance_period = 43200
# changed from 30 minutes to 12 hours


• disable the scheduler on the indexers (if they are not also search-heads with schedules jobs)

in $SPLUNK_HOME/etc/system/local/default-mode.conf

[pipeline:scheduler]
disabled_processors = LiveSplunks


[yannK]

1 - Yes, the workaround will make the acceleration feature less useful, because only older events will be accelerated, and for recent events, they will be based on on a regular search results.

1- This is an approximation, I would say that large is more than 100. Of course it depends also of the number of accelerated searches, volume of data, and server capacity...

[osunjio]

Hi,

Can you explain more on this statement :

"Yes, the workaround will make the acceleration feature less useful, because only older events will be accelerated, and for recent events, they will be based on regular search results."

Is it referring to DM acceleration? If yes, does it mean that the DM will not be up to date ?

[fwidmer]

Just 2 questions related to this workaround :
1. What are side-effects of the first workaround (increasing the maintenance_period interval) ? Does that mean that reports/dashboards based on Data Models won't include data of up to the last 12 hours (rather than 30min)?
2. To be sure whether our issue may be the same : what is a "large number of indexes" ?
Thanks for you contributions!