Getting Data In
Highlighted

Indexers with large number of indexes becoming unresponsive because of acceleration

Splunk Employee
Splunk Employee

Known issue SPL-76956, http://docs.splunk.com/Documentation/Splunk/6.0/ReleaseNotes/KnownIssues#Data_model_and_Pivot_issues

Symptom : "SummaryDirector" searches running every 30 minutes on all instances with indexes defined (even empty).
and on the process explorer, and dispatch folder, a large number of "SummaryDirector" search jobs are visible.

On splunk 6.0 and 6.0.1 on the indexers/search-head/cluster-masters with a large number of indexes.
Because of the data model acceleration maintenance, a large number of SummaryDirector searches are triggered and the instances can become unresponsive eg. high CPU.

Highlighted

Re: Indexers with large number of indexes becoming unresponsive because of acceleration

Splunk Employee
Splunk Employee

Workarounds :

  • the default for this SummaryDirector task is 30 minutes. Increase the interval for the maintenance_period (in seconds) in limits.conf on the Search Head(s) and Cluster Master


[auto_summarizer]
maintenance_period = 43200
# changed from 30 minutes to 12 hours

  • disable the scheduler on the indexers (if they are not also search-heads with schedules jobs)

in $SPLUNKHOME/etc/system/local/default-mode.conf
`
[pipeline:scheduler]
disabled
processors = LiveSplunks
`

View solution in original post

Highlighted

Re: Indexers with large number of indexes becoming unresponsive because of acceleration

Engager

Just 2 questions related to this workaround :
1. What are side-effects of the first workaround (increasing the maintenance_period interval) ? Does that mean that reports/dashboards based on Data Models won't include data of up to the last 12 hours (rather than 30min)?
2. To be sure whether our issue may be the same : what is a "large number of indexes" ?
Thanks for you contributions!

0 Karma
Highlighted

Re: Indexers with large number of indexes becoming unresponsive because of acceleration

Splunk Employee
Splunk Employee

1 - Yes, the workaround will make the acceleration feature less useful, because only older events will be accelerated, and for recent events, they will be based on on a regular search results.

1- This is an approximation, I would say that large is more than 100. Of course it depends also of the number of accelerated searches, volume of data, and server capacity...