We currently have the limits.conf maxmemusagemb parameter value set to 2000, which is 10x the default value (200). We have noticed incidences of splunk helper processes being killed due to OOM, and is suspecting that it might have something to do with maxmemusagemb value being too high:
01-27-2014 15:54:18.645 -0500 ERROR STMgr - dir='/opt/splunk/var/lib/splunk/internaldb/db/hotv143' out of memory failure rc=1 warmrc[-2,12] from sttxnstart
01-27-2014 15:54:18.645 -0500 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
01-27-2014 15:54:18.694 -0500 FATAL ProcessRunner - Unexpected EOF from process runner child!
This limit was set to a high value to accommodate the end user running extremely large ad hoc queries. This decision was taken since the system has an extremely large amount of memory (160 GB) and a large number of cores (32) available.
As a rule of thumb, what is the maximum range of values that can be assigned to maxmemusage_mb? Does it have a relation to the max available memory (160 GB in this case)?
The release 6.2.1 contains improvement on the way this parameter is applied. So you might consider an upgrade:
maxmemusage_mb = < non-negative integer >
*Provides a limitation to the amount of RAM a batch of events or results will use
in the memory of search processes.
*Operates on an estimation of memory use which is not exact.
*The limitation is applied in an unusual way; if the number of results or events
exceeds maxresults, AND the estimated memory exceeds this limit, the data is
spilled to disk.
*This means, as a general rule, lower limits will cause a search to use more disk
I/O and less RAM, and be somewhat slower, but should cause the same results to
typically come out of the search in the end.
*This limit is applied currently to a number, but not all search processors.
However, more will likely be added as it proves necessary.
*The number is thus effectively a ceiling on batch size for many components of
search for all searches run on this system.
**0 will specify the size to be unbounded. In this case searches may be allowed to
grow to arbitrary sizes.