topic Re: Indexing Throttling in Getting Data In

Indexing Throttling

Edub — Wed, 20 Jul 2011 20:32:42 GMT

A network socket process went bug-eyed today creating more than 7 million /var/log/messages events in 15min. The indexer threw an error stating "applying indexing throttle for..."

This is a decent feature considering we didn't need all those identical events in the index, but I could not find documentation anywhere that describes when and how this function operates.

What is limited and what is not being kept? When will it stop throttling? What parameters determine when it will throttle? How can I adjust them? etc ad-naseum

Anyone have some links to some details?

Thanks
E.

Re: Indexing Throttling

rgcurry — Thu, 09 Feb 2012 16:57:05 GMT

Edub, great question. I'd like to know about this too as it is a condition I have on one of my Splunk instances.

Looking thru the online doc for "index throttling condition" which is mentioned in two places in the doc concerning indexes (see below for related links), this condition is mentioned but I could find no other reference to it and what it entails. So I call out to those Splunk experts that know of this to describe what this means, what causes it and how does one resolve it?

The "index throttling condition" is mentioned but not explained in the following manuals/sections:

Admin Manual - indexes.conf > http://docs.splunk.com/Documentation/Splunk/4.3/Admin/Indexesconf
REST API Reference - Indexes > http://docs.splunk.com/Documentation/Splunk/4.3/RESTAPI/RESTindex

Re: Indexing Throttling

mark — Tue, 09 Oct 2012 06:11:03 GMT

I'm interested in throttling an index when events occur that cause excess and generally useless data.

Did you ever get to the bottom of how this feature works?

Re: Indexing Throttling

Drainy — Tue, 09 Oct 2012 08:33:14 GMT

When looking through limits.conf for this I came across this;

[thruput]

maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.

Now, I've used maxKBps for forwarder throttling in the past but I hadn't considered its use on an indexer, this seems to suggest it may be capable of performing the throttling behaviour you are looking for?