Indexing Volume Limiting Strategies

[beaunewcomb]

For enterprise customers, what are your strategies for keeping a handle on the volume of data being logged to Splunk to avoid going over your license? Aside from finding the heavy hitters and seeing what changed, then filtering data or stopping splunkd on that forwarder, is there a way to automatically shut off an input after a certain amount of data has passed through it, or just drop events on the floor at the indexer level?

Seems the only way to manage this is on a reactive basis.. there has to be a more proactive approach.

[piebob · Accepted solution]

you can throttle the indexer such that it will only index at a given rate by setting the value of maxKbps in a copy of limits.conf. this is typically used to control CPU usage, but can also be employed to limit the amount of data processed in a given time. you will have to experiment with rates to find out what works for your environment.

maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.