Prevent over-index volume usage for free edition?

[ayachem]

Our normal index volume is less than 100 MB a day, putting us squarely in the free edition usage range. However, over the weekend, an SQL server hit a size limit and proceeded to dump 20 GB of logs (8 GB/day) to the windows event log. Yea, gigabytes. So subsequently my Splunk went over-limit for those 3 days and I can no longer search. Is there any option to set a limit from the Splunk data input side to prevent this sort of massive overage?

Any advice would be greatly appreciated.

Thank You,

~ Michelle

[seanwong]

You may want to consider using a Splunk Forwarder on your SQL server. On the splunk forwarder configure your inputs.conf as normal, but you'll need to modify your maxkbps throughput limit, to prevent over indexing.

limits.conf:
[thruput]
maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.

If you're reading from an open socket for your data instead of log files, you may also want to consider using persistentqueues to prevent dropping data if you cannot read/send fast enough.