Limit Data send from Forwarder to Indexer

[chrisitanmoleck]

Hello,

is it possible to limit the data which will be send to the forwarder, like 10 MB/day?

One of our application servers had a strange behaviour and create a logfile of 700MB.
We have only a 500MB licencse.

We want to avoid that.

Best Regards
Christian

[stmcmahon_splun]

Hello

I think your question/issue is not a matter of data rates, more the application and sourcetype providing more data than your license allows. You can limit the total volume of data as described here: http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Configureindexstoragesize

If you have more data than your license allows, you essentially have 2 choices:

1) get a bigger license
2) edit your retirement policy so that data rolls out more quickly http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Setaretirementandarchivingpolicy

I might be wrong in interpretation of your question, but you should evaluate where data is coming in from and how valuable it is for your business to use. If users are not searching against it, great, you can be more aggressive about moving to frozen.

[MuS]

Hi chrisitanmolecki,

well you can use the maxKBps option in limits.conf

[thruput]
maxKBps = <integer>
* If specified and not zero, this limits the speed through the thruput processor to the specified 
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer 
processes to the rate (in KBps) you specify. 

taking 24*60*60=86400 seconds per day and 10485760 bytes to be sent out during the day, do some math and you will get something like 0.118518519 KBps .

There are downsides to this, like events coming too late and other things. Also you should read this before doing any limit settings Use persistent queues.

hope this helps ...

cheers, MuS

[lguinn2]

The "Use persistent queues" link in the above answer is not good for current versions (6.x) of Splunk, although the content is still good. The page moved to

http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Usepersistentqueues

[Ayn · Accepted solution]

No, there's no way of setting an absolute data limit. What you can do is set a limit on how much bandwidth the forwarder uses for forwarding traffic. Universal Forwarders are capped at 256Kbps by default. See more here: http://docs.splunk.com/Documentation/Splunk/6.1/Admin/limitsconf (the config directive is "maxKbps" under the thruput stanza.

[sinash]

Just a correction: The default limit is 256KBps (Kilobytes per second) not 256Kbps (Kilobits per second).

[MuS]

HeHe, i was typing for too long 🙂

[chrisitanmoleck]

Thank your for your answer, but to limit the bandwith, doesn't help me.