topic Re: wheater splunk forwarder can compress data brfore forward in Getting Data In

wheater splunk forwarder can compress data brfore forward

perlish — Tue, 16 Oct 2012 07:17:18 GMT

Hi, I use splunk forwarder to forward data.
But the data was too much.
I want compress data before forward, wheather splunk support this?

Thank you

Re: wheater splunk forwarder can compress data brfore forward

MuS — Tue, 16 Oct 2012 07:29:48 GMT

hi perlish

if I understand your request correct, you want to filter out unneeded data on the forwarder, right?
if so, you can setup a so call heavy forwarder and setup filters and routes on it.

Filters and routes could also be setup on the indexer.

cheers,
MuS

Re: wheater splunk forwarder can compress data brfore forward

perlish — Tue, 16 Oct 2012 07:48:16 GMT

Oh,sorry, it`s my fault

I want to compress data.

Sorry.

Re: wheater splunk forwarder can compress data brfore forward

MuS — Tue, 16 Oct 2012 07:56:21 GMT

you could limit the transfer rate on the forwarder likt this http://splunk-base.splunk.com/answers/52066/data-transfer-rate-between-forward-and-indexer
but I'm not aware of any compress feature.

Re: wheater splunk forwarder can compress data brfore forward

perlish — Tue, 16 Oct 2012 07:58:29 GMT

But if i limit the rate, the data will loss.

Re: wheater splunk forwarder can compress data brfore forward

MuS — Tue, 16 Oct 2012 08:02:06 GMT

why should this happen?

Re: wheater splunk forwarder can compress data brfore forward

perlish — Tue, 16 Oct 2012 08:20:23 GMT

Because if I dont limit, the rate will be 20M/s If i limit, for example limit 10M/s. I cant finished forward in everyday.

Re: wheater splunk forwarder can compress data brfore forward

MuS — Tue, 16 Oct 2012 08:22:59 GMT

and you really need every single bit of the data you are forwarding?

Re: wheater splunk forwarder can compress data brfore forward

perlish — Tue, 16 Oct 2012 08:23:59 GMT

yes.
It is.

Re: wheater splunk forwarder can compress data brfore forward

MuS — Tue, 16 Oct 2012 08:55:16 GMT

when you are saying you are loosing data, what kind of data is it and why does it get lost if it is not forwarded with in one day?

Re: wheater splunk forwarder can compress data brfore forward

Ayn — Tue, 16 Oct 2012 09:23:24 GMT

Docs?

http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Configureforwarderswithoutputs.confd#Forwarding_attributes

 compressed     false   global or target group stanza   Specifies whether the forwarder sends compressed data.

Re: wheater splunk forwarder can compress data brfore forward

perlish — Tue, 16 Oct 2012 10:07:40 GMT

Everyday,every second,i need forward data,the rate will be 20M/s, So if i limit rate, i can`t forward all data in oneday。

Re: wheater splunk forwarder can compress data brfore forward

DaveSavage — Tue, 16 Oct 2012 11:21:29 GMT

Perlish - have you considered zip'ing the files and then using a heavy forwarder to send them to your back end indexer?
There are quite a few threads here which discuss it, not least being: http://splunk-base.splunk.com/answers/52976/indexing-log-files-which-are-in-zip-format
Good luck. I'm still struggling to get my head around the quantities you must be talking about!
Br
Dave

Re: wheater splunk forwarder can compress data brfore forward

kristian_kolb — Thu, 18 Oct 2012 15:28:21 GMT

Are you saying that you need sustained data transfer rates of 20 MB/s? That's just over 1,7 TB per day. Seriously, that is a LOT.

With those requirements (and that kind of license), I believe that Splunk Support will help you out. File a support case at splunk.com/support