wheater splunk forwarder can compress data brfore forward

[perlish]

Hi, I use splunk forwarder to forward data.
But the data was too much.
I want compress data before forward, wheather splunk support this?

Thank you

[DaveSavage]

Perlish - have you considered zip'ing the files and then using a heavy forwarder to send them to your back end indexer?
There are quite a few threads here which discuss it, not least being: http://splunk-base.splunk.com/answers/52976/indexing-log-files-which-are-in-zip-format
Good luck. I'm still struggling to get my head around the quantities you must be talking about!
Br
Dave

[Ayn]

Docs?

http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Configureforwarderswithoutputs.confd#Forwardi...

 compressed     false   global or target group stanza   Specifies whether the forwarder sends compressed data. 

[MuS]

hi perlish

if I understand your request correct, you want to filter out unneeded data on the forwarder, right?
if so, you can setup a so call heavy forwarder and setup filters and routes on it.

Filters and routes could also be setup on the indexer.

cheers,
MuS

[kristian_kolb]

Are you saying that you need sustained data transfer rates of 20 MB/s? That's just over 1,7 TB per day. Seriously, that is a LOT.

With those requirements (and that kind of license), I believe that Splunk Support will help you out. File a support case at splunk.com/support

[perlish]

Everyday,every second,i need forward data,the rate will be 20M/s, So if i limit rate, i can`t forward all data in oneday。

[MuS]

when you are saying you are loosing data, what kind of data is it and why does it get lost if it is not forwarded with in one day?

[perlish]

yes.
It is.

[MuS]

and you really need every single bit of the data you are forwarding?

[perlish]

Because if I dont limit, the rate will be 20M/s
If i limit, for example limit 10M/s.
I cant finished forward in everyday.

[MuS]

why should this happen?

[perlish]

But if i limit the rate, the data will loss.

[MuS]

you could limit the transfer rate on the forwarder likt this http://splunk-base.splunk.com/answers/52066/data-transfer-rate-between-forward-and-indexer
but I'm not aware of any compress feature.

[perlish]

Oh,sorry, it`s my fault

I want to compress data.

Sorry.