topic Re: Configure heavy forwarder to send data periodically in Getting Data In

Configure heavy forwarder to send data periodically

moohkhol — Mon, 28 Sep 2020 15:37:47 GMT

Hi Guys,

I set-up Heavy forwarder at Machine-1 and wants to send data on Machine-2, since network in involve in between so i want to minimize the network traffic to prevent certain type of events and send the imp event immediately but less imp event per day or per hours.
Many Places i read that using universal forwarder we cann't do filter as well as interval setting, that's why i moved to Heavy forwarder, I can able to filter out events in heavy forwarders but unable to set periodicity.

It would be great help if some one can tell me stanza for setting interval of polling,
lets say, I want to monitor, "testlog" all time and "testlog1" after every 30 mins and "testlog2" after every 1 hours.

Right now my inputs.conf is below,

[monitor:///usr/local/preview/splunk/testlog/*]
index = main
sourcetype = testlog

[splunktcp://9997]
connection_host = ip
_TCP_ROUTING = splunkindexer_9997

Re: Configure heavy forwarder to send data periodically

Ayn — Fri, 10 Jan 2014 10:43:42 GMT

This is not unique to Universal Forwarders - no Splunk instance has the option to send logs just periodically. So, moving to a heavy forwarder does not help.

Re: Configure heavy forwarder to send data periodically

moohkhol — Fri, 10 Jan 2014 10:49:21 GMT

What's the best practice for splunk to achieve this, lets say i wants to use log for data analytics perspective, i don't need real time information, even if it's hours old or day old it would me fine for me, it would be very general requirement.
Any suggestion for that would help me a lots.

Re: Configure heavy forwarder to send data periodically

moohkhol — Fri, 10 Jan 2014 10:50:03 GMT

Re: Configure heavy forwarder to send data periodically

Ayn — Fri, 10 Jan 2014 10:52:59 GMT

Commonly this is not a problem as most environments don't have so little bandwidth. If it's bandwidth you're concerned about, you can limit at which rate forwarders send their data in limits.conf. Default for a Universal Forwarder is 256kB/s.

Re: Configure heavy forwarder to send data periodically

gfuente — Fri, 10 Jan 2014 12:36:16 GMT

Maybe, you could configure a regular monitor. Then use cron or any other scheduler tool to start and stop the UF as needed

You could install several instances of uf in the same host, if needed, one for real time monitoring and the other one for batch (with the start and stop trick...)

Regards

Re: Configure heavy forwarder to send data periodically

jbrodsky_splunk — Fri, 10 Jan 2014 21:27:38 GMT

Also, on the bandwidth consumption concern, note that if you enable useclientSSLcompression on the universal forwarder, you will achieve significant compression benefit. See here:

http://answers.splunk.com/answers/92067/forwarder-output-compression-ratio

The heavyweight forwarder is not nearly as efficient. General rule of thumb - if you are dropping more than about 60% of the traffic, then it warrants an HWF.