topic Re: Splunk Forwarder to forward 14GB log data daily in Getting Data In

Splunk Forwarder to forward 14GB log data daily

ritchieyang — Thu, 12 Dec 2013 15:51:44 GMT

We would like to know what is the best practice to use Splunk Heavy Forwarder to forward Verint Integration Services log data to a Splunk Indexer which total log size is around 14G each day. Any performance issue on Forwarder? Any special configuration to consider?

Any help are appreciated.

Re: Splunk Forwarder to forward 14GB log data daily

jtrucks — Thu, 12 Dec 2013 16:12:52 GMT

14GB/day is only about 10MB/minute. That means there is not likely much performance issues involved unless the traffic is extremely bursty in large multi-GB chunks in very short time periods, say, several GB in a few seconds.

If this data is spread out over even 5 hours rather than a full 24, that's still under 50MB/minute.

Therefore, I wouldn't be concerned much about performance being an issue on the systems or the network.

Re: Splunk Forwarder to forward 14GB log data daily

ritchieyang — Thu, 12 Dec 2013 16:26:20 GMT

jtrucks, Thanks so much for your confirmation. If we are going to use Regex to filter log data to send to Inderex, any performance concerns on the log parsing process on Forwarder side? Thanks again.

Re: Splunk Forwarder to forward 14GB log data daily

jtrucks — Thu, 12 Dec 2013 17:15:57 GMT

unless you are doing crazy complicated filtering on a rather underpowered machine, I doubt you will see a performance issue. Under 100MB/minute isn't a lot for a modern machine to process, as long as the network isn't congested from other things.

Don't forget to vote up the answer if it works for you, too. 🙂

Re: Splunk Forwarder to forward 14GB log data daily

ritchieyang — Thu, 12 Dec 2013 20:30:49 GMT

jtrucks, I appreciated your fast reply.