Splunk Forwarder to forward 14GB log data daily

[ritchieyang]

We would like to know what is the best practice to use Splunk Heavy Forwarder to forward Verint Integration Services log data to a Splunk Indexer which total log size is around 14G each day. Any performance issue on Forwarder? Any special configuration to consider?

Any help are appreciated.

[jtrucks]

14GB/day is only about 10MB/minute. That means there is not likely much performance issues involved unless the traffic is extremely bursty in large multi-GB chunks in very short time periods, say, several GB in a few seconds.

If this data is spread out over even 5 hours rather than a full 24, that's still under 50MB/minute.

Therefore, I wouldn't be concerned much about performance being an issue on the systems or the network.

[ritchieyang]

jtrucks, I appreciated your fast reply.

[jtrucks]

unless you are doing crazy complicated filtering on a rather underpowered machine, I doubt you will see a performance issue. Under 100MB/minute isn't a lot for a modern machine to process, as long as the network isn't congested from other things.

Don't forget to vote up the answer if it works for you, too. 🙂

[ritchieyang]

jtrucks, Thanks so much for your confirmation. If we are going to use Regex to filter log data to send to Inderex, any performance concerns on the log parsing process on Forwarder side? Thanks again.