I'm planning a Splunk deployment that will involve 2 indexers, 1 search head and 4 forwarders spread across various networks in various geographic locations. We are using a forwarder (UF or HWF, we're not settled on which one yet) to collect syslog traffic on private networks and forward this traffic over a VPN to a remote site where the index and search heads will be located.
In our labs we've spent time testing the forwarding speed we can expect to achieve and found that compression severely affects the maximum EPS results, however Splunk appears to have very low CPU utilisation when compression is enabled.
Servers:
Dell R410
2x Xeon E5606 Quad-core processors
16GB DDR3 RAM
2x 600GB SAS 15k RAID-1 disks
2x Bonded gigabit network cards
Centos 6.2 64-bit minimal
Connected via dedicated gigabit switch
Testing scenario:
2 servers generating UDP syslog datagrams at an EPS level we can control, which forwards to:
1 Splunk forwarder, we've tried both UF and HWF with the same results, which forwards to:
1 full Splunk indexing the events showing us the incoming EPS on a 30s average graph
The most representative example I can give of the problem is this:
Set the two syslog generators to 20k EPS each (40k EPS total)
Disable compression on the forwarder
End Splunk server receives 40k EPS
top reports CPU usage around 250-300%, about what we expect for the hardware based on reading other splunk base posts. We top out around 100k EPS in our testing.
However:
Keep the two syslog generators at 20k EPS each (40k EPS total, still)
Enable compression on the forwarder
End Splunk server receives around 13.5-14.5k EPS, never above 15k EPS
top reports CPU usage around 120%, rarely above 150%
The concerning factor for us is the relatively light CPU usage. We will be deploying forwarders on dedicated hardware and we want Splunk to utilise as much of their power as possible.
Are there any settings in Splunk that we can tune or tweak to make Splunk UF/HWF more "greedy" and help increase our EPS rate with compression? We've tweaked all the options we can find in Splunk inputs.conf, outputs.conf and at a kernel level for UDP buffers and queues. However this just delays the appearance of the problem of being limited to ~14k EPS when compression is enabled, whilst using relatively low resource utilisation.
Thanks!
... View more