Getting Data In

Splunk forwarder throughput to indexer doesn't improve even after giving unlimited bandwidth maxKbps=0

rajakannan
Engager

Splunk heavy forwarder throughput to indexer doesn't improve even after giving unlimited bandwidth maxKbps=0 , it's only getting 4MBps on a 24 core box with 128 GB RAM reading from nfs mount and forwarding to indexer on a 2x 10Gbps on a bonded interface.

Reading from NFS is not an issues as we were able to read/write at 30MB/s outside the forwarder using typical copy (cp)

What are the other limiting factors and what else can we tune from the Splunk side ? Please advise.
Also we noted it's using only 1 TCP connection to indexer.

0 Karma
1 Solution

rajakannan
Engager

MaxQueueSize=30MB was increased from the default values in the output.conf increased the performance significantly. I Would expect the forwarder to auto tune itself to meet the demand based on the system capacity and configurations, not really impressed with the defaults.

View solution in original post

rajakannan
Engager

MaxQueueSize=30MB was increased from the default values in the output.conf increased the performance significantly. I Would expect the forwarder to auto tune itself to meet the demand based on the system capacity and configurations, not really impressed with the defaults.

martin_mueller
SplunkTrust
SplunkTrust

Check what's the bottleneck by using the forwarder's metrics logs in _internal. Possible culprits include inefficient regular expressions for filtering/routing/masking.
Also check if your indexer is busy or not, the distributed management console will help there.

Once that's checked and optimized, you can tell the forwarder to use multiple pipeline sets to parallelize ingestion, processing, and indexing.
http://docs.splunk.com/Documentation/Forwarder/6.4.3/Forwarder/Configureaforwardertohandlemultiplepi...

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...