Getting Data In

Splunk forwarder throughput to indexer doesn't improve even after giving unlimited bandwidth maxKbps=0

rajakannan
Engager

Splunk heavy forwarder throughput to indexer doesn't improve even after giving unlimited bandwidth maxKbps=0 , it's only getting 4MBps on a 24 core box with 128 GB RAM reading from nfs mount and forwarding to indexer on a 2x 10Gbps on a bonded interface.

Reading from NFS is not an issues as we were able to read/write at 30MB/s outside the forwarder using typical copy (cp)

What are the other limiting factors and what else can we tune from the Splunk side ? Please advise.
Also we noted it's using only 1 TCP connection to indexer.

0 Karma
1 Solution

rajakannan
Engager

MaxQueueSize=30MB was increased from the default values in the output.conf increased the performance significantly. I Would expect the forwarder to auto tune itself to meet the demand based on the system capacity and configurations, not really impressed with the defaults.

View solution in original post

rajakannan
Engager

MaxQueueSize=30MB was increased from the default values in the output.conf increased the performance significantly. I Would expect the forwarder to auto tune itself to meet the demand based on the system capacity and configurations, not really impressed with the defaults.

martin_mueller
SplunkTrust
SplunkTrust

Check what's the bottleneck by using the forwarder's metrics logs in _internal. Possible culprits include inefficient regular expressions for filtering/routing/masking.
Also check if your indexer is busy or not, the distributed management console will help there.

Once that's checked and optimized, you can tell the forwarder to use multiple pipeline sets to parallelize ingestion, processing, and indexing.
http://docs.splunk.com/Documentation/Forwarder/6.4.3/Forwarder/Configureaforwardertohandlemultiplepi...

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...