https://community.splunk.com/t5/Getting-Data-In/Splunk-forwarder-throughput-to-indexer-doesn-t-improve-even/m-p/248925#M47900 <P>Check what's the bottleneck by using the forwarder's metrics logs in _internal. Possible culprits include inefficient regular expressions for filtering/routing/masking.<BR /> Also check if your indexer is busy or not, the distributed management console will help there.</P> <P>Once that's checked and optimized, you can tell the forwarder to use multiple pipeline sets to parallelize ingestion, processing, and indexing.<BR /> <A href="http://docs.splunk.com/Documentation/Forwarder/6.4.3/Forwarder/Configureaforwardertohandlemultiplepipelinesets">http://docs.splunk.com/Documentation/Forwarder/6.4.3/Forwarder/Configureaforwardertohandlemultiplepipelinesets</A></P> Sat, 27 Aug 2016 22:34:44 GMT martin_mueller 2016-08-27T22:34:44Z https://community.splunk.com/t5/Getting-Data-In/Splunk-forwarder-throughput-to-indexer-doesn-t-improve-even/m-p/248924#M47899 <P>Splunk heavy forwarder throughput to indexer doesn't improve even after giving unlimited bandwidth maxKbps=0 , it's only getting 4MBps on a 24 core box with 128 GB RAM reading from nfs mount and forwarding to indexer on a 2x 10Gbps on a bonded interface.</P> <P>Reading from NFS is not an issues as we were able to read/write at 30MB/s outside the forwarder using typical copy (cp)</P> <P>What are the other limiting factors and what else can we tune from the Splunk side ? Please advise.<BR /> Also we noted it's using only 1 TCP connection to indexer. </P> Sat, 27 Aug 2016 21:21:08 GMT https://community.splunk.com/t5/Getting-Data-In/Splunk-forwarder-throughput-to-indexer-doesn-t-improve-even/m-p/248924#M47899 rajakannan 2016-08-27T21:21:08Z https://community.splunk.com/t5/Getting-Data-In/Splunk-forwarder-throughput-to-indexer-doesn-t-improve-even/m-p/248925#M47900 <P>Check what's the bottleneck by using the forwarder's metrics logs in _internal. Possible culprits include inefficient regular expressions for filtering/routing/masking.<BR /> Also check if your indexer is busy or not, the distributed management console will help there.</P> <P>Once that's checked and optimized, you can tell the forwarder to use multiple pipeline sets to parallelize ingestion, processing, and indexing.<BR /> <A href="http://docs.splunk.com/Documentation/Forwarder/6.4.3/Forwarder/Configureaforwardertohandlemultiplepipelinesets">http://docs.splunk.com/Documentation/Forwarder/6.4.3/Forwarder/Configureaforwardertohandlemultiplepipelinesets</A></P> Sat, 27 Aug 2016 22:34:44 GMT https://community.splunk.com/t5/Getting-Data-In/Splunk-forwarder-throughput-to-indexer-doesn-t-improve-even/m-p/248925#M47900 martin_mueller 2016-08-27T22:34:44Z https://community.splunk.com/t5/Getting-Data-In/Splunk-forwarder-throughput-to-indexer-doesn-t-improve-even/m-p/248926#M47901 <P>MaxQueueSize=30MB was increased from the default values in the output.conf increased the performance significantly. I Would expect the forwarder to auto tune itself to meet the demand based on the system capacity and configurations, not really impressed with the defaults.</P> Sun, 28 Aug 2016 21:38:22 GMT https://community.splunk.com/t5/Getting-Data-In/Splunk-forwarder-throughput-to-indexer-doesn-t-improve-even/m-p/248926#M47901 rajakannan 2016-08-28T21:38:22Z