Configure forwarder to read 3MBps from server

[jayannah]

I have to Config Splunk forwarder to read 3MBps data from JMS queue. Installed heavy forwarder on the server and configured to read the JMS queue with 20 parallel connections per Queue. Total of 6 Queues. I have increased the splunk throughput rate in limits.conf to 4000 KBps. But still, I couldn't able to read more than 1MBps. What could be the issue? Could you please help if anyone has tried similar or larger configuration than mine.

[jayannah]

Finally, I could achieve the desired result by scaling the spunk forwarders vertically i.e configured the multiple splunk forwarders on the same system with different management port & 20 parallel tcp connections to each JMS queue and able to read in JMS data in real time (about 0-2 sec latency).

[ndoshi · Accepted solution]

Why use a heavy forwarder? Could you not use an Universal Forwarder which scales better? You may try to use maxthruput in limits.conf to -1 to not throttle it at all. Also, you could also set up multiple forwarders on the same machine (make sure they listen on a different port) and each one can monitor, say 2 queues, at a time. That way, you'll have more listeners. Each forwarder can be configured to send to more than one indexer (auto load balance) and that may also help you scale.

[jayannah]

Here is the CO :

Server (JMS Queue + Splunk heavyforwarder) ------------> SPlunk Indexer (also SH)
SPlunk Heavyforwarder throughput set to 4000 KBps

On the SH when I issue the below command, I see that max(kbps), avg(kbps) of heavyforwarder and max(kbps) of indexers are almost same all the time, but avg(kbps) of indexer is 50% of above 3 stats. I thought avg(kbps) of indexer and heavyforwarder should be same.