All Apps and Add-ons

Configure forwarder to read 3MBps from server

jayannah
Builder

I have to Config Splunk forwarder to read 3MBps data from JMS queue. Installed heavy forwarder on the server and configured to read the JMS queue with 20 parallel connections per Queue. Total of 6 Queues. I have increased the splunk throughput rate in limits.conf to 4000 KBps. But still, I couldn't able to read more than 1MBps. What could be the issue? Could you please help if anyone has tried similar or larger configuration than mine.

0 Karma
1 Solution

ndoshi
Splunk Employee
Splunk Employee

Why use a heavy forwarder? Could you not use an Universal Forwarder which scales better? You may try to use maxthruput in limits.conf to -1 to not throttle it at all. Also, you could also set up multiple forwarders on the same machine (make sure they listen on a different port) and each one can monitor, say 2 queues, at a time. That way, you'll have more listeners. Each forwarder can be configured to send to more than one indexer (auto load balance) and that may also help you scale.

View solution in original post

jayannah
Builder

Finally, I could achieve the desired result by scaling the spunk forwarders vertically i.e configured the multiple splunk forwarders on the same system with different management port & 20 parallel tcp connections to each JMS queue and able to read in JMS data in real time (about 0-2 sec latency).

ndoshi
Splunk Employee
Splunk Employee

Why use a heavy forwarder? Could you not use an Universal Forwarder which scales better? You may try to use maxthruput in limits.conf to -1 to not throttle it at all. Also, you could also set up multiple forwarders on the same machine (make sure they listen on a different port) and each one can monitor, say 2 queues, at a time. That way, you'll have more listeners. Each forwarder can be configured to send to more than one indexer (auto load balance) and that may also help you scale.

jayannah
Builder

Here is the CO :

Server (JMS Queue + Splunk heavyforwarder) ------------> SPlunk Indexer (also SH)
SPlunk Heavyforwarder throughput set to 4000 KBps

On the SH when I issue the below command, I see that max(kbps), avg(kbps) of heavyforwarder and max(kbps) of indexers are almost same all the time, but avg(kbps) of indexer is 50% of above 3 stats. I thought avg(kbps) of indexer and heavyforwarder should be same.

0 Karma
Get Updates on the Splunk Community!

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...