Kafka Messaging Modular Input: How to set up multiple heavy forwarder agents on the same server to pull from the same Kafka topic?

[pcressy]

I tried to set up multiple heavy forwarder agents on the same server pulling from a Kafka topic, but it appeared as if only one agent would take any load (I was monitoring agent via top cmd, java cpu load, maybe there is a better method, say via index=_internal?). I've done a similar thing with jms with success, but I wasn't able to get past 300 tps. I haven't tried the same thing over multiple servers yet, but I'm wondering if there is a obvious setting, maybe in the inputs.conf that I need to modify.

[Damien_Dallimor]

What does your Kafka configuration (from inputs.conf) look like ?

Also, try the latest release of the Kafka Mod Input.It should perform better with the new HEC output option.

http://blogs.splunk.com/2015/09/22/turbo-charging-modular-inputs-with-the-hec-http-event-collector-i...

[pcressy]

Thanks for the reply I'll try 8.1
here is the inputs.conf I'm currently using
[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181

[pcressy]

Thanks for the reply I'll try 8.1.
Here is the inputs.conf I'm currently using:

[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181