All Apps and Add-ons

Kafka Messaging Modular Input: How to set up multiple heavy forwarder agents on the same server to pull from the same Kafka topic?

pcressy
Engager

I tried to set up multiple heavy forwarder agents on the same server pulling from a Kafka topic, but it appeared as if only one agent would take any load (I was monitoring agent via top cmd, java cpu load, maybe there is a better method, say via index=_internal?). I've done a similar thing with jms with success, but I wasn't able to get past 300 tps. I haven't tried the same thing over multiple servers yet, but I'm wondering if there is a obvious setting, maybe in the inputs.conf that I need to modify.

0 Karma

Damien_Dallimor
Ultra Champion

What does your Kafka configuration (from inputs.conf) look like ?

Also, try the latest release of the Kafka Mod Input.It should perform better with the new HEC output option.

http://blogs.splunk.com/2015/09/22/turbo-charging-modular-inputs-with-the-hec-http-event-collector-i...

0 Karma

pcressy
Engager

Thanks for the reply I'll try 8.1
here is the inputs.conf I'm currently using
[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181

0 Karma

pcressy
Engager

Thanks for the reply I'll try 8.1.
Here is the inputs.conf I'm currently using:

[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxx
zookeeper_connect_port = 2181

[kafka://tsys_kafka_10.xxxxx]
group_id = splunk
index = tsys
sourcetype = tsys_kafka
topic_name = splunk_auth_stream
zookeeper_connect_host = 10.xxxx
zookeeper_connect_port = 2181
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...