Getting Data In

Can a Splunk forwarder send data to Apache Kafka and then to our Splunk cluster? Bonus: Are heavy forwarders deprecated?

ESMaletMa
Explorer

Hi

Due to architecture reasons I need to use Apache Kafka as a message broker between Splunk Forwarders and Splunk cluster.

So, the data flow would be something like:

Splunk Forwarder ----(SSL)---> Kafka Topic ----(SSL)---> Splunk Indexers

So my questions would be:

  1. Can Splunk forwarder send data directly to Kafka topic? I see the same question in 2015, we are in 2017. The answer was NO. Is it the same answer today?

    https://answers.splunk.com/answers/234448/can-splunk-forwarder-universalheavyweight-send-dat.html?ut...

  2. I see that Indexers can read from Kafka using modular inputs or add-ons so, this point shouldn't be a problem.

  3. Can Splunk send data to Kafka topic? (in order for instance to send alerts to other platforms) I see the answer is no, is it correct in 2017:

    https://answers.splunk.com/answers/551309/can-i-export-data-from-splunk-to-kafka-topic-with-1.html?u...

Both links above suggest use Heavy Forwarders. Are Heavy Forwarders deprecated? I have heard that. Is it recommended to use them to provide a solution for this?

Thanks

rdagan_splunk
Splunk Employee
Splunk Employee

Hi, Although this data is not cooked (it does not contained timestamp, host, etc ..) you can send data to third party: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd
From there send it to Kafka.

0 Karma

ddrillic
Ultra Champion
0 Karma

ddrillic
Ultra Champion

I hear that people skip altogether the forwarders and use Kafka instead ... anybody has more insight into it?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...