Getting Data In

Can a Splunk forwarder send data to Apache Kafka and then to our Splunk cluster? Bonus: Are heavy forwarders deprecated?

ESMaletMa
Explorer

Hi

Due to architecture reasons I need to use Apache Kafka as a message broker between Splunk Forwarders and Splunk cluster.

So, the data flow would be something like:

Splunk Forwarder ----(SSL)---> Kafka Topic ----(SSL)---> Splunk Indexers

So my questions would be:

  1. Can Splunk forwarder send data directly to Kafka topic? I see the same question in 2015, we are in 2017. The answer was NO. Is it the same answer today?

    https://answers.splunk.com/answers/234448/can-splunk-forwarder-universalheavyweight-send-dat.html?ut...

  2. I see that Indexers can read from Kafka using modular inputs or add-ons so, this point shouldn't be a problem.

  3. Can Splunk send data to Kafka topic? (in order for instance to send alerts to other platforms) I see the answer is no, is it correct in 2017:

    https://answers.splunk.com/answers/551309/can-i-export-data-from-splunk-to-kafka-topic-with-1.html?u...

Both links above suggest use Heavy Forwarders. Are Heavy Forwarders deprecated? I have heard that. Is it recommended to use them to provide a solution for this?

Thanks

rdagan_splunk
Splunk Employee
Splunk Employee

Hi, Although this data is not cooked (it does not contained timestamp, host, etc ..) you can send data to third party: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd
From there send it to Kafka.

0 Karma

ddrillic
Ultra Champion
0 Karma

ddrillic
Ultra Champion

I hear that people skip altogether the forwarders and use Kafka instead ... anybody has more insight into it?

0 Karma
Get Updates on the Splunk Community!

Buttercup Games: Further Dashboarding Techniques (Part 7)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

What are Community Office Hours? Community Office Hours is an interactive 60-minute Zoom series where ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...