Getting Data In

Can a Splunk forwarder send data to Apache Kafka and then to our Splunk cluster? Bonus: Are heavy forwarders deprecated?

ESMaletMa
Explorer

Hi

Due to architecture reasons I need to use Apache Kafka as a message broker between Splunk Forwarders and Splunk cluster.

So, the data flow would be something like:

Splunk Forwarder ----(SSL)---> Kafka Topic ----(SSL)---> Splunk Indexers

So my questions would be:

  1. Can Splunk forwarder send data directly to Kafka topic? I see the same question in 2015, we are in 2017. The answer was NO. Is it the same answer today?

    https://answers.splunk.com/answers/234448/can-splunk-forwarder-universalheavyweight-send-dat.html?ut...

  2. I see that Indexers can read from Kafka using modular inputs or add-ons so, this point shouldn't be a problem.

  3. Can Splunk send data to Kafka topic? (in order for instance to send alerts to other platforms) I see the answer is no, is it correct in 2017:

    https://answers.splunk.com/answers/551309/can-i-export-data-from-splunk-to-kafka-topic-with-1.html?u...

Both links above suggest use Heavy Forwarders. Are Heavy Forwarders deprecated? I have heard that. Is it recommended to use them to provide a solution for this?

Thanks

rdagan_splunk
Splunk Employee
Splunk Employee

Hi, Although this data is not cooked (it does not contained timestamp, host, etc ..) you can send data to third party: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd
From there send it to Kafka.

0 Karma

ddrillic
Ultra Champion
0 Karma

ddrillic
Ultra Champion

I hear that people skip altogether the forwarders and use Kafka instead ... anybody has more insight into it?

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...