Getting Data In

Can a Splunk forwarder send data to Apache Kafka and then to our Splunk cluster? Bonus: Are heavy forwarders deprecated?

ESMaletMa
Explorer

Hi

Due to architecture reasons I need to use Apache Kafka as a message broker between Splunk Forwarders and Splunk cluster.

So, the data flow would be something like:

Splunk Forwarder ----(SSL)---> Kafka Topic ----(SSL)---> Splunk Indexers

So my questions would be:

  1. Can Splunk forwarder send data directly to Kafka topic? I see the same question in 2015, we are in 2017. The answer was NO. Is it the same answer today?

    https://answers.splunk.com/answers/234448/can-splunk-forwarder-universalheavyweight-send-dat.html?ut...

  2. I see that Indexers can read from Kafka using modular inputs or add-ons so, this point shouldn't be a problem.

  3. Can Splunk send data to Kafka topic? (in order for instance to send alerts to other platforms) I see the answer is no, is it correct in 2017:

    https://answers.splunk.com/answers/551309/can-i-export-data-from-splunk-to-kafka-topic-with-1.html?u...

Both links above suggest use Heavy Forwarders. Are Heavy Forwarders deprecated? I have heard that. Is it recommended to use them to provide a solution for this?

Thanks

rdagan_splunk
Splunk Employee
Splunk Employee

Hi, Although this data is not cooked (it does not contained timestamp, host, etc ..) you can send data to third party: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd
From there send it to Kafka.

0 Karma

ddrillic
Ultra Champion
0 Karma

ddrillic
Ultra Champion

I hear that people skip altogether the forwarders and use Kafka instead ... anybody has more insight into it?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...