Getting Data In

Is there an Integration between Spark and Splunk?

mhsilva
Explorer

Hello everyone,

I want to integrate Spark and Splunk, using Spark to process searches faster.

With Splunk Analytics for Hadoop, I can set a HDFS as a Virtual Indexer, but this uses a Hadoop/MapReduce to get data. How can I use Spark instead of? Thanks everyone

Ps: i tried to sign up and get Splunk MLTK Connector for Apache Spark, but apparently, there are no vacancies.

0 Karma

ddrillic
Ultra Champion

Right, that was the basic design -

-- With Splunk Analytics for Hadoop, i can set a hdfs as a Virtual Indexer, but this uses a Hadoop/MapReduce to get data

I wonder if the product has evolved... @rdagan_splunk can shed light on this topic.

mhsilva
Explorer

Thank you for this, but these solutions propose using spark only to process data, but still use Splunk Analytics
for Hadoop to search the data, so it still use MapReduce, or not?
Thanks again

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Hi, these solutions actually used Splunk DB Connect with JDBC to talk to Spark SQL

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...