topic Re: Is there an Integration between Spark and Splunk? in Getting Data In

Is there an Integration between Spark and Splunk?

mhsilva — Thu, 28 Mar 2019 12:56:14 GMT

Hello everyone,

I want to integrate Spark and Splunk, using Spark to process searches faster.

With Splunk Analytics for Hadoop, I can set a HDFS as a Virtual Indexer, but this uses a Hadoop/MapReduce to get data. How can I use Spark instead of? Thanks everyone

Ps: i tried to sign up and get Splunk MLTK Connector for Apache Spark, but apparently, there are no vacancies.

Re: Is there an Integration between Spark and Splunk?

adonio — Thu, 28 Mar 2019 20:15:24 GMT

3 great presentations that i hope will take you wherever you need:

https://conf.splunk.com/files/2017/slides/advanced-analytics-with-splunk-using-apache-spark-machine-learning-and-spark-graph.pdf
https://conf.splunk.com/files/2016/slides/splunk-and-open-source-integrations-with-spark-solr-hadoop-and-nosql-storage.pdf
https://conf.splunk.com/files/2017/slides/unleash-your-machine-data-with-context-from-historical-and-transactional-data-using-hadoop-rdbms-and-splunk.pdf

hope it helps

Re: Is there an Integration between Spark and Splunk?

ddrillic — Thu, 28 Mar 2019 20:26:19 GMT

Right, that was the basic design -

-- With Splunk Analytics for Hadoop, i can set a hdfs as a Virtual Indexer, but this uses a Hadoop/MapReduce to get data

I wonder if the product has evolved... @rdagan_splunk can shed light on this topic.

Re: Is there an Integration between Spark and Splunk?

mhsilva — Fri, 29 Mar 2019 11:21:10 GMT

Thank you for this, but these solutions propose using spark only to process data, but still use Splunk Analytics
for Hadoop to search the data, so it still use MapReduce, or not?
Thanks again

Re: Is there an Integration between Spark and Splunk?

rdagan_splunk — Tue, 05 May 2020 17:02:41 GMT

Hi, these solutions actually used Splunk DB Connect with JDBC to talk to Spark SQL