topic Re: Is Splunk Hadoop Connect app supported in a clustered Search head? in All Apps and Add-ons

Is Splunk Hadoop Connect app supported in a clustered Search head?

saranya_fmr — Wed, 20 Sep 2017 10:19:57 GMT

Goal is to set export Data from Splunk Indexes to Hadoop.

We have a clustered Search head cluster. The doc doesn't mention about deploying the Splunk Hadoop Connector app in a cluster.

The below page says its not supported in a clustered SH.
https://answers.splunk.com/answers/368847/is-hadoop-connect-supported-in-a-splunk-search-hea.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev

So my Question is:

Is it supported in a Search Head Cluster? If no , what are the other options to send Splunk Index data into Hadoop ?

Re: Is Splunk Hadoop Connect app supported in a clustered Search head?

sloshburch — Wed, 20 Sep 2017 19:13:49 GMT

If you want to get data export to indexers, how about Hadoop Data Roll?

Brief answer cause I wasn't positive why and the specifics about getting the data to Hadoop. Knowing more about that would help provide alternative solutions for this. Cool?

Re: Is Splunk Hadoop Connect app supported in a clustered Search head?

saranya_fmr — Fri, 22 Sep 2017 12:20:41 GMT

Hi BUrch,

The goal is to move the data from Splunk to Hadoop/S3 for longer data retention. Currently we store data in Splunk only for two months.
We want to send data to Hadoop and the analytics team wants to further analyse this HIVE , etc
I understand
1) We can simply forward data to another system as it arrives at Splunk
or
2) we can export data after it has been cooked by Splunk.
However not sure how to achieve first part.
Hadoop connect allows data export to Hadoop but it doesnt seem to be supported in a clustered architecture. Thus looking out for options.

I believe Hadoop Data Roll is used when I want to use Splunk again for archived data analytics, which is not needed for us. This archived index will follow the same two month retention period right? -- Please correct me if Im wrong.

Re: Is Splunk Hadoop Connect app supported in a clustered Search head?

sloshburch — Fri, 22 Sep 2017 12:34:39 GMT

Thanks for the extra detail. It sounds like Hadoop Data Roll is exactly what you want. It's the ideal way to roll data from Splunk to Hadoop. Yes it will allow you to search it still but you don't have to do that AND it will have DIFFERENT retention than the index did in Splunk. Read through the documentation and I'm confident you'll agree that it's 100% exactly what you want to use.

Re: Is Splunk Hadoop Connect app supported in a clustered Search head?

saranya_fmr — Fri, 22 Sep 2017 13:51:19 GMT

Thakyou for the notes. I will go through the documentation and get back incase of further queries.

Re: Is Splunk Hadoop Connect app supported in a clustered Search head?

saranya_fmr — Tue, 26 Sep 2017 11:16:16 GMT

Hi @SloshBurch ,

I did go through the documentation and had few queries - I have posted these queries at the below link as well:

https://answers.splunk.com/answers/577310/difference-between-data-format-for-the-data-sent-t.html?minQuestionBodyLength=80

Firstly what is the difference between the data format of data that is sent via Hadoop Connect Export and Hadoop Data Roll? I believe Hadoop Connect exports search results and Hadoop Data Roll send the raw data journal.gz
Can I use Hadoop Techniques like Hive, Pig..etc for analytics on the archived data sent to Hadoop via Hadoop Data Roll?
AND
Can I use Hadoop Techniques like Hive, Pig..etc for analytics on the archived data sent to Hadoop via Splunk Hadoop Connect Export?
I came across Splunk Archive Bucket Reader - an additional app is required to analyze the archived data via Hadoop's applications like Pig , Hive , Spark.
Is this a mandatory app required If I want to analyse the Hadoop data?
Is Splunk Analytics for Hadoop license license required for sending archived data to Hadoop?

Re: Is Splunk Hadoop Connect app supported in a clustered Search head?

sloshburch — Fri, 06 Oct 2017 21:45:12 GMT

Thanks @saranya_fmr. I was at .conf2017 and then another work trip hence the delayed response. I will respond in the new thread you created. Thanks for starting that.

Also, if we answered your initial question, please make sure to accept that answer so others know if its worth exploring this thread or not.