topic Re: Export data to Hadoop in All Apps and Add-ons

Export data to Hadoop

siva_cg — Thu, 18 Oct 2018 11:59:37 GMT

Hi Team,

I want to export data (not archiving data) from Splunk running on version 7.1.3 to Hadoop server in two cases.

1st case - Set of data using Splunk search
2nd case - All the events in a particular sourcetype from an index (I can create a separate index for that specific sourcetype and send whole index data to Hadoop)

In both cases, Can I use Hadoop Connect app to export data? I read that from Splunk version 6.5, we need to use Splunk Analytics for Hadoop but didn't find any document that says about data export using Splunk Analytics for Hadoop. Could you please suggest which app to use? Thanks in advance.

Re: Export data to Hadoop

inventsekar — Thu, 18 Oct 2018 12:34:58 GMT

Hi Siva, i am not sure.. but did you check about "Hadoop Data Roll"

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ArchivingindexestoHadoop

Re: Export data to Hadoop

siva_cg — Thu, 18 Oct 2018 14:04:46 GMT

Hi @inventsekar,
As per my understand, "Hadoop Data Roll" is used to move data from Splunk to Hadoop and use hdfs search for any future reports or searches. But in my usecase, I just want to copy some data to Hadoop server.

Re: Export data to Hadoop

rdagan_splunk — Thu, 18 Oct 2018 16:14:10 GMT

Yes, for both use cases Splunk Hadoop Connect is the right App.
Here is the link to the doc that will help you schedule that search and export:
http://docs.splunk.com/Documentation/HadoopConnect/latest/DeployHadoopConnect/Appdashboard

Just be aware that Splunk will normally recommend a dedicated Search Head if you plan on moving a large amount of data per day.

Re: Export data to Hadoop

siva_cg — Fri, 19 Oct 2018 08:20:25 GMT

Hi @rdagan,

Thank you for your response. Yeah, we will use dedicated Search Head for this purpose. Will Splunk Hadoop Connect support Splunk 7.1.3 version ?

Re: Export data to Hadoop

rdagan_splunk — Mon, 22 Oct 2018 01:05:33 GMT

Regarding your question about Splunk Hadoop Connect support Splunk 7.1.3 version - Splunk Hadoop Connect works without a problem with that version. And my recommendation is for you to ask your Splunk sales team to send that request to Splunk support.

Re: Export data to Hadoop

siva_cg — Mon, 22 Oct 2018 14:05:19 GMT

Thank you @rdagan. Sure, I will contact my Splunk sales team for the request.