Getting Data In

Export large volume of historical data from Splunk to Hadoop(HDFS) which can be implemented in production as well

dipranjan
New Member

Hello Splunk Team,
I have  been exploring how to connect SPLUNK with Hadoop to export large volume of data(Historical). Could you please help me and provide us the best way to export data from Splunk to Hadoop(HDFS). We learned while exploring that Hadoop connect would be a way but it is now your legacy product and we cannot implement that in production. We also explored Hadoop Data Roll but it can only export in particular data format. We wanted to know the best method available for exporting large volume from Splunk to HDFS. 

Labels (2)
0 Karma

dipranjan
New Member

One of the use case is to export all the metadata from Splunk to Hadoop and it is in GB's. Rest API process may take months to do that and we are looking for better solution. HDR seems to be a solution for exporting in native Splunk format. Is there anyway to export different file formats from splunk and dump on HDFS in let say csv, json etc

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What are your requirements for exporting data?  It appears you don't want the data in native Splunk format so what format do you want?  How much data will you be exporting?

---
If this reply helps you, Karma would be appreciated.
0 Karma

dipranjan
New Member

By the way we wanted to know the best way to export data from splunk to hadoop even if it is in native splunk format and it can also be implemented in production. Hadoop connect is legacy and not supported any further and due to this we can't implement.

If let say native splunk format is only possibility then how to read or convert them into other formats? such as CSV or JSON

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...