Export large volume of historical data from Splunk to Hadoop(HDFS) which can be implemented in production as well

[dipranjan]

Hello Splunk Team,
I have  been exploring how to connect SPLUNK with Hadoop to export large volume of data(Historical). Could you please help me and provide us the best way to export data from Splunk to Hadoop(HDFS). We learned while exploring that Hadoop connect would be a way but it is now your legacy product and we cannot implement that in production. We also explored Hadoop Data Roll but it can only export in particular data format. We wanted to know the best method available for exporting large volume from Splunk to HDFS. 

[dipranjan]

One of the use case is to export all the metadata from Splunk to Hadoop and it is in GB's. Rest API process may take months to do that and we are looking for better solution. HDR seems to be a solution for exporting in native Splunk format. Is there anyway to export different file formats from splunk and dump on HDFS in let say csv, json etc

[richgalloway]

What are your requirements for exporting data?  It appears you don't want the data in native Splunk format so what format do you want?  How much data will you be exporting?

[dipranjan]

By the way we wanted to know the best way to export data from splunk to hadoop even if it is in native splunk format and it can also be implemented in production. Hadoop connect is legacy and not supported any further and due to this we can't implement.

If let say native splunk format is only possibility then how to read or convert them into other formats? such as CSV or JSON