All Apps and Add-ons

HUNK vs Splunk Hadoop Connect: How to move data from Splunk to Hadoop and search the data in Hadoop?

kml_uvce
Builder

my requirement is to move data from splunk to hadoop and also search the data in hadoop

0 Karma

ChrisG
Splunk Employee
Splunk Employee

Hunk is for searching and analyzing data stored in Hadoop. Hadoop Connect enables you to move data in both directions between Splunk Enterprise and Hadoop. It's not an either/or. For your requirements, it's both.

See these two documentation topics for the overview:
- About Splunk Hadoop Connect
- Meet Hunk

kml_uvce
Builder

I think , I can also move data from splunk to hadoop by archiving old data in indexes and send it to hadoop

but in hadoop connect I can send aggregated search result data in Hadoop If I want but in HUNK I can not do this , I am not sure about it..
Also to read data from Hadoop to splunk ,HUNK does not require to index data in splunk but in hadoop connect we need to index data from Hadoop before reading ?

I heard HUNK is not stable , is this true ? please share your thoughts on this..

0 Karma

ChrisG
Splunk Employee
Splunk Employee

rdagan answered your archiving question, my apologies for leaving out that detail!

Hunk is stable and reliable, what specifically did you hear to the contrary?

0 Karma

kml_uvce
Builder

which one is better in terms of easy to use ?
Also to read/search data from Hadoop to splunk ,HUNK does not require to index data in splunk but in hadoop connect we need to first index data from Hadoop before reading/searching ?

0 Karma

ChrisG
Splunk Employee
Splunk Employee

Hunk uses virtual indexes to enable searching of Hadoop data.

Hadoop Connect imports the data into Splunk Enterprise, where it gets indexed like other data. Hadoop Connect provides an exploration feature so you can preview the files you want to index in Splunk Enterprise. You can also use the HDFS Explore feature to read results of a MapReduce job and display it alongside Splunk search results. If you use HDFS Explore in this way, the data is not indexed, but the feature is much more limited compared with regular Splunk search capabilities that are available in Hunk.

kml_uvce
Builder

Thanks Chris
Can you please give me some scenario/use cases where we should use HUNK and where we should use Splunk Hadoop Connect ?

0 Karma

ChrisG
Splunk Employee
Splunk Employee

If you have not already done so, I strongly suggest you look at the materials available on the Hunk product page on splunk.com, including the white paper and the customer stories. The Hadoop Connect use cases are pretty simple and are well-described on the Hadoop Connect app page.

If you have more detailed questions about the use cases for these products, you should contact Splunk Sales for more information.

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

After the data was indexed in Splunk, you can use the below options to move data to HDFS
1) Hadoop Connect Export. This is the output of a Search and is done from the Search Head to HDFS
2) Hunk Archiving. For this option we copy the raw data directly from the Indexer (journal.gz file) to HDFS

0 Karma