topic Re: How do I forward log files to Hunk? in Getting Data In

How do I forward log files to Hunk?

gibu_george — Tue, 23 Aug 2016 14:58:49 GMT

Hi All,

I am new to Hunk and would like to know if it is possible to directly send log file data to Hunk using the Splunk forwarder?
What is the recommended way to do this? Could you also share a link to docs for the same?

Thanks in advance.
--gibu

Re: How do I forward log files to Hunk?

ChrisG — Tue, 23 Aug 2016 16:17:12 GMT

Are you talking about forwarding typical log data to a regular Splunk index? If so, Hunk works exactly the way Splunk Enterprise does, and you can use the Splunk Universal Forwarder documentation.

Re: How do I forward log files to Hunk?

kschon_splunk — Wed, 24 Aug 2016 22:04:00 GMT

By "forwarding to Hunk" I assume you mean placing the data on HDFS (or another Hadoop-compatible file system) to be searched via a virtual index?

The most common workflow is to add the log files to a regular Splunk index (using any input method, such as a Splunk forwarder), and set up that index to archive to HDFS. This means you can use a Splunk-managed index to get fast performance on the most recent data, and you can use Hadoop (via Hunk) to search the much larger pool of older data. You can find more information about getting data into Splunk indexes here:

http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Getstartedwithgettingdatain

You can find documentation about archiving here:

https://docs.splunk.com/Documentation/Hunk/6.4.3/Hunk/ArchivingSplunkindexes

If you want to copy the data directly to HDFS without first adding it to a regular Splunk index, you cannot currently do this via a Splunk forwarder. There are a number of third party tools that can be used to do this, e.g. Apache Flume (https://flume.apache.org/).