topic Re: How archiving splunk logs to hdfs work in Getting Data In

How archiving splunk logs to hdfs work

gaddyh — Tue, 23 Jun 2015 13:47:29 GMT

I know that Hunk issues the archivebuckets command, which will start the archiving process on each indexer.
What is the archiving process?

Do all indexer machines need access to hadoop (hdfs)?
Does hunk copy the files locally and than to hdfs?

please explain exactly what happens in this process.

Re: How archiving splunk logs to hdfs work

acharlieh — Tue, 23 Jun 2015 14:17:49 GMT

The announcement blog indicates all indexers need Java and Hadoop clients so I would guess that they're talking directly to HDFS.

Re: How archiving splunk logs to hdfs work

rdagan_splunk — Tue, 23 Jun 2015 16:01:12 GMT

Archiving indexers from Splunk to HDFS require the bucket to be in the Warm or Cold stage. Therefore, we can either wait a day, or restart Hunk.
The Splunk_Archiver (New App with Hunk 6.2.1) using the Bundle Replication is distributed to all the Indexers.
Every 60 Minutes, the App on each Indexer executes the search = | archivebuckets This command will Triggers bucket copying From the Indexer to HDFS (Similar to Hadoop fs –put BUCKET /HDFS)

Re: How archiving splunk logs to hdfs work

Ledion_Bitincka — Tue, 23 Jun 2015 19:17:16 GMT

Do all indexer machines need access to hadoop (hdfs)?
Yes, they need network access, Java and Hadoop libraries (ie you should be able to successfully run hadoop fs -copyFromLocal .... from every indexer)

Does hunk copy the files locally and than to hdfs?
No, the buckets are copied from indexers directly to HDFS

Re: How archiving splunk logs to hdfs work

gaddyh — Sun, 28 Jun 2015 09:14:56 GMT

Which ports does Hunk use to connect to the splunk indexers?

What Splunk version need to be installed on the indexer machines to support archiving indexes?
What else needs to be installed on the Splunk indexer machines?

Re: How archiving splunk logs to hdfs work

Ledion_Bitincka — Mon, 29 Jun 2015 17:55:02 GMT

To connect to indexers, the same Splunk Enterprise ports are needed (8089 by default). For Hadoop, ports depend on the Hadoop distro - indexers need access to the Secondary/Namenode (usually port 8020), and DataNode transfer ports (usually port 50010). Indexer Splunk version doesn't matter, but you need Java and Hadoop libraries on the indexers.