https://community.splunk.com/t5/Getting-Data-In/Hunk-bucket-archive-question/m-p/259169#M49749 <P>When HUNK does its bucket pushes to HDFS, it also pushes a couple small supporting files, metadata, etc... With Hadoop's issues handling small files, I was wondering if that is something that's been looked at or not? </P> <P>For using the HUNK archiving on clustered indexers, I understand that if the buckets have 2 or greater searchable copies of the data, searches against that archived data will return duplicated results, is that correct? </P> Tue, 22 Mar 2016 17:25:10 GMT tsunamii 2016-03-22T17:25:10Z https://community.splunk.com/t5/Getting-Data-In/Hunk-bucket-archive-question/m-p/259169#M49749 <P>When HUNK does its bucket pushes to HDFS, it also pushes a couple small supporting files, metadata, etc... With Hadoop's issues handling small files, I was wondering if that is something that's been looked at or not? </P> <P>For using the HUNK archiving on clustered indexers, I understand that if the buckets have 2 or greater searchable copies of the data, searches against that archived data will return duplicated results, is that correct? </P> Tue, 22 Mar 2016 17:25:10 GMT https://community.splunk.com/t5/Getting-Data-In/Hunk-bucket-archive-question/m-p/259169#M49749 tsunamii 2016-03-22T17:25:10Z https://community.splunk.com/t5/Getting-Data-In/Hunk-bucket-archive-question/m-p/259170#M49750 <P>Hi</P> <P>1) jepp. There will be some overhead. Since there are not millions of buckets I don't think this will be an issue... </P> <P>2) Every indexer will try to copy "it's own" bucket to HDFS. If there is already a valid copy of this bucket the indexer will skip this.<BR /> There should be no duplicates.</P> <P>HTH,</P> <P>Holger</P> Tue, 22 Mar 2016 21:29:16 GMT https://community.splunk.com/t5/Getting-Data-In/Hunk-bucket-archive-question/m-p/259170#M49750 hsesterhenn 2016-03-22T21:29:16Z