topic Re: Splunk: Archive to S3 or S3 compatible Object Store using Hadoop ? in Getting Data In

Splunk: Archive to S3 or S3 compatible Object Store using Hadoop ?

sambhram — Wed, 30 Sep 2020 02:05:55 GMT

I am seeing the following error message while trying to archive to S3. The logs are from "splunk_archiver.log". Any pointer as how to fix this ?

2019-09-09 06:09:11.127 -0700 ERROR Roller - Exception when deciding if bucket should be copied. bucket_name=db_1566816265_1543370757_53_D55DFE1B-5E59-4A40-A430-7A8334A5FE47, splunk_index=main, dir=/opt/splunk/var/lib/splunk/defaultdb/db/db_1566816265_1543370757_53_D55DFE1B-5E59-4A40-A430-7A8334A5FE47
2019-09-09 06:20:54.669 -0700 ERROR Roller - Exception when deciding if bucket should be copied. bucket_name=db_1553167044_1545418447_34_D55DFE1B-5E59-4A40-A430-7A8334A5FE47, splunk_index=main, dir=/opt/splunk/var/lib/splunk/defaultdb/db/db_1553167044_1545418447_34_D55DFE1B-5E59-4A40-A430-7A8334A5FE47
2019-09-09 06:33:11.420 -0700 ERROR Roller - Exception when deciding if bucket should be copied. bucket_name=db_1554020317_1546292628_98_D55DFE1B-5E59-4A40-A430-7A8334A5FE47, splunk_index=main, dir=/opt/splunk/var/lib/splunk/defaultdb/db/db_1554020317_1546292628_98_D55DFE1B-5E59-4A40-A430-7A8334A5FE47
2019-09-09 06:44:50.075 -0700 ERROR Roller - Exception when deciding if bucket should be copied. bucket_name=db_1549687163_1546343729_93_D55DFE1B-5E59-4A40-A430-7A8334A5FE47, splunk_index=main, dir=/opt/splunk/var/lib/splunk/defaultdb/db/db_1549687163_1546343729_93_D55DFE1B-5E59-4A40-A430-7A8334A5FE47
2019-09-09 06:57:14.182 -0700 ERROR Roller - Exception when deciding if bucket should be copied. bucket_name=db_1548954761_1546346643_1_D55DFE1B-5E59-4A40-A430-7A8334A5FE47, splunk_index=main, dir=/opt/splunk/var/lib/splunk/defaultdb/db/db_1548954761_1546346643_1_D55DFE1B-5E59-4A40-A430-7A8334A5FE47

Re: Splunk: Archive to S3 or S3 compatible Object Store using Hadoop ?

rdagan_splunk — Mon, 23 Sep 2019 14:39:42 GMT

It looks as if these buckets that have errors are going back 10 months (November 27, 2018).
Can you check if the data is still in Splunk? If it is, can you check if these buckets made it to S3 despite of the exception?

Re: Splunk: Archive to S3 or S3 compatible Object Store using Hadoop ?

sambhram — Wed, 02 Oct 2019 00:35:08 GMT

Yes the data are present in Splunk Indexer and it never make to S3 (the archive target)

Re: Splunk: Archive to S3 or S3 compatible Object Store using Hadoop ?

rdagan_splunk — Fri, 04 Oct 2019 22:48:17 GMT

Can I assume that majority of the buckets made it to S3, but the above 5 did not?
Are you using S3A or S3 in the VIX? S3A does not have size limitation, so I am trying to eliminate that as the cause

Re: Splunk: Archive to S3 or S3 compatible Object Store using Hadoop ?

sambhram — Wed, 16 Oct 2019 20:49:37 GMT

Nothing move to S3 at all. I am using S3A

Re: Splunk: Archive to S3 or S3 compatible Object Store using Hadoop ?

rdagan_splunk — Fri, 25 Oct 2019 21:13:25 GMT

Let me know and we can try to debug these issues? My email is rdagan@splunk.com