I am trying to configure Hadoop Data Roll to archive data to a S3-compatible data store.
I can confirm that I can access the data store via s3cmd (https://s3tools.org/usage) as well as Hadoop on the local machine.
Below is the error that I'm seeing after the | archivebuckets
saved search ran.
2019-11-19 18:17:09.064 +0200 ERROR SearchOutputStream - java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
java.lang.Exception: java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
at com.splunk.mr.SplunkMR.run(SplunkMR.java:1567)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)
Caused by: java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
at com.splunk.roll.RollHandler.execute(RollHandler.java:82)
at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802)
at com.splunk.mr.SplunkMR.run(SplunkMR.java:1551)
... 3 more
The following message also appears - which ties back to the error in the message above.
I can see that directories have been created in the bucket - but no actual data has been transferred.
Any ideas on how to resolve the issue? I'm pretty sure that I'm missing something obvious ... so any help will be appreciated.