I am trying to configure Hadoop Data Roll to archive data to a S3-compatible data store.
I can confirm that I can access the data store via s3cmd (https://s3tools.org/usage) as well as Hadoop on the local machine.
Below is the error that I'm seeing after the | archivebuckets saved search ran.
2019-11-19 18:17:09.064 +0200 ERROR SearchOutputStream - java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
java.lang.Exception: java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
at com.splunk.mr.SplunkMR.run(SplunkMR.java:1567)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)
Caused by: java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
at com.splunk.roll.RollHandler.execute(RollHandler.java:82)
at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802)
at com.splunk.mr.SplunkMR.run(SplunkMR.java:1551)
... 3 more
The following message also appears - which ties back to the error in the message above.
I can see that directories have been created in the bucket - but no actual data has been transferred.
Any ideas on how to resolve the issue? I'm pretty sure that I'm missing something obvious ... so any help will be appreciated.
... View more
