Splunk Dev

Hadoop Data Roll cannot create a directory - because it sees it as a file?

derick_kotze
New Member

I am trying to configure Hadoop Data Roll to archive data to a S3-compatible data store.

I can confirm that I can access the data store via s3cmd (https://s3tools.org/usage) as well as Hadoop on the local machine.

Below is the error that I'm seeing after the | archivebuckets saved search ran.

2019-11-19 18:17:09.064 +0200 ERROR SearchOutputStream - java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
java.lang.Exception: java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
            at com.splunk.mr.SplunkMR.run(SplunkMR.java:1567)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
            at com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)
Caused by: java.lang.RuntimeException: {exception=org.apache.hadoop.fs.FileAlreadyExistsException, message=Can't make directory for path 's3a://sam-splunk-archive/_internal_archive/tmp' since it is a file., stack=[org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1731), org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1681), org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216), com.splunk.roll.Transactor.prepare(Transactor.java:84), com.splunk.roll.Roller.doCopyBucket(Roller.java:158), com.splunk.roll.Roller.copy(Roller.java:87), com.splunk.roll.RollHandler.doCopyFromSplunkIndex(RollHandler.java:182), com.splunk.roll.RollHandler.executeCopyFromSplunkIndex(RollHandler.java:145), com.splunk.roll.RollHandler.executeCopyToVix(RollHandler.java:130), com.splunk.roll.RollHandler.execute(RollHandler.java:68), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802), com.splunk.mr.SplunkMR.run(SplunkMR.java:1551), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1839)]}
            at com.splunk.roll.RollHandler.execute(RollHandler.java:82)
            at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1802)
            at com.splunk.mr.SplunkMR.run(SplunkMR.java:1551)
            ... 3 more

The following message also appears - which ties back to the error in the message above.

alt text

I can see that directories have been created in the bucket - but no actual data has been transferred.

Any ideas on how to resolve the issue? I'm pretty sure that I'm missing something obvious ... so any help will be appreciated.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...