Splunk Enterprise

Hadoop Data Roll - Error Searching an S3 Archived Index

SteveE
New Member

I am running Splunk Enterprise 8.0.6 and have Hadoop Data Roll configured, using Hadoop 3.2.1 with Java 1.8.0_282-b08. I have a virtual index configured to archive an index to AWS S3. The Hadoop Data Roll archiving process to S3 works, and the archived index is created in S3. However, when I try to search that archived index located in S3, I get the error below (which is from search.log). Has anyone run into this problem and know of a solution?

 

04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  SearchOutputStream - java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  java.lang.Exception: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1569)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  Caused by: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.force(ConfU.java:39)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createV3(PathResolver.java:229)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1553)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	... 3 more
04-16-2021 08:59:02.038 INFO  ERP.s3_provider -  SplunkMR - finishing, version=6.2 ...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - Stopping s3a-file-system metrics system...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system stopped.
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system shutdown complete.
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsConfig - Loaded properties from hadoop-metrics2.properties
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system started
04-16-2021 08:59:02.092 ERROR ERP.s3_provider -   Error while invoking command: /opt/hadoop/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255
04-16-2021 08:59:02.092 INFO  ERPSearchResultCollector - ERP peer=s3_provider is done reading search results.

 

 

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...