Splunk Enterprise

Hadoop Data Roll - Error Searching an S3 Archived Index

SteveE
New Member

I am running Splunk Enterprise 8.0.6 and have Hadoop Data Roll configured, using Hadoop 3.2.1 with Java 1.8.0_282-b08. I have a virtual index configured to archive an index to AWS S3. The Hadoop Data Roll archiving process to S3 works, and the archived index is created in S3. However, when I try to search that archived index located in S3, I get the error below (which is from search.log). Has anyone run into this problem and know of a solution?

 

04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  SearchOutputStream - java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  java.lang.Exception: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1569)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  Caused by: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.force(ConfU.java:39)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createV3(PathResolver.java:229)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1553)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	... 3 more
04-16-2021 08:59:02.038 INFO  ERP.s3_provider -  SplunkMR - finishing, version=6.2 ...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - Stopping s3a-file-system metrics system...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system stopped.
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system shutdown complete.
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsConfig - Loaded properties from hadoop-metrics2.properties
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system started
04-16-2021 08:59:02.092 ERROR ERP.s3_provider -   Error while invoking command: /opt/hadoop/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255
04-16-2021 08:59:02.092 INFO  ERPSearchResultCollector - ERP peer=s3_provider is done reading search results.

 

 

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...