Hadoop Data Roll - Error Searching an S3 Archived Index

[SteveE]

I am running Splunk Enterprise 8.0.6 and have Hadoop Data Roll configured, using Hadoop 3.2.1 with Java 1.8.0_282-b08. I have a virtual index configured to archive an index to AWS S3. The Hadoop Data Roll archiving process to S3 works, and the archived index is created in S3. However, when I try to search that archived index located in S3, I get the error below (which is from search.log). Has anyone run into this problem and know of a solution?

04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  SearchOutputStream - java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  java.lang.Exception: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1569)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  Caused by: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.force(ConfU.java:39)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createV3(PathResolver.java:229)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1553)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	... 3 more
04-16-2021 08:59:02.038 INFO  ERP.s3_provider -  SplunkMR - finishing, version=6.2 ...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - Stopping s3a-file-system metrics system...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system stopped.
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system shutdown complete.
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsConfig - Loaded properties from hadoop-metrics2.properties
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system started
04-16-2021 08:59:02.092 ERROR ERP.s3_provider -   Error while invoking command: /opt/hadoop/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255
04-16-2021 08:59:02.092 INFO  ERPSearchResultCollector - ERP peer=s3_provider is done reading search results.