Splunk Enterprise

Hadoop Data Roll - Error Searching an S3 Archived Index

SteveE
New Member

I am running Splunk Enterprise 8.0.6 and have Hadoop Data Roll configured, using Hadoop 3.2.1 with Java 1.8.0_282-b08. I have a virtual index configured to archive an index to AWS S3. The Hadoop Data Roll archiving process to S3 works, and the archived index is created in S3. However, when I try to search that archived index located in S3, I get the error below (which is from search.log). Has anyone run into this problem and know of a solution?

 

04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  SearchOutputStream - java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  java.lang.Exception: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1569)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  Caused by: java.lang.RuntimeException: Configuration was not set. stacktrace=[com.splunk.roll.util.ConfU.force(ConfU.java:38), com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56), com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48), com.splunk.roll.PathResolver.createV3(PathResolver.java:229), com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211), com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644), com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62), com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34), com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809), com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089), com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906), com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804), com.splunk.mr.SplunkMR.run(SplunkMR.java:1553), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76), org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90), com.splunk.mr.SplunkMR.main(SplunkMR.java:1841)]
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.force(ConfU.java:39)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRemoteHome(ConfU.java:56)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.util.ConfU.getRollRoot(ConfU.java:48)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createV3(PathResolver.java:229)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.createWithVersion(PathResolver.java:211)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.roll.PathResolver.resolveBuckets(PathResolver.java:152)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1644)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VirtualIndex.generateSplits(VirtualIndex.java:1609)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:62)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.input.VixSplitGenerator.generateSplits(VixSplitGenerator.java:34)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.streamData(SplunkMR.java:809)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.executeImpl(SplunkMR.java:1089)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR$SearchHandler.execute(SplunkMR.java:906)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.runImpl(SplunkMR.java:1804)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	at com.splunk.mr.SplunkMR.run(SplunkMR.java:1553)
04-16-2021 08:59:01.938 ERROR ERP.s3_provider -  	... 3 more
04-16-2021 08:59:02.038 INFO  ERP.s3_provider -  SplunkMR - finishing, version=6.2 ...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - Stopping s3a-file-system metrics system...
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system stopped.
04-16-2021 08:59:02.044 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system shutdown complete.
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsConfig - Loaded properties from hadoop-metrics2.properties
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
04-16-2021 08:59:02.069 INFO  ERP.s3_provider -  MetricsSystemImpl - s3a-file-system metrics system started
04-16-2021 08:59:02.092 ERROR ERP.s3_provider -   Error while invoking command: /opt/hadoop/bin/hadoop com.splunk.mr.SplunkMR - Return code: 255
04-16-2021 08:59:02.092 INFO  ERPSearchResultCollector - ERP peer=s3_provider is done reading search results.

 

 

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...