All Apps and Add-ons

Shuttl: coldToFrozenScript Exception in thread "main" - Unkown index with bucket path

lmyrefelt
Builder

Environment: 4x indexers 3 search-heads 1 job-server, red hat linux, splunk 5.0.3 / 5.0.7 .
Trying to get Shuttl running in our environment using local storage ...

We have configured shuttl to be "running" on our jobserver, that is storage mounted on job-server, shuttl configuration pointing to jobb-server etc etc as described at github.

However we are getting the following error from the indexers when trying to role data from cold to frozen.

10-29-2014 13:27:28.118 +0100 INFO BucketMover - will attempt to freeze: candidate='/opt/cold02/db_splunk/testindex/colddb/db_1336690292_970437600_2' because frozenTimePeriodInSecs=120 exceeds difference between now=1414585648 and latest=1336690292
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript Exception in thread "main" com.splunk.shuttl.archiver.copy.IndexScanner$UnknownIndexPathException: Unknown index with bucket path: /opt/cold02/db_splunk/testindex/colddb/db_1414489876_1336745601_0
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.IndexScanner.getIndex(IndexScanner.java:41)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.IndexScanner.getIndexNameByBucketPath(IndexScanner.java:46)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.EntryPointUtil.getIndexNameForBucketDir(EntryPointUtil.java:35)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.runMainWithDependencies(BucketFreezer.java:118)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.main(BucketFreezer.java:106)
10-29-2014 13:28:14.319 +0100 ERROR BucketMover - coldToFrozenScript cmd='/opt/hot02/splunk/etc/apps/shuttl/bin/coldToFrozenScript.sh /opt/cold02/db_splunk/testindex/colddb/db_1414489876_1336745601_0' exited with non-zero status='exited with code 1'*

IF i am trying to use the testArchivingBucket.sh script i am getting the following error;

If you're having trouble, you can customize the shuttl vars with usage:
/opt/hot02/splunk/etc/apps/shuttl/bin/testArchivingBucket.sh [shuttl_port] [shuttl_host]

Shuttling bucket with:
index=TestIndexForTryingOutShuttlArchiving
shuttl_host=svvploggjs01
shuttl_port=9090

HTTP ERROR 500

Problem accessing /shuttl/rest/archiver/bucket/archive. Reason:

    java.io.FileNotFoundException: Could not find directory: /tmp/db_1336330530_1336330530_0.6Ia4Rn

Caused by:

java.lang.RuntimeException: java.io.FileNotFoundException: Could not find directory: /tmp/db_1336330530_1336330530_0.6Ia4Rn
at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:167)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:594)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:485)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:412)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:351)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:451)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:931)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: Could not find directory: /tmp/db_1336330530_1336330530_0.6Ia4Rn
at com.splunk.shuttl.archiver.model.LocalBucket.verifyDirectory(LocalBucket.java:57)
at com.splunk.shuttl.archiver.model.LocalBucket.<init>(LocalBucket.java:45)
at com.splunk.shuttl.archiver.model.LocalBucket.<init>(LocalBucket.java:34)
at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.isPathReplicatedBucketWithRawdataOnly(ArchiveBucketEndpoint.java:79)
at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:57)
... 41 more

We are setting $SPLUNK_HOME as well as a few other variables for use with database / volum definisions inside indexes.conf .
And we are deploying are index ( indexes.conf etc etc ) using a centraly management deployment server.
Both the buckets inside the testindex are existent and valid, and using the testArchiveScript the "tmp" buckets gets created at the "client" side. (not in archive @ jobserver)

Any advice / experiences to share?

0 Karma

bkondakindi
Path Finder

check with path for particular index path on index.conf as well Hot/Cold/frozen path .
might be some duplicate index.conf files on indexer server side too.

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...