Shuttl: coldToFrozenScript Exception in thread "main" - Unkown index with bucket path

[lmyrefelt]

Environment: 4x indexers 3 search-heads 1 job-server, red hat linux, splunk 5.0.3 / 5.0.7 .
Trying to get Shuttl running in our environment using local storage ...

We have configured shuttl to be "running" on our jobserver, that is storage mounted on job-server, shuttl configuration pointing to jobb-server etc etc as described at github.

However we are getting the following error from the indexers when trying to role data from cold to frozen.

10-29-2014 13:27:28.118 +0100 INFO BucketMover - will attempt to freeze: candidate='/opt/cold02/db_splunk/testindex/colddb/db_1336690292_970437600_2' because frozenTimePeriodInSecs=120 exceeds difference between now=1414585648 and latest=1336690292
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript Exception in thread "main" com.splunk.shuttl.archiver.copy.IndexScanner$UnknownIndexPathException: Unknown index with bucket path: /opt/cold02/db_splunk/testindex/colddb/db_1414489876_1336745601_0
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.IndexScanner.getIndex(IndexScanner.java:41)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.IndexScanner.getIndexNameByBucketPath(IndexScanner.java:46)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.EntryPointUtil.getIndexNameForBucketDir(EntryPointUtil.java:35)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.runMainWithDependencies(BucketFreezer.java:118)
10-29-2014 13:28:14.305 +0100 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.main(BucketFreezer.java:106)
10-29-2014 13:28:14.319 +0100 ERROR BucketMover - coldToFrozenScript cmd='/opt/hot02/splunk/etc/apps/shuttl/bin/coldToFrozenScript.sh /opt/cold02/db_splunk/testindex/colddb/db_1414489876_1336745601_0' exited with non-zero status='exited with code 1'*

IF i am trying to use the testArchivingBucket.sh script i am getting the following error;

If you're having trouble, you can customize the shuttl vars with usage:
/opt/hot02/splunk/etc/apps/shuttl/bin/testArchivingBucket.sh [shuttl_port] [shuttl_host]

Shuttling bucket with:
index=TestIndexForTryingOutShuttlArchiving
shuttl_host=svvploggjs01
shuttl_port=9090

HTTP ERROR 500

Problem accessing /shuttl/rest/archiver/bucket/archive. Reason:

    java.io.FileNotFoundException: Could not find directory: /tmp/db_1336330530_1336330530_0.6Ia4Rn

Caused by:

java.lang.RuntimeException: java.io.FileNotFoundException: Could not find directory: /tmp/db_1336330530_1336330530_0.6Ia4Rn

        at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:73)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:167)

        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)

        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)

        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:594)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:485)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:412)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)

        at org.eclipse.jetty.server.Server.handle(Server.java:351)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:451)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:931)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.io.FileNotFoundException: Could not find directory: /tmp/db_1336330530_1336330530_0.6Ia4Rn

        at com.splunk.shuttl.archiver.model.LocalBucket.verifyDirectory(LocalBucket.java:57)

        at com.splunk.shuttl.archiver.model.LocalBucket.<init>(LocalBucket.java:45)

        at com.splunk.shuttl.archiver.model.LocalBucket.<init>(LocalBucket.java:34)

        at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.isPathReplicatedBucketWithRawdataOnly(ArchiveBucketEndpoint.java:79)

        at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:57)

        ... 41 more

We are setting $SPLUNK_HOME as well as a few other variables for use with database / volum definisions inside indexes.conf .

And we are deploying are index ( indexes.conf etc etc ) using a centraly management deployment server.

Both the buckets inside the testindex are existent and valid, and using the testArchiveScript the "tmp" buckets gets created at the "client" side. (not in archive @ jobserver)

Any advice / experiences to share?

[bkondakindi]

check with path for particular index path on index.conf as well Hot/Cold/frozen path .
might be some duplicate index.conf files on indexer server side too.