All Apps and Add-ons

How can I configure shuttl to work in a cluster Splunk?

Explorer

I have a cluster with Splunk 5.2, consisting of 1 master node, 5 indexer node and 1 search head and I'm trying to get shuttl to archive frozen bucket, avoiding, in this way, buckets duplication problem. I have not decided yet which backend to use, it could be an attached storage or amazon glacier.

I red the article http://blogs.splunk.com/2013/03/21/parallel-data-transfer-in-shuttl-0-8-0/ and it seems that installing shuttl on search head you can orchestrate the archiviation in the cluster but I cannot find instructions on how to set this up.

Firstly, I'm unclear about on which nodes have to install shuttl, both search head and indexers?

Secondly, I'm unclear about how setting up shuttl configuration files (archiver.xml/splunk.xml/server.xml) to identify the cluster.

Thirdly, indexes.conf file, distribuited by master node to indexers, must be configured to call the shuttl archiver script with "coldToFrozenScript = $SPLUNK_HOME/etc/apps/shuttl/bin/coldToFrozenScript.sh" for each index that must be archived, as indicated in shuttl documentation? In this case should be shuttl installed also on indexers, but how and who recognize duplicate buckets?

Thanks,
clagese

Tags (3)
0 Karma

Explorer

Thank you for the response. Unfortunately I have some problems.

I configured my cluster to archive data on s3 with shuttl. I installed Shuttl on all nodes. My data are really archived on s3 bucket but there is a problem with archiviation path.
Any path values I set as "archivePath" in archiver.xml( /myrootpath or myrootpath or or /), Shuttl store my data in a root path "//" within my s3 bucket.
The result is like a directory without name, in the root of s3 bucket, that contains the directories structure created by shuttle.
For example if I set /myrootpath in archiver.xml, I find on s3 splunk buckets of type "s3://mys3bucket//myrootpath/archivedata/mycluster/splunkIndex01/mytestdb/db1397659802139765544819B95021DE-89AB-4A9D-B924-575736C54B81"

In the shuttl log I found these:


2014-04-16 18:56:22,037 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="mytestdb" path="/opt/splunk/var/lib/splunk/mytestdb/colddb/rb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81"
2014-04-16 18:56:22,744 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="mytestdb" path="/opt/splunk/var/lib/splunk/mytestdb/db/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82"
2014-04-16 18:56:25,643 INFO com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[LocalBucket [getDirectory()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=SPLUNKBUCKET, getPath()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912], LocalBucket [getDirectory()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/rb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81, getName()=rb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81, getIndex()=mytestdb, getFormat()=SPLUNKBUCKET, getPath()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/rb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81, getEarliest()=Wed Apr 16 17:27:39 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=16379]]"
2014-04-16 18:56:29,649 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=SPLUNKBUCKET, getPath()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912]"
2014-04-16 18:56:30,700 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive
data%2Fmycluster%2FsplunkIndex02%2Fmytestdb%2Fdb1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:30,701 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive
data%2Fmycluster%2FsplunkIndex02%2Fmytestdb%2Fdb1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message
2014-04-16 18:56:31,089 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl
archiver/data/format-export-dir/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNKBUCKET/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttlarchiver/data/format-export-dir/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNKBUCKET/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912]" destination="/myrootpath/archivedata/mycluster/splunkIndex02/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82/CSV"
2014-04-16 18:56:31,095 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/opt/splunk/shuttl
archiver/data/format-export-dir/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNKBUCKET/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttlarchiver/data/format-export-dir/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNKBUCKET/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912], remoteTemp=/myrootpath/temporarydata/splunkIndex02/myrootpath/archivedata/mycluster/splunkIndex02/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82/CSV, dst=/myrootpath/archivedata/mycluster/splunkIndex02/mytestdb/db1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82/CSV]"
2014-04-16 18:56:31,230 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchivedata%2Fmycluster%2FsplunkIndex02%2Fmytestdb%2Fdb1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,230 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchivedata%2Fmycluster%2FsplunkIndex02%2Fmytestdb%2Fdb1397667252139766176022B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message
2014-04-16 18:56:31,608 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/rb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81, getName()=db1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81, getIndex()=mytestdb, getFormat()=SPLUNKBUCKET, getPath()=/opt/splunk/shuttlarchiver/data/safe-buckets/mytestdb/rb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81, getEarliest()=Wed Apr 16 17:27:39 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=16379]"
2014-04-16 18:56:31,741 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,741 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath' - Received error response with XML message
2014-04-16 18:56:31,988 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive
data%2Fmycluster%2FsplunkIndex01%2Fmytestdb%2Fdb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,989 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive
data%2Fmycluster%2FsplunkIndex01%2Fmytestdb%2Fdb1397667252139766205921B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Received error response with XML message
2014-04-16 18:56:32,024 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Ftemporary
data' - Unexpected response code 404, expected 200
2014-04-16 18:56:32,024 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Ftemporary_data' - Received error response with XML message

In spite of this problem the Thaw and flush process of splunk buckets from search head interface works well.

I hope you can give me an answer as soon as possible.
Thanks in advance

0 Karma

Splunk Employee
Splunk Employee

Hi Clagese,

  1. You should install Shuttl on all nodes, indexers and search heads. This enables your search heads to thaw data back to their original indexers. Archivation is done automatically on the indexers when needed.
  2. How to setup the configuration files is made more clear on the github page (github.com/splunk/splunk-shuttl), but to put it short:
  3. archiver.xml is for choosing backend storage and how things are stored.
  4. splunk.xml is the configuration to your local Splunk instance. The indexer or search head that it's installed on.
  5. server.xml is for configuring the Shuttl server that's running on the local Splunk instance. There's even more configuration in conf/backend that you should take a look at too.
  6. Shuttl will recognize duplication of buckets when transferring them. Shuttl will dedup the bucket replication for you.

Thanks,
- Petter
Shuttl Dev

0 Karma