Splunk shuttl and Hdfs on different machines

[nawneel]

Hello
I have couple of issues regarding Shuttl with HDFS archiving . situation is as follows.

i have a CDH3 cluster and on another machine i have my splunk indexer where i have put shuttl app.
I have also copied core and guava-r09-jarjar.jar to $SPLUNK_HOME/etc/apps/shuttl/lib as per requirement.

first thing arises is , is this a correct architecture for Splunk shuttl deployment.
secondaly , while configuring xmls (archiver.xml,server.xml,splunk.xml) which configuration file should i use to point my CDH3 hosts, i.e how will my Splunk/Shuttl will know where to archive my Splunk data .

Thanks in Advance

[Petter_Eriksson]

If you're using 0.7.x+ with the new configuration, then you should use shuttl/conf/backend/hdfs.properties to point to your NameNode of your Hadoop setup/cluster. The NameNode and Shuttl will co-ordinate where the files will go.

Also, when you're using CDH3, you might have to replace the hadoop.jar as well? I'm not sure about this, but if you're having troubles, that might be the problem.

The latest version of Shuttl is 0.8.2 as of writing this message. Highly recommend using it.

[Petter_Eriksson]

The annoying part is that you'll have to kill the Shuttl process every Splunk restart until Splunk has a solution for killing scripted inputs, which hopefully will be sooner rather than later.

[Petter_Eriksson]

The reason why you're getting "BindException: Address already in use" is because Splunk is not killing scripted inputs correctly (Shuttl is a configured to be a scripted input). This is a known issue and you can read more about it here: http://splunk-base.splunk.com/answers/28733/scripted-input-without-a-shell

However, to fix your error you can kill Shuttl and Splunk will restart it:
ps -ef | grep Shuttl #to get the pid of Shuttl
kill <process id of shuttl> #to kill the Shuttl process

It's safe to kill the Shuttl process. Data won't be lost.

• Petter

[nawneel]

I am currently using 0.8.1 shuttl version , and i am also using shuttl/conf/backend/hdfs.properties to point to my NameNode of Hadoop setup/cluster.
i have also copied core jar and guava 0.9 jar to my shuyyl/lib folder

i am not able to see any details on dashboard .
when i see my shutl log i am getting this error

ERROR ShuttlServer: Error during startup java.net.BindException: Address already in use