Error trying to push app from deployer

[gwobben]

I'm getting this error when I try to push an app to my search head cluster:
Error when getting master uri from target to do a rolling-restart Service Unavailable

I've used the command:
sudo ./splunk apply shcluster-bundle -target https://[SOME SEARCH HEAD]:8089 -auth admin:'[THE PASSWORD]'

It stopped working since we've fixed the IP addresses of the Splunk nodes. Anyone an idea how to fix/debug this?

[jkat54 · Accepted solution]

You need to correct anything wrong with your SHC first.

Run this command from your search head(s) and let us know the output:

 splunk show shcluster-status -auth <username>:<password>

Also it is best practice to create your own splunk user & group, and run splunk as that user.

#If splunk is running as sudo or root, stop it first.
groupadd splunk
useradd splunk -g splunk
chown -Rf splunk. /opt/splunk  #or wherever you have splunk installed

You'll probably have to fix/rebuild the SHC using some of these commands from each search head:

 splunk init shcluster-config -auth <username>:<password> -mgmt_uri <URI>:<management_port> -replication_port <replication_port> -replication_factor <n> -conf_deploy_fetch_url <URL>:<management_port> -secret <security_key>

 splunk restart

http://docs.splunk.com/Documentation/Splunk/6.2.0/DistSearch/ViewSHCstatus
http://docs.splunk.com/Documentation/Splunk/6.2.0/DistSearch/SHCdeploymentoverview

[gwobben]

We've rebuild the SHC (splunk init shcluster-config ... ) and started the election for captain again. Then things started working again. Thanks guys!

[jeffland]

I don't know if this causes your problem, but generally, I'd advise against sudoing splunk. It causes all kind of undesired behavior with folder/file ownership and permissions if some splunk actions are performed as a different user.