Since then we added more feature to the code and create an splunk addon which run as splunk app. Setting permission on the meta to only allow admin for visible and RW. Now we can do everything from stop/start/restart deployment server or search head cluster via the splunk UI. We can also create new splunk apps and push to any environment and apply cluster reload from the same UI. With this splunk admin no longer need to logon to the unix server to do splunk cli. High level - We took one of the searchhead convert it as splunk admin that will do all administrator task. Like push git to all deployment server and reload via the UI. We also create a pre-can splunk app template, with all the necessity for barebone app to run. We have a python script that will modify local.meta, app.conf, a dashboard focus on the team data. We create a dashboard call " create new app " that call to the custom script with a set of parameter from dashboard textbox. Similar to the splunk app creation, only different is that we added option to push the new app to test, dev, prod searchhead cluster. Sorry for any confusion as I am trying to keep it at high level, please PM me if you need more clarity. ... View more

If you have git, you can do is create a simple python that do git pull the app into shcluster, and also do an apply after the pull. There are git module out there you can use. Here is a quick simple one that I wrote up awhile back to do the cluster apply. Something like this --- ... def dirsWalkSHCluster(): dirwalk = os.listdir( shcluster_Path ) #print dirwalk print "Updating Search Head Cluster apps" for file in dirwalk: if not file.startswith('.'): if not file.startswith('README'): if not file.startswith('splunk_admin_app'): if not file.startswith('sentinel.txt'): joinpath = os.path.join(shcluster_Path, file) os.chdir( joinpath ) ## Git Process git_process = subprocess.Popen(["git", "pull", "origin", "master"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) processOutput = git_process.communicate()[0] print "datetime="+now + " " + " " + "gitrepo="+file + " " + "message="+processOutput ## Apply Search Head Cluster changes splunkCommand = ("sudo /opt/splunk/bin/./splunk apply shcluster-bundle --answer-yes -target https://%s:8089 -auth git_deploy:git_deploy_pw") %(searchhead) splunkOutput = subprocess.Popen(splunkCommand, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True) msgOut = splunkOutput.communicate()[0] print "datetime="+now + " " + "command_run=Reload SHCluster" + " " + " " + "message=" +msgOut run the script as splunk user, in sudoers file only allow splunk to run splunk command and you can specific with command. You can probably obscure the pw too. ... View more