Not saying i disagree with you but would you care to explain how pushing arbitrary code for execution from a deployment server to any endpoint being run with potentially local system/root is saner and safer then using the splunk api remotely from the same box?
We use a Splunk_Restart class with a dummy Splunk_Restart app that has "Restart Splunkd" enabled. When we want to restart a host we just add it to the clients list and remove it 5 minutes later.
This is only necessary if you're making changes to an already deployed .conf file. If you're making major changes, you can use ./splunk reload deploy-server -class
if they are talking to deployment server, you can pick an app that is deployed to all forwarders, probably an outputs app, and add the restart flag to it. then reload your apps to the forwarders
go to "forwarder Mangement" ->apps tab -> choose and app -> edit -> mark :Restart Splunkd" -> go to $SPLUNK_HOME/bin on Deploymet Server -> run:
splunk reload deploy-server
hope it helps
thanks! this seems obvious but is not. fixed my problems not getting any data from my deployment clients even though the apps were pushed out successfully.
However I had to remove all my apps, reload deploy-server, and then add them again and reload in order for the restart to actually take place. I think the restart is only triggered during an initial app installation and not just when you update the app. Not positive about that though.
If the splunk service is down on the forwarder, then it cannot reach the deployment server, hence cannot receive the app update and restart request.
You may have to actually start splunk service directly on the client (with a CLI command or any remote admin management tool)