Just to address ddrillic’s questions:
If the forwarders are actively phoning home, then the forwarder service is probably running. If you have an entry in your forwarder management app where the forwarder is showing up, but you suspect the service isn’t running, delete the record and allow it to phone home again to verify. Now, I have run into a weird case where the forwarder was running, and phoning home, but was in an errored state and not forwarding logs (this was on Windows), and I had to restart it to get it forward data again.
If the forwarder service is not running, you will not be able to push an app to it.
Forwarder management does have the option to restart a forwarder, but only after a successful installation of an app, not manually. You can either use the GUI in the forwarder management app to check the “Restart Splunkd” or edit your serverclass.conf file with restartSplunkd = true
If a forwarder is down (as in the service is not running), you don’t necessarily have to log into the server to restart it. You could either do it remotely via a management application (like SCCM for Windows or set it up with something like Puppet for Linux), a remote script, or create a scheduled task with a local script to check the status of the service, and restart it if it is down (Windows) or set up a cron job with something like a bash or python script to query the status and restart and/or start it if it is down.
