I understand that splunk can monitor application very well. Can it (natively or through an add-on) also restart the service if it's found as failed? If there's an add-on, what add-on is it? Lastly, how well has a solution worked in your environment?
So, for example, I have ypbind (NIS client) running and it stops functioning (i.e. the process dies and is no longer in the process table) or hangs, for whatever reason. I would still want a notification that it died or was having an issue, but in order to increase uptime, I'd still want the application to be restarted automatically at least n times. Can splunk do this? I'm guessing the forwarder is only an agent for the purpose of forwarding log information to an aggregator, so I'd have to use some kind of add-on. I'd like to see if having this functionality is feasible in Splunk, and if so, how well it works.
Yes, you can restart your application as long as it is connected with Splunk. You can create an alert and add the custom script. Once your alert is triggered with the help of your script it will pass a command to application for restart.