I have an app server running a custom application that is, unfortunately, a bit buggy. This bug causes it's service to spike in CPU usage and degrade performance. There's a fix in the works but because I can manually resolve it by restarting the service it is lower on the priority list.
I currently use Splunk to send me an alert when CPU usage gets to 80% or more - this lets me get in there to do the reset before performance degrades.
It looks like Splunk used to have a simple feature to run a script on the UF's /bin/ directory, which would have made this pretty simple - but it is deprecated and I assume doesn't work at all. Now, however, we're supposed to create a custom alert action to reinvent this alert action.
Following the basic directions here, I've come to find I don'thave the ability to create a new Alert Action: Create alert actions - Splunk Documentation
I can "Browse More" and view the existing ones, but there's no ability to create anything new. Is there some sort of pre-requisite before these can be done? It does not appear to be mentioned in this documentation if that's the case.
Alternatively, does Splunk still trigger scripts even though the feature is deprecated? The above needs learned but seems like a lot of overhead to have one specific server run net stop [service] && net start [service].
