Can we run scripts on forwarder from an alert action configured on splunk

[rajeev_ku]

Hi,
I have a scenario to configure alert actions with following requirements.
1- Forwarders will send logs to Splunk indexers.
2 - Alert actions will be configured to execute scripts on a search.
3 - That script should run on the log originated forwarder to perform certain actions like to restart service etc.

if we can done above requirement, then how? please brief.

Thanks
Rajeev

[skazako]

Did you manage to make it work at the end? Couldn't find any working example...

[cpetterborg]

Is it possible? Yes.
Is it recommended? Probably not, depending on what you want to do.

The biggest problem is going to be permissions. You can just have the search head tell the forwarder to run something on demand. It doesn't really work that way. If you are running forwarders as root/admin (not recommended in most situations), there is permission that could be used to restart a process/service. But likely this is not going to help that much either. You would probably have to have the search head touch a file or something like that on the forwarder that the forwarder has been set up to check and if the conditions are met, it could restart. Not the best method, but possible.

If you aren't running as root/admin, then the possibility of running a restart script (let's use linux as the example here). You would have to have the search head able to run remotely as root to the forwarder machine to run the commands to restart the service, or like the above situation, change some file on the forwarder machine so that a cron job could run and understand that it is supposed to restart the process. both are cumbersome, but possible. The latter is more easily done, but may be less likely to work in a pinch situation, as it depends on a process that checks a condition on a schedule and you hope that it gets it right.

Does either of these scenarios have the potential of working in your case?

The other thing to watch out for is the case where you don't want it to restart the process (say you are doing maintenance on the machine) and it restarts the process which causes problems on the server. That's probably the biggest reason against using a process to automatically restart a process. Still people do it.

[rajeev_ku]

Thanks,
I wanted to configure kind of self healing for any service/process down, if splunk detect that service is down, wanted to configure splunk alert to run script on remote host to start the service.
or space utilization is above 90% and i know which file is causing the problem and wanted to run force log rotate on the remote server.

is this recommended?
how i can achieve this.
I am new to this kind of requirement.

Thanks in advance.
Rajeev

[cpetterborg]

Any kind of self-healing is tricky. It has to be smart, or it will bite you in the butt. I've been doing monitoring for 11 years now, and there are lots of things to watch out for. It is very dependent on your situation, but you can get into race conditions and conflicting needs. Like I said before, it isn't recommended, but lots of people do it. If you have an operations center that is manned 24x7, then it is better to alert the operations center and have a person decide if it is the right thing to do according to some knowledge-base/run-book/etc.