How to build an alert that will trigger any server issue and do an automatic restart of the server through splunk alert without human intervention?

[Prakash493]

Hi , I am looking for to automate jobs for splunk , i want to build an alert that will trigger if any server has issue as well as do a automatic restart of the server through the splunk alert without human intervention , How can i do it ?

[woodcock]

Splunk recently bought Phantom so if you are looking for an add-on solution, I would start there because it will obviously be heavily supported and probably fully integrated at some point.

[sduff_splunk]

You can look at creating an alert action script, as documented at https://docs.splunk.com/Documentation/Splunk/7.2.5/AdvancedDev/ModAlertsIntro

You will need to write a Splunk query that identifies the hosts which are not responsive. At the end of the search, you want a list of servers. Save this search as an alert, then assign an alert action to it.

Your alert action script will need to read a CSV file which contains your servers, then decide what you want to do with them. I suggest using something like ssh with pre-shared certificates, then doing ssh root@remote-server.com /sbin/reboot . Just do that for each host that is listed in the CSV.

Splunk also supported very basic alert action scripts, which I summarised at https://simonduff.net/splunk_alert_script/ . This has been deprecated, but still works.

Alternatively, you can also look at Splunk Phantom. That has many more features, running advanced playbooks, etc... which is probably overkill for what you require.