Alerting

How to build an alert that will trigger any server issue and do an automatic restart of the server through splunk alert without human intervention?

Prakash493
Communicator

Hi , I am looking for to automate jobs for splunk , i want to build an alert that will trigger if any server has issue as well as do a automatic restart of the server through the splunk alert without human intervention , How can i do it ?

0 Karma

woodcock
Esteemed Legend

Splunk recently bought Phantom so if you are looking for an add-on solution, I would start there because it will obviously be heavily supported and probably fully integrated at some point.

0 Karma

sduff_splunk
Splunk Employee
Splunk Employee

You can look at creating an alert action script, as documented at https://docs.splunk.com/Documentation/Splunk/7.2.5/AdvancedDev/ModAlertsIntro

You will need to write a Splunk query that identifies the hosts which are not responsive. At the end of the search, you want a list of servers. Save this search as an alert, then assign an alert action to it.

Your alert action script will need to read a CSV file which contains your servers, then decide what you want to do with them. I suggest using something like ssh with pre-shared certificates, then doing ssh root@remote-server.com /sbin/reboot . Just do that for each host that is listed in the CSV.

Splunk also supported very basic alert action scripts, which I summarised at https://simonduff.net/splunk_alert_script/ . This has been deprecated, but still works.

Alternatively, you can also look at Splunk Phantom. That has many more features, running advanced playbooks, etc... which is probably overkill for what you require.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...