topic Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ? in Alerting

What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Mon, 19 Jul 2021 04:08:47 GMT

Can you provide an example of a search query or script I can use to tell if a windows server is shutdown or down.i am looking for the best way to set up an shutdown or down status alert for windows server.

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

venkatasri — Mon, 19 Jul 2021 04:26:19 GMT

Hi @Vishal2

May be this would work based on Windows EventCodes description. Assuming you have Windows add-on running and indexing the WinEventLogs from these windows servers that you want to find when they shutdown.

index=<your_index> source=WinEventLog* EventCode=41 OR EventCode=1074 OR EventCode=6006 OR EventCode=6008 | stats count by host | where count > 1

Event ID	Description
41	The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.
1074	Logged when an app (ex: Windows Update) causes the system to restart, or when a user initiates a restart or shutdown.
6006	Logged as a clean shutdown. It gives the message "The Event log service was stopped".
6008	Logged as a dirty shutdown. It gives the message "The previous system shutdown at time on date was unexpected".

---

An upvote would be appreciated and Accept solution if this reply helps!

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

gcusello — Mon, 19 Jul 2021 07:09:17 GMT

Hi @Vishal2,

the best approach is to create a lookup (called e.g. perimeter.csv) containing all the hosts to monitor.

Then you could run a simple search like this:

using this search (without the last row) you can also create a dashboard displaying the status of all the monitored hosts:

Ciao.

Giuseppe

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Mon, 19 Jul 2021 07:51:09 GMT

Is it used for linux servers ??

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

gcusello — Mon, 19 Jul 2021 08:00:01 GMT

Hi @Vishal2,

it's not relevant the kind of server because it uses the Splunk Forwarder's logs.

In this way you're sure that if the server is up and the Forwarder is running you have logs.

Ciao.

Giuseppe

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Mon, 19 Jul 2021 08:10:27 GMT

Thanks for reply

Please can you provide exact querys for alert creation if windows and Linux servers shutdown.

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

gcusello — Mon, 19 Jul 2021 08:21:11 GMT

Hi @Vishal2,

as I said, it isn't relevant if the Operative System is Windows or Linux, it's only relevant the list of hosts to monitor that you put in the perimeter.csv lookup.

If you put only windows servers, you'll monitor only windows servers!

if you like (but I think that's unuseful) you can create two perimeters files (called e.g. win_perimeter.csv and x_perimeter.csv to separately monitor winsows and Linux server and create two different alerts, one for windows and one for Linux, but I don't like this.

If you like you can also insert in the perimeter.csv file the information about the Operative System (so you'll have in this lookup two fields "host", "os") and display it in the alert search:

Ciao.

Giuseppe

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Tue, 20 Jul 2021 08:47:44 GMT

if I have only one Linux host I'd then what is the search query for shutdown or down or up to alert creation

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

gcusello — Tue, 20 Jul 2021 09:10:28 GMT

i @Vishal2,

as I said, the operative systems isn't relevant, so if you have to monitor many windows servers and one Linux server, you can add also a Linux hostname in the perimeter.csv lookup containing all the windows servers.

If instead you have only one server to monitor, you can use a simpler search:

index=_internal host=my_hostname

My hint is to build a complete control using my previous answer and the perimeter.csv, so, when you'll have more servers to monitor, you're already ready.

Ciao.

Giuseppe

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Tue, 20 Jul 2021 10:05:13 GMT

Hi,

Can you post that alert settings...?

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

gcusello — Tue, 20 Jul 2021 10:18:51 GMT

Hi @Vishal2,

settings I use in this alert (that I configure in every installation I do!) are:

search is the one in my previous answers,
Alert Type: Scheduled
Time Ranhe depends on your requirements, I usually use 5 minutes (300 seconds).
Frequency, depends on the time frafe and it's every 5 minutes, I usually use this cron expression:
- */5 * * * *
Expires: 24 hours
Trigger Alert when: Number of results=0
Trigger: Once
Throttle: depends on your reaction time, e.g. 1 hour
Add Actions:
- Add to Triggered Alerts with High Severity
- email or a script that opens a case on your troubleticketing system

Ciao.

Giuseppe

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Tue, 20 Jul 2021 15:14:50 GMT

Hi,

you are talking about the logs monitoring but if universal forwarer is failed that time logs not coming to splunk, I don't need that, I need server shutdown or down related query

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

gcusello — Tue, 20 Jul 2021 15:31:23 GMT

Hi @Vishal2,

if Forwarder isn't sending logs, you cannot monitor your host, so it's better to monitor Forwarder: if you have an alert you have two choices:

forwarder down, and you must interviene otherwise you're blind,
host down, and you must intervene.

in both cases you must intervene!

If you want, but I don't hint this: instead of the index=_internal, you can use index=* but it's the same thing.

Ciao.

Giuseppe

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Tue, 27 Jul 2021 05:46:14 GMT

Hi @venkatasri

Can you please post the query for Linux server as well ....

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

venkatasri — Tue, 27 Jul 2021 05:54:46 GMT

@Vishal2 As @gcusello answered without forwarder installed on Linux it's not possible. You have the solution already. As your original query was already answered regarding windows you could Accept the solution and open a new post for someone to answer related to linux.

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Tue, 27 Jul 2021 06:09:29 GMT

Hi @venkatasri

Windows query is under testing once it's successful I accept the solution & coming to Linux, Forwader is already installed and data is reporting to splunk.

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

venkatasri — Tue, 27 Jul 2021 06:38:38 GMT

@Vishal2 You have the solution from @gcusello Already for Linux/Windows which is a lookup based. Hope that helps!

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

Vishal2 — Wed, 28 Jul 2021 10:39:12 GMT

Hi @venkatasri, @gcusello

By using event codes it's working but can please post by using lookup containing the all the hosts to monitor.

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

I-C-U — Thu, 30 Nov 2023 08:54:31 GMT

Power Shell for linux?

cat filename | grep or awk?

Re: What's the best way to create an alert to tell whether a windows server is shutdown or down ?

gcusello — Thu, 30 Nov 2023 09:12:46 GMT

Hi @Vishal2 ,

you already have the solution: you have to create a lookup containing the list of monitored hosts and run the above search, what's your doubt?

Ciao.

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉