Alert for Linux server

[Vishal2]

Can you provide the An example of search query or script. If Linux server is shutdown or down or up. I am looking for the best way to setup an shutdown or down or up status alert for Linux server.

[gcusello]

Hi @Vishal2,

my hint is to monitor the Splunk Universal Forwarder that takes logs from that server so, when you don't receive Splunk internal logs from that server, you have to intervene because the server is down or there's a problem one the Forwarder or the network that requires yout intervene.

You can monitor a list of hosts (not depending on the operative system) creating a lookup (called e.g. perimeter.csv) contaning only one column (called host) and run a search like the following:

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

Then you can schedule this search as an alert to run e.g. every five minutes.

Youcan also save this search (without the last row) to display a situation of your infrastructure (total=0 means down, total>0 means up).

Ciao.

Giuseppe

[Vishal2]

Hi,

I know that, you are talking about the logs monitoring but if universal forwarer is failed that time logs not coming to splunk, I don't need that, I need server shutdown or down related query 

Is it possible to write the query for Linux server down or shutdown or up ??? If it is possible what is that query ?? 

[gcusello]

Hi @Vishal2,

the message to search when a server is shutting down is "The system is going down", so you can run something like this:

index=os "The system is going down"

if you have results there was a shutdown, but this doesn't say if the server was still down or not.

In addition , if you don't have the message, it's possible that there was a crash or the Forwarder is down.

For this reason I hint to check the Splunk internal logs.

Ciao.

Giuseppe