Splunk Search

How to create search that pulls the server status of running vs. shutdown?

Uday
Explorer

Can you please help me with a search to display a list of servers with a status Running or Shutdown?

I have a list of hostnames, but I am not sure how to show if the server status is Running or Shutdown. Eventually I have to build a dashboard out of it.

Labels (1)
Tags (3)
0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @Uday,

as I said, you use the Splunk internal logs that are always present.

Using | metasearch you have a very quick search, so you don't need to use only one event.

You can run this search as an alert every e.g. 5 minutes on a timeframe of 5 minutes having a very quick answer.

In this way you almost immediately have an alert when a server is down.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
Esteemed Legend

Hi @Uday,

I can image that you have the Splunk UNiversal Forwarder installed in each of the monitored servers, so you can monitor the internal Splunk logs to know if the server is up or down.

You have to put the servers to monitor list in a lookup (called e.g. perimeter) with at least one field contaning hostnames (called host).

With these premises, you can run something like this:

 

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| eval status=if(total=0,"Down","Up")
| table host status

 

You can display the results in a dashboard, that you can also have in graphic mode following the example "Table Icon Set (Rangemap)" that you can find in the Splunk Dashboard Examples app (https://splunkbase.splunk.com/app/1603/).

Or simply in an alert deleting the last two rows and adding the condition | where total=0

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

Ciao.

Giuseppe

0 Karma

Uday
Explorer

Can you please help me to plug in the rangemap query to the below ?

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| eval status=if(total=0,"Down","Up")
| table host status

 

0 Karma

gcusello
Esteemed Legend

Hi @Uday,

please, try this:

 

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| rangemap field=total severe=0-0 low=1-1000000000 default=severe
| eval status=if(total=0,"Down","Up")
| table host status range

 

In the table you can eventually choose to display only host and range.

You can also display results in graphic mode, following the instruction in"Table Icon Set (Rangemap)" dashboard in the "Splunk Dashboard Examples" App (https://splunkbase.splunk.com/app/1603/).

Ciao.

Giuseppe

0 Karma

Uday
Explorer

I just tried this and the query helped. Can you please help with the query using range map to display the status Up/Down on the dashboard?

 

0 Karma

Uday
Explorer

Thank you. Is there a way we can do it based on the last event in the log?

0 Karma

gcusello
Esteemed Legend

Hi @Uday,

as I said, you use the Splunk internal logs that are always present.

Using | metasearch you have a very quick search, so you don't need to use only one event.

You can run this search as an alert every e.g. 5 minutes on a timeframe of 5 minutes having a very quick answer.

In this way you almost immediately have an alert when a server is down.

Ciao.

Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Do you have events in Splunk that show the status of the servers?

---
If this reply helps you, Karma would be appreciated.
0 Karma

Uday
Explorer

No, unfortunately the logs does n't have any thing that indicates what time the server started or shutdown.

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...