Splunk Search

How to create search that pulls the server status of running vs. shutdown?

Uday
Explorer

Can you please help me with a search to display a list of servers with a status Running or Shutdown?

I have a list of hostnames, but I am not sure how to show if the server status is Running or Shutdown. Eventually I have to build a dashboard out of it.

Labels (1)
Tags (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @Uday,

as I said, you use the Splunk internal logs that are always present.

Using | metasearch you have a very quick search, so you don't need to use only one event.

You can run this search as an alert every e.g. 5 minutes on a timeframe of 5 minutes having a very quick answer.

In this way you almost immediately have an alert when a server is down.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Uday,

I can image that you have the Splunk UNiversal Forwarder installed in each of the monitored servers, so you can monitor the internal Splunk logs to know if the server is up or down.

You have to put the servers to monitor list in a lookup (called e.g. perimeter) with at least one field contaning hostnames (called host).

With these premises, you can run something like this:

 

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| eval status=if(total=0,"Down","Up")
| table host status

 

You can display the results in a dashboard, that you can also have in graphic mode following the example "Table Icon Set (Rangemap)" that you can find in the Splunk Dashboard Examples app (https://splunkbase.splunk.com/app/1603/).

Or simply in an alert deleting the last two rows and adding the condition | where total=0

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

Ciao.

Giuseppe

0 Karma

Uday
Explorer

Can you please help me to plug in the rangemap query to the below ?

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| eval status=if(total=0,"Down","Up")
| table host status

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Uday,

please, try this:

 

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| rangemap field=total severe=0-0 low=1-1000000000 default=severe
| eval status=if(total=0,"Down","Up")
| table host status range

 

In the table you can eventually choose to display only host and range.

You can also display results in graphic mode, following the instruction in"Table Icon Set (Rangemap)" dashboard in the "Splunk Dashboard Examples" App (https://splunkbase.splunk.com/app/1603/).

Ciao.

Giuseppe

0 Karma

Uday
Explorer

I just tried this and the query helped. Can you please help with the query using range map to display the status Up/Down on the dashboard?

 

0 Karma

Uday
Explorer

Thank you. Is there a way we can do it based on the last event in the log?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Uday,

as I said, you use the Splunk internal logs that are always present.

Using | metasearch you have a very quick search, so you don't need to use only one event.

You can run this search as an alert every e.g. 5 minutes on a timeframe of 5 minutes having a very quick answer.

In this way you almost immediately have an alert when a server is down.

Ciao.

Giuseppe

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Do you have events in Splunk that show the status of the servers?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Uday
Explorer

No, unfortunately the logs does n't have any thing that indicates what time the server started or shutdown.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.