topic Re: Is there a way to monitor servers through Splunk? in Splunk Search

Is there a way to monitor servers through Splunk?

rahul1502133 — Fri, 06 Dec 2019 07:48:15 GMT

Hey everyone,

I just had a small search, is there any way to monitor servers using Splunk and get data on their availability statistics.
FYI I am using Splunk Cloud,
Pls respond ASAP

Re: Is there a way to monitor servers through Splunk?

gcusello — Fri, 06 Dec 2019 08:16:21 GMT

Hi @rahul1502133,
Splunk is born to do this, and then is also used for many other uses (security, business insight, etc...).

Anyway stop with marketing, you have to take logs from your servers using an agent called Universal Forwarder (it's no cost) that sends logs to Splunk Cloud.
Usually when it's used Splunk Cloud, there are two intermediate Heavy Forwarders to concentrate logs and opening only one port between targets and Splunk Cloud.

To configure Universal Forwarders to take logs are used some dedicated apps, called Technical_Add-ons (TAs), that contains all the inputs to take logs, execute scripts, etc... that you can create custom by yourself or take from apps.splunk.com,

With the few information you share, this is what I can hint, in addition I suggest to read at https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/User/DataSplunkCloudcanindex how to get data in Splunk Cloud and see at apps.splunk.com the list of the sources you can take.

Ciao.
Giuseppe

Re: Is there a way to monitor servers through Splunk?

rahul1502133 — Fri, 06 Dec 2019 09:50:10 GMT

So what u are saying is that Install Universal forwarder on the servers and then index the data in splunk thats fine .....but here my question is that how to find the availability (UP OR DOWN) status of a server based in the indexed data
Coz if we consider a scenario of logs not coming into splunk as benchmark to say that a server is up or down it could only mean that the agent is down and the server might be running.

Re: Is there a way to monitor servers through Splunk?

gcusello — Fri, 06 Dec 2019 12:00:31 GMT

Hi @rahul1502133,
you should creata a lookup (called e.g. perimeter.csv) containing all the servers to monitor (in a column called host) and then run a search like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=upper(host) | fields host count ]
| stats sum(count) AS total BY host

In this way all the servers with total=0 are down and the other are up.
You can create an alert adding at the end of the search | where total=0 triggering when there are down servers.

If instead you want a dashboard, eventually in graphic mode, you can see my answer https://answers.splunk.com/answers/788233/servers-availability-status.html#answer-787047

If you want, you can also enrich your panel, adding to the lookup other information (e.g. IP, role, description, etc...) and adding them to the search, something like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=upper(host) | fields host count IP Role Description ]
| stats values(IP) AS IP values(Role) AS Role values(Description) AS Description sum(count) AS total BY host

Ciao.
Giuseppe

Re: Is there a way to monitor servers through Splunk?

rahul1502133 — Fri, 06 Dec 2019 13:57:32 GMT

HI @gcusello
Thank you for the query ......but the above query only works if the logs are coming, what if the
Splunk agent on the server is down but the server is still up..... so according to the query that server will be concluded as down... could you pls come up with a solution for this scenario.

Re: Is there a way to monitor servers through Splunk?

gcusello — Fri, 06 Dec 2019 14:33:24 GMT

Hi @rahul1502133,
If the Splunk Universal Forwarder is down you cannot monitor your server, so I think that there's a good reason to intervene even if the server is up because without UF you're blind!

If instead for you is acceptable that the UF is down and the server is up, you have to create a different way to check this, e.g. a script on one Splunk server that pings all the servers indexing the results, then the search it's very similar.

I don't hint this solution, because (as I said) without UF you're completely blind!

Ciao.
Giuseppe

Re: Is there a way to monitor servers through Splunk?

jackin — Tue, 09 Nov 2021 04:53:07 GMT

Hi @gcusello

can you provide same for single host

Re: Is there a way to monitor servers through Splunk?

gcusello — Tue, 09 Nov 2021 06:51:11 GMT

Hi @jackin,

for a single host try something like this:

index=your_index host=your_host

if you have events the server is up, if you haven't events the server is down.

Ciao.

Giuseppe

Re: Is there a way to monitor servers through Splunk?

gcusello — Tue, 09 Nov 2021 06:58:38 GMT

HI @rahul1502133,

if the server is up and the Splunk agent is down you're completely blind so it's a good idea to have an alert when this occurs!

Anyway, you could create a script in another server that pings all ther servers in your network and sends the output of the script to Splunk, so you can immediately find if the ping fails, but I don't like this solution because the most important thing isn't to test if the server is up, but if the Splunk agent is up and sending logs.

Ciao.

Giuseppe