Splunk Search

Is there a way to monitor servers through Splunk?

rahul1502133
Explorer

Hey everyone,

I just had a small search, is there any way to monitor servers using Splunk and get data on their availability statistics.
FYI I am using Splunk Cloud,
Pls respond ASAP

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @rahul1502133,
Splunk is born to do this, and then is also used for many other uses (security, business insight, etc...).

Anyway stop with marketing, you have to take logs from your servers using an agent called Universal Forwarder (it's no cost) that sends logs to Splunk Cloud.
Usually when it's used Splunk Cloud, there are two intermediate Heavy Forwarders to concentrate logs and opening only one port between targets and Splunk Cloud.

To configure Universal Forwarders to take logs are used some dedicated apps, called Technical_Add-ons (TAs), that contains all the inputs to take logs, execute scripts, etc... that you can create custom by yourself or take from apps.splunk.com,

With the few information you share, this is what I can hint, in addition I suggest to read at https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/User/DataSplunkCloudcanindex how to get data in Splunk Cloud and see at apps.splunk.com the list of the sources you can take.

Ciao.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rahul1502133,
Splunk is born to do this, and then is also used for many other uses (security, business insight, etc...).

Anyway stop with marketing, you have to take logs from your servers using an agent called Universal Forwarder (it's no cost) that sends logs to Splunk Cloud.
Usually when it's used Splunk Cloud, there are two intermediate Heavy Forwarders to concentrate logs and opening only one port between targets and Splunk Cloud.

To configure Universal Forwarders to take logs are used some dedicated apps, called Technical_Add-ons (TAs), that contains all the inputs to take logs, execute scripts, etc... that you can create custom by yourself or take from apps.splunk.com,

With the few information you share, this is what I can hint, in addition I suggest to read at https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/User/DataSplunkCloudcanindex how to get data in Splunk Cloud and see at apps.splunk.com the list of the sources you can take.

Ciao.
Giuseppe

0 Karma

rahul1502133
Explorer

So what u are saying is that Install Universal forwarder on the servers and then index the data in splunk thats fine .....but here my question is that how to find the availability (UP OR DOWN) status of a server based in the indexed data
Coz if we consider a scenario of logs not coming into splunk as benchmark to say that a server is up or down it could only mean that the agent is down and the server might be running.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rahul1502133,
you should creata a lookup (called e.g. perimeter.csv) containing all the servers to monitor (in a column called host) and then run a search like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=upper(host) | fields host count ]
| stats sum(count) AS total BY host

In this way all the servers with total=0 are down and the other are up.
You can create an alert adding at the end of the search | where total=0 triggering when there are down servers.

If instead you want a dashboard, eventually in graphic mode, you can see my answer https://answers.splunk.com/answers/788233/servers-availability-status.html#answer-787047

If you want, you can also enrich your panel, adding to the lookup other information (e.g. IP, role, description, etc...) and adding them to the search, something like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=upper(host) | fields host count IP Role Description ]
| stats values(IP) AS IP values(Role) AS Role values(Description) AS Description sum(count) AS total BY host

Ciao.
Giuseppe

0 Karma

jackin
Path Finder

Hi @gcusello 

can you provide same for single host

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jackin,

for a single host try something like this:

 

index=your_index host=your_host

 

if you have events the server is up, if you haven't events the server is down.

Ciao.

Giuseppe

0 Karma

rahul1502133
Explorer

HI @gcusello
Thank you for the query ......but the above query only works if the logs are coming, what if the
Splunk agent on the server is down but the server is still up..... so according to the query that server will be concluded as down... could you pls come up with a solution for this scenario.

0 Karma

gcusello
SplunkTrust
SplunkTrust

HI @rahul1502133,

if the server is up and the Splunk agent is down you're completely blind so it's a good idea to have an alert when this occurs!

Anyway, you could create a script in another server that pings all ther servers in your network and sends the output of the script to Splunk, so you can immediately find if the ping fails, but I don't like this solution because the most important thing isn't to test if the server is up, but if the Splunk agent is up and sending logs.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rahul1502133,
If the Splunk Universal Forwarder is down you cannot monitor your server, so I think that there's a good reason to intervene even if the server is up because without UF you're blind!

If instead for you is acceptable that the UF is down and the server is up, you have to create a different way to check this, e.g. a script on one Splunk server that pings all the servers indexing the results, then the search it's very similar.

I don't hint this solution, because (as I said) without UF you're completely blind!

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...