Splunk Search

How to create a table of hosts running or not running a given Linux service?

bsg273
Path Finder

I'm trying to create a statistics table for whether or not a given Linux service is running on a set of hosts.  For example, for service "rhnsd" running on hosts "my-*" ...

Host State
my-db-1 Running
my-db-2 Stopped
my-web-1 Running
my-web-2 Stopped

 

I have the ps module enabled, so I can use that as a source/sourcetype, but not sure how to eloquently display all hosts and the state of the given service like I've illustrated above.  Any help is greatly appreciated.

Labels (3)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @bsg273,

if the field containing the service name is extracted and called "service" (if you need help in field extraction, you can ask to the community sharing a sample of your logs), you have to create a lookup (called e.g. services.csv) containing the services to monitor, eventually associated to the host, something like this:

host   service
host1  rhnsd
host1  service1
host2  rhnsd
host2  service2

 then you have to run a search like the following:

index=os
| eval host=lower(host), service=lower(service)
| stats count BY host service
| append [ | inputlookup services.csv | eval host=lower(host), service=lower(service), count=0 | fields host service count ]
| stats sum(count) AS total BY host service
| rename host AS Host service AS Service
| eval Status=if(total=0,"Not Present","Present")
| table Host Service Status

Ciao.

Giuseppe

bsg273
Path Finder

I'm not seeing where this uses ps or anything else to see if a given service/process is running.  Could you please advise?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @bsg273,

as @richgalloway said, this is a search on a Splunk index where linux logs are stored, but you have to run ps command on the machines and send these logs to Splunk.

To do this you have to install a Universal Forwarder on the target servers and deploy to them the Splunk Add-On for Unix and Linux (https://splunkbase.splunk.com/app/833/).

In this way, you'll have in an index (usually os) the results of the ps command to use for a search like the one I hinted.

Remember to enable the inputs that you need (in your case the ps) and insert the option for the index to use.

Ciao.

Giuseppe

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

ps runs on each server and sends data to Splunk at intervals.  It is not run at search-time.  The data usually is stored in the 'os' index.  Giuseppe's query searches that index.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Finding something that is not there is not Splunk's strong suit.  See this blog entry for a good write-up on it.

https://www.duanewaddle.com/proving-a-negative/

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...