Splunk IT Service Intelligence

How to add to ITSI dashboard the Status of services/ processes running in the OS?

nclancy_splunk
Splunk Employee
Splunk Employee

One requirement is monitoring the Status of Services/processes running in operating systems. Using Splunk ITSI is there a way to do provide the necessary data input to display as an ITSI monitored icon?

Tags (2)
0 Karma

nclancy_splunk
Splunk Employee
Splunk Employee

The key point here is that ITSI works entirely off numeric values and with this in mind if your script or routine returns a numeric value it can be used in ITSI's dashboard.

So a quick google search turns up the utility sc on windows to query services and their running status

sc query

without arguments it returns a list of services and details about them including their current state. The state contains a numeric value that you could extract and use in ITSI. In this example a running service is showing 4 and a stopped service is showing 1. You can then assign a threshold for your KPI where above 3 is started / green and below 2 is failed and stopped. Anything in between could be orange where the service is either starting up or stopping.

SERVICE_NAME: wuauserv
DISPLAY_NAME: Windows Update
TYPE : 20 WIN32_SHARE_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_PRESHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

An example of a stopped process.

C:\Users\Administrator>sc query ALG

SERVICE_NAME: ALG
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1077 (0x435)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

For processes in windows you have "tasklist" so if the process is present in the list it has a 1 and if not a 0.

For linux you can also use the process table to check if it is running [ps -eaf] and most services in linux have a status command so although painful you could run it for each service you need to check. Again you need to select for a numeric criteria and based on this criteria generate a number that can be passed to ITSI.

e.g. $SPLUNK_HOME/bin/splunk status

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...