Splunk IT Service Intelligence

How to add to ITSI dashboard the Status of services/ processes running in the OS?

nclancy_splunk
Splunk Employee
Splunk Employee

One requirement is monitoring the Status of Services/processes running in operating systems. Using Splunk ITSI is there a way to do provide the necessary data input to display as an ITSI monitored icon?

Tags (2)
0 Karma

nclancy_splunk
Splunk Employee
Splunk Employee

The key point here is that ITSI works entirely off numeric values and with this in mind if your script or routine returns a numeric value it can be used in ITSI's dashboard.

So a quick google search turns up the utility sc on windows to query services and their running status

sc query

without arguments it returns a list of services and details about them including their current state. The state contains a numeric value that you could extract and use in ITSI. In this example a running service is showing 4 and a stopped service is showing 1. You can then assign a threshold for your KPI where above 3 is started / green and below 2 is failed and stopped. Anything in between could be orange where the service is either starting up or stopping.

SERVICE_NAME: wuauserv
DISPLAY_NAME: Windows Update
TYPE : 20 WIN32_SHARE_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_PRESHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

An example of a stopped process.

C:\Users\Administrator>sc query ALG

SERVICE_NAME: ALG
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1077 (0x435)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

For processes in windows you have "tasklist" so if the process is present in the list it has a 1 and if not a 0.

For linux you can also use the process table to check if it is running [ps -eaf] and most services in linux have a status command so although painful you could run it for each service you need to check. Again you need to select for a numeric criteria and based on this criteria generate a number that can be passed to ITSI.

e.g. $SPLUNK_HOME/bin/splunk status

0 Karma
Get Updates on the Splunk Community!

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...