Splunk ITSI

How to add to ITSI dashboard the Status of services/ processes running in the OS?

nclancy_splunk
Splunk Employee
Splunk Employee

One requirement is monitoring the Status of Services/processes running in operating systems. Using Splunk ITSI is there a way to do provide the necessary data input to display as an ITSI monitored icon?

Tags (2)
0 Karma

nclancy_splunk
Splunk Employee
Splunk Employee

The key point here is that ITSI works entirely off numeric values and with this in mind if your script or routine returns a numeric value it can be used in ITSI's dashboard.

So a quick google search turns up the utility sc on windows to query services and their running status

sc query

without arguments it returns a list of services and details about them including their current state. The state contains a numeric value that you could extract and use in ITSI. In this example a running service is showing 4 and a stopped service is showing 1. You can then assign a threshold for your KPI where above 3 is started / green and below 2 is failed and stopped. Anything in between could be orange where the service is either starting up or stopping.

SERVICE_NAME: wuauserv
DISPLAY_NAME: Windows Update
TYPE : 20 WIN32_SHARE_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_PRESHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

An example of a stopped process.

C:\Users\Administrator>sc query ALG

SERVICE_NAME: ALG
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1077 (0x435)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

For processes in windows you have "tasklist" so if the process is present in the list it has a 1 and if not a 0.

For linux you can also use the process table to check if it is running [ps -eaf] and most services in linux have a status command so although painful you could run it for each service you need to check. Again you need to select for a numeric criteria and based on this criteria generate a number that can be passed to ITSI.

e.g. $SPLUNK_HOME/bin/splunk status

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...