Splunk ITSI

How to add to ITSI dashboard the Status of services/ processes running in the OS?

nclancy_splunk
Splunk Employee
Splunk Employee

One requirement is monitoring the Status of Services/processes running in operating systems. Using Splunk ITSI is there a way to do provide the necessary data input to display as an ITSI monitored icon?

Tags (2)
0 Karma

nclancy_splunk
Splunk Employee
Splunk Employee

The key point here is that ITSI works entirely off numeric values and with this in mind if your script or routine returns a numeric value it can be used in ITSI's dashboard.

So a quick google search turns up the utility sc on windows to query services and their running status

sc query

without arguments it returns a list of services and details about them including their current state. The state contains a numeric value that you could extract and use in ITSI. In this example a running service is showing 4 and a stopped service is showing 1. You can then assign a threshold for your KPI where above 3 is started / green and below 2 is failed and stopped. Anything in between could be orange where the service is either starting up or stopping.

SERVICE_NAME: wuauserv
DISPLAY_NAME: Windows Update
TYPE : 20 WIN32_SHARE_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_PRESHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

An example of a stopped process.

C:\Users\Administrator>sc query ALG

SERVICE_NAME: ALG
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1077 (0x435)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

For processes in windows you have "tasklist" so if the process is present in the list it has a 1 and if not a 0.

For linux you can also use the process table to check if it is running [ps -eaf] and most services in linux have a status command so although painful you could run it for each service you need to check. Again you need to select for a numeric criteria and based on this criteria generate a number that can be passed to ITSI.

e.g. $SPLUNK_HOME/bin/splunk status

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...