Alerting

checking if a a process is still running (time based comparison of event counts)

petersob
Explorer

Hello,

I want to check if a process is still running. The process is logging periodically a short info on polling a directory.
Now, I will to use that info to detect if the process is not running anymore.
Normally, a simple search for the info would be:

search myprocess is doing the work

and then defining the alert for eventcount = 0 would do the job. But...
I would like to do it in a generic way, that means, that "is doing the work" is actually unknown. This has to work with every process which is logging something periodically.

Now, I have a defined a search doing a simple event count over time periods (hourly):

source=/logs/processes.log earliest=-3h@h | chart count over process by _time span=1h 

this give me a table like this:

process    1381651200    1381654800    1381658400
myproc1    12            12            1
myproc2    15            15            0
myproc3    233           243           102

well, that means that the process "myproc2" did not log anything in the last hour and I need an alert for that issue.

What I've tried is to extract somehow the last column and evaluate the apopriate process name (process column) but I couldn't get it done.

Any Ideas hot to do that?
Maybe I'm thinking too complicated?

Regards,
Peter

0 Karma

petersob
Explorer

after hour of reading several postings here finally I found a solution for this problem.

my problem is nothing else as determining the gap between the last log info and now. Well, this search does it for me now:

source=/logs/processes.log earliest=-2h | stats max(_time) As LatestTime by process | eval Gap=(now()-LatestTime) | search Gap>3600

means: every process what was "seen" in the last 2 hours must be "seen" during the last hour also. If not I can raise an alert now!

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Mastering Threat Intelligence in ES 8.5, Splunk AI Assistant v2, and More from Splunk ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

Break the Build: Inside the KubeDoom Lounge at .conf26

    You step up to the machine. The pixelated corridors of a certain 1993 FPS load in front of you, EMP Pulse ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...