topic Re: Monitoring Linux processes and getting the number of process count. in All Apps and Add-ons

Monitoring Linux processes and getting the number of process count.

dpark1113 — Fri, 17 Apr 2020 15:48:26 GMT

I'm new to Splunk and just installed Splunk Add-on for Unix and Linux.

We have 2 same processes running, both named processA.

I ran a search for "last 30 seconds":

sourcetype="ps" processA | stats count

This gives event count(1) which I don't want. How do I modify the search so that it gives the search count which is the number of process, in this case 2.

I checked many samples which uses rex which doesn't work for me.

Thanks in advance!

Re: Monitoring Linux processes and getting the number of process count.

manjunathmeti — Fri, 17 Apr 2020 17:19:59 GMT

Do count by process ID and process name.

sourcetype="ps" processA | stats count by process_name, pid

Re: Monitoring Linux processes and getting the number of process count.

dpark1113 — Fri, 17 Apr 2020 17:33:11 GMT

I think this only works with Splunk Add-on for Linux which has "process_name" and "pid". I'm using Splunk Add-on for *nix.

This is my sample output:

USER PID PSR %CPU TIME %MEM RSZ VSZ TT S ELAPSED COMMAND
root 1 2 0.0 00:19:17 0.0 7584 194636 ? S 64-17:28:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 22

Re: Monitoring Linux processes and getting the number of process count.

to4kawa — Fri, 17 Apr 2020 20:34:00 GMT

will you provide whole one event?
Do you divide events each one line?

Re: Monitoring Linux processes and getting the number of process count.

dpark1113 — Wed, 30 Sep 2020 05:05:59 GMT

This is only one event, there are 2 processes showing in the event. I just want to search that process and output the count as 2. Not sure if I need to do any other configuration after I install the add-on.

4/17/20
8:44:19.000 PM

USER PID PSR %CPU TIME %MEM RSZ VSZ TT S ELAPSED COMMAND
root 1 2 0.0 00:19:20 0.0 7584 194636 ? S 64-20:40:32 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root 2 6 0.0 00:00:03 0.0 0 0 ? S 64-20:40:32 [kthreadd]
root 3 0 0.0 00:00:08 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/0]
root 5 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [kworker/0:0H]
root 7 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [migration/0]
root 8 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [rcu_bh]
root 9 7 0.0 00:14:50 0.0 0 0 ? S 64-20:40:32 [rcu_sched]
root 10 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [lru-add-drain]
root 11 0 0.0 00:00:17 0.0 0 0 ? S 64-20:40:32 [watchdog/0]
root 12 1 0.0 00:00:17 0.0 0 0 ? S 64-20:40:32 [watchdog/1]
root 13 1 0.0 00:00:11 0.0 0 0 ? S 64-20:40:32 [migration/1]
root 14 1 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/1]
root 16 1 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [kworker/1:0H]
root 17 2 0.0 00:00:14 0.0 0 0 ? S 64-20:40:32 [watchdog/2]
root 18 2 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [migration/2]
root 19 2 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/2]

Re: Monitoring Linux processes and getting the number of process count.

to4kawa — Fri, 17 Apr 2020 21:31:41 GMT

I don't know ps output.
output the count as 2
which one?
[ksoftirqd/2]
this digit?

if you want to count PSR ,

sourcetype="ps" "Process Name"
| streamstats count as session
| stats max(PSR)  as counts by session
| stats sum(counts)

Re: Monitoring Linux processes and getting the number of process count.

dpark1113 — Sat, 18 Apr 2020 18:39:05 GMT

What is PSR?

And the command didn't work for me. Getting only 1 for "stats max(PSR) as counts by session", no result if I add "stats sum(counts)"

I'm still learning how to do this and shouldn't be this difficult to output the # of the processes in a single event. To simply put, my goal is to counting the matches in every row or line in a single event.

Re: Monitoring Linux processes and getting the number of process count.

to4kawa — Sat, 18 Apr 2020 21:35:07 GMT

I see you don't extract fields appropriately.

you should extract fields. where is the process number field?
I don't know what it counts numbers. If you know that, please provide here.
stats count counts the events. if one event = one process is good. but it is not, you should count other thing.
Breaking lines is easy. I just don't know if it's needed.

Re: Monitoring Linux processes and getting the number of process count.

dpark1113 — Sun, 19 Apr 2020 12:50:19 GMT

Sorry if I didn't make myself clear. Still learning and probably not using the correct Splunk terminology.

There are 2 processes running and my goal is to make sure 2 are running. So I want to run a search and output telling me 2 are running.

There is no process number field, just process name.

So I take this is not possible in Splunk? Maybe I'm asking too specific. 😞

Basically, I have a bunch lines in a single event. I want to search a specific word in that event and output the total number of the word I'm searching in generally speaking.

Re: Monitoring Linux processes and getting the number of process count.

to4kawa — Sun, 19 Apr 2020 21:42:10 GMT

I don't know what it counts numbers. If you know that, please provide here.

There are 2 processes running
you know that, but we can't.

There is only one COMMAND,

you say What is PSR? so, this is not the reason.

why do you say There are 2 processes running ?

Re: Monitoring Linux processes and getting the number of process count.

dpark1113 — Mon, 20 Apr 2020 13:00:32 GMT

As I described in my first post, I have two same processes running. I need to find whether those 2 processes are running or not. I take this is not possible in Splunk. Monitoring a single process is easy, 1 event = 1 process when you do a process search.

I might have to submit a ticket to Splunk for this but I'm afraid they wouldn't know the answer either.

Re: Monitoring Linux processes and getting the number of process count.

to4kawa — Mon, 20 Apr 2020 20:59:08 GMT

There is no log, nobody make query. the log indicates two process runnig, splunk can find and count.