Monitoring Linux processes and getting the number of process count.

[dpark1113]

I'm new to Splunk and just installed Splunk Add-on for Unix and Linux.

We have 2 same processes running, both named processA.

I ran a search for "last 30 seconds":

sourcetype="ps" processA | stats count

This gives event count(1) which I don't want. How do I modify the search so that it gives the search count which is the number of process, in this case 2.

I checked many samples which uses rex which doesn't work for me.

Thanks in advance!

[manjunathmeti]

Do count by process ID and process name.

sourcetype="ps" processA | stats count by process_name, pid

[dpark1113]

I think this only works with Splunk Add-on for Linux which has "process_name" and "pid". I'm using Splunk Add-on for *nix.

This is my sample output:

USER PID PSR %CPU TIME %MEM RSZ VSZ TT S ELAPSED COMMAND
root 1 2 0.0 00:19:17 0.0 7584 194636 ? S 64-17:28:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 22

[to4kawa]

will you provide whole one event?
Do you divide events each one line?

[dpark1113]

This is only one event, there are 2 processes showing in the event. I just want to search that process and output the count as 2. Not sure if I need to do any other configuration after I install the add-on.

4/17/20
8:44:19.000 PM

USER PID PSR %CPU TIME %MEM RSZ VSZ TT S ELAPSED COMMAND
root 1 2 0.0 00:19:20 0.0 7584 194636 ? S 64-20:40:32 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root 2 6 0.0 00:00:03 0.0 0 0 ? S 64-20:40:32 [kthreadd]
root 3 0 0.0 00:00:08 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/0]
root 5 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [kworker/0:0H]
root 7 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [migration/0]
root 8 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [rcu_bh]
root 9 7 0.0 00:14:50 0.0 0 0 ? S 64-20:40:32 [rcu_sched]
root 10 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [lru-add-drain]
root 11 0 0.0 00:00:17 0.0 0 0 ? S 64-20:40:32 [watchdog/0]
root 12 1 0.0 00:00:17 0.0 0 0 ? S 64-20:40:32 [watchdog/1]
root 13 1 0.0 00:00:11 0.0 0 0 ? S 64-20:40:32 [migration/1]
root 14 1 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/1]
root 16 1 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [kworker/1:0H]
root 17 2 0.0 00:00:14 0.0 0 0 ? S 64-20:40:32 [watchdog/2]
root 18 2 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [migration/2]
root 19 2 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/2]

[to4kawa]

I don't know ps output.
output the count as 2
which one?
[ksoftirqd/2]
this digit?

if you want to count PSR ,

sourcetype="ps" "Process Name"
| streamstats count as session
| stats max(PSR)  as counts by session
| stats sum(counts)

[dpark1113]

What is PSR?

And the command didn't work for me. Getting only 1 for "stats max(PSR) as counts by session", no result if I add "stats sum(counts)"

I'm still learning how to do this and shouldn't be this difficult to output the # of the processes in a single event. To simply put, my goal is to counting the matches in every row or line in a single event.

[to4kawa]

I see you don't extract fields appropriately.

1. you should extract fields. where is the process number field?
2. I don't know what it counts numbers. If you know that, please provide here.
3. stats count counts the events. if one event = one process is good. but it is not, you should count other thing.
4. Breaking lines is easy. I just don't know if it's needed.

[dpark1113]

Sorry if I didn't make myself clear. Still learning and probably not using the correct Splunk terminology.

There are 2 processes running and my goal is to make sure 2 are running. So I want to run a search and output telling me 2 are running.

There is no process number field, just process name.

So I take this is not possible in Splunk? Maybe I'm asking too specific. 😞

Basically, I have a bunch lines in a single event. I want to search a specific word in that event and output the total number of the word I'm searching in generally speaking.

[to4kawa]

I don't know what it counts numbers. If you know that, please provide here.

There are 2 processes running
you know that, but we can't.

There is only one COMMAND,

you say What is PSR? so, this is not the reason.

why do you say There are 2 processes running ?

[dpark1113]

As I described in my first post, I have two same processes running. I need to find whether those 2 processes are running or not. I take this is not possible in Splunk. Monitoring a single process is easy, 1 event = 1 process when you do a process search.

I might have to submit a ticket to Splunk for this but I'm afraid they wouldn't know the answer either.

[to4kawa]

There is no log, nobody make query. the log indicates two process runnig, splunk can find and count.