All Apps and Add-ons

Monitoring Linux processes and getting the number of process count.

dpark1113
Explorer

I'm new to Splunk and just installed Splunk Add-on for Unix and Linux.

We have 2 same processes running, both named processA.

I ran a search for "last 30 seconds":

sourcetype="ps" processA | stats count

This gives event count(1) which I don't want. How do I modify the search so that it gives the search count which is the number of process, in this case 2.

I checked many samples which uses rex which doesn't work for me.

Thanks in advance!

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Do count by process ID and process name.

sourcetype="ps" processA | stats count by process_name, pid
0 Karma

dpark1113
Explorer

I think this only works with Splunk Add-on for Linux which has "process_name" and "pid". I'm using Splunk Add-on for *nix.

This is my sample output:

USER PID PSR %CPU TIME %MEM RSZ VSZ TT S ELAPSED COMMAND
root 1 2 0.0 00:19:17 0.0 7584 194636 ? S 64-17:28:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 22

0 Karma

to4kawa
Ultra Champion

will you provide whole one event?
Do you divide events each one line?

0 Karma

dpark1113
Explorer

This is only one event, there are 2 processes showing in the event. I just want to search that process and output the count as 2. Not sure if I need to do any other configuration after I install the add-on.

4/17/20
8:44:19.000 PM

USER PID PSR %CPU TIME %MEM RSZ VSZ TT S ELAPSED COMMAND
root 1 2 0.0 00:19:20 0.0 7584 194636 ? S 64-20:40:32 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root 2 6 0.0 00:00:03 0.0 0 0 ? S 64-20:40:32 [kthreadd]
root 3 0 0.0 00:00:08 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/0]
root 5 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [kworker/0:0H]
root 7 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [migration/0]
root 8 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [rcu_bh]
root 9 7 0.0 00:14:50 0.0 0 0 ? S 64-20:40:32 [rcu_sched]
root 10 0 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [lru-add-drain]
root 11 0 0.0 00:00:17 0.0 0 0 ? S 64-20:40:32 [watchdog/0]
root 12 1 0.0 00:00:17 0.0 0 0 ? S 64-20:40:32 [watchdog/1]
root 13 1 0.0 00:00:11 0.0 0 0 ? S 64-20:40:32 [migration/1]
root 14 1 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/1]
root 16 1 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [kworker/1:0H]
root 17 2 0.0 00:00:14 0.0 0 0 ? S 64-20:40:32 [watchdog/2]
root 18 2 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [migration/2]
root 19 2 0.0 00:00:00 0.0 0 0 ? S 64-20:40:32 [ksoftirqd/2]

0 Karma

to4kawa
Ultra Champion

I don't know ps output.
output the count as 2
which one?
[ksoftirqd/2]
this digit?

if you want to count PSR ,

sourcetype="ps" "Process Name"
| streamstats count as session
| stats max(PSR)  as counts by session
| stats sum(counts)
0 Karma

dpark1113
Explorer

What is PSR?

And the command didn't work for me. Getting only 1 for "stats max(PSR) as counts by session", no result if I add "stats sum(counts)"

I'm still learning how to do this and shouldn't be this difficult to output the # of the processes in a single event. To simply put, my goal is to counting the matches in every row or line in a single event.

0 Karma

to4kawa
Ultra Champion

I see you don't extract fields appropriately.

  1. you should extract fields. where is the process number field?
  2. I don't know what it counts numbers. If you know that, please provide here.
  3. stats count counts the events. if one event = one process is good. but it is not, you should count other thing.
  4. Breaking lines is easy. I just don't know if it's needed.
0 Karma

dpark1113
Explorer

Sorry if I didn't make myself clear. Still learning and probably not using the correct Splunk terminology.

There are 2 processes running and my goal is to make sure 2 are running. So I want to run a search and output telling me 2 are running.

There is no process number field, just process name.

So I take this is not possible in Splunk? Maybe I'm asking too specific. 😞

Basically, I have a bunch lines in a single event. I want to search a specific word in that event and output the total number of the word I'm searching in generally speaking.

0 Karma

to4kawa
Ultra Champion

I don't know what it counts numbers. If you know that, please provide here.

There are 2 processes running
you know that, but we can't.

There is only one COMMAND,

you say What is PSR? so, this is not the reason.

why do you say There are 2 processes running ?

0 Karma

dpark1113
Explorer

As I described in my first post, I have two same processes running. I need to find whether those 2 processes are running or not. I take this is not possible in Splunk. Monitoring a single process is easy, 1 event = 1 process when you do a process search.

I might have to submit a ticket to Splunk for this but I'm afraid they wouldn't know the answer either.

0 Karma

to4kawa
Ultra Champion

There is no log, nobody make query. the log indicates two process runnig, splunk can find and count.

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...