Splunk Search

How do I sum columns in a multiline event

cdo_splunk
Splunk Employee
Splunk Employee

We have a script that gets the output of the command below and output it as a single event with multiline

ps -wweo uname,pid,psr,pcpu,cputime,pmem,rsz,vsz,tty,s,etime,args

I would like to write a search that filters through the lines to look for the the a certain string looks for a particular sting in a certain column, if exists sum the value of another column and return the string and the sum. E.g.

USER PID PSR pctCPU CPUTIME pctMEM RSZ_KB VSZ_KB TTY S ELAPSED COMMAND ARGS
root 1848 0 0.0 00:00:00 0.0 2128 476368 ? S 18:19:03 automount --pid-file_/var/run/autofs.pid
apache 2082 0 0.1 00:00:37 0.4 17868 402748 ? S 06:26:57 httpd <noArgs>
apache 2083 0 0.1 00:00:36 0.4 17872 402748 ? S 06:26:56 httpd <noArgs>
apache 2084 0 0.1 00:00:37 0.4 17908 402932 ? S 06:26:56 httpd <noArgs>
apache 2383 0 0.1 00:00:36 0.4 17872 402748 ? S 06:25:53 httpd <noArgs>
root 4951 0 0.0 00:00:00 0.0 1300 79968 ? S 18:17:40 sshd <noArgs>
root 5354 0 0.0 00:00:01 0.0 2696 91304 ? S 18:15:48 sendmail: accepting_connections
smmsp 5361 0 0.0 00:00:00 0.0 2108 82756 ? S 18:15:48 sendmail: Queue_runner@01:00:00_for_/var/spool/clientmqueue

The result of the search would be:

.4 httpd 
Tags (3)

bshuler_splunk
Splunk Employee
Splunk Employee
|localop| stats count | eval data="USER PID PSR pctCPU CPUTIME pctMEM RSZ_KB VSZ_KB TTY S ELAPSED COMMAND ARGS
root 1848 0 0.0 00:00:00 0.0 2128 476368 ? S 18:19:03 automount --pid-file_/var/run/autofs.pid
apache 2082 0 0.1 00:00:37 0.4 17868 402748 ? S 06:26:57 httpd <noArgs>
apache 2083 0 0.1 00:00:36 0.4 17872 402748 ? S 06:26:56 httpd <noArgs>
apache 2084 0 0.1 00:00:37 0.4 17908 402932 ? S 06:26:56 httpd <noArgs>
apache 2383 0 0.1 00:00:36 0.4 17872 402748 ? S 06:25:53 httpd <noArgs>
root 4951 0 0.0 00:00:00 0.0 1300 79968 ? S 18:17:40 sshd <noArgs>
root 5354 0 0.0 00:00:01 0.0 2696 91304 ? S 18:15:48 sendmail: accepting_connections
smmsp 5361 0 0.0 00:00:00 0.0 2108 82756 ? S 18:15:48 sendmail: Queue_runner@01:00:00_for_/var/spool/clientmqueue" | rex max_match=9999 field=data "(?P<lines>[^\n]+)" | table lines | mvexpand lines | rex field=lines "^(?<USER>\S+)\s(?<PID>\S+)\s(?<PSR>\S+)\s(?<pctCPU>\S+)\s(?<CPUTIME>\S+)\s(?<pctMEM>\S+)\s(?<RSZ_KB>\S+)\s(?<VSZ_KB>\S+)\s(?<TTY>\S+)\s(?<S>\S+)\s(?<ELAPSED>\S+)\s(?<COMMAND>\S+)\s(?<ARGS>.*)" | fields - lines | search USER!="USER"

woodcock
Esteemed Legend

Please re-edit your question; it has obvious cut/paste errors and is incomplete/incomprehensible. For example, you do not list either field name for your example.

0 Karma
Get Updates on the Splunk Community!

Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud ...

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it ...

Real-Time Fraud Detection: How Splunk Dashboards Protect Financial Institutions

Financial fraud isn't slowing down. If anything, it's getting more sophisticated. Account takeovers, credit ...

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...