NIX add-on for Splunk: How to return a value from a subsearch and populating it in a field for each entry

[vader13]

I'm using the *NIX add-on for Splunk.

We receive "TOP.sh" information into Splunk. Top provides the process information and pctCPU.

I'm trying to run a statistics table search to do the following:

Pull the TOP data for a specific host.
Return the COMMAND (Process Name) and pctCPU.
Sum the pctCPU for all similar process names.
Create a new column with the total pctCPU for every process.
Divide the pctCPU total for each process by the total pctCPU of the box and put that in a separate field (cpuShare).
Return the process names and cpuShare.

I have a piechart that returns the data and creates the slices. But the statistical format of that doesn't have the percentages. Is there a hidden field I can display or do I need to do the division myself? This is going in a dashboard. I have obscured the real host name with $host_name$

Example of pie chart:

index=os host=$host_name$ source="top" COMMAND=* 
| table _time, host, USER, COMMAND, PID, pctCPU 
| chart sum(pctCPU) over COMMAND

Example of statistical table to evaluate the CPU share:

index=os host=$host_name$ source="top" COMMAND=* 
| table COMMAND, pctCPU 
| stats sum(pctCPU) by COMMAND 
| sort - sum(pctCPU) 
| eval bretCPU = 
    [ search index=os host=$host_name$ source="top" COMMAND=* 
    | table pctCPU 
    | stats sum(pctCPU) as totalPctCPU 
    | return totalPctCPU]

[woodcock]

Try this:

| makeresults
| eval raw="COMMAND=processA,pctCPU=103.8
COMMAND=processB,pctCPU=27.1
COMMAND=processC,pctCPU=3103.9:::COMMAND=processA,pctCPU=431.8
COMMAND=processB,pctCPU=89.1
COMMAND=processC,pctCPU=300.9"
| makemv delim=":::" raw
| mvexpand raw
| makemv delim="
" raw
| mvexpand raw
| rename raw AS _raw
| kv
| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"
| chart sum(pctCPU) AS pctCPU OVER COMMAND
| eventstats sum(pctCPU) AS pctCPUtotal
| eval pctCPUpct = round((100 * pctCPU / pctCPUtotal), 3)

[woodcock]

You need to rework that 2nd search to be all 1 search.

[woodcock]

When I do a pie chart it automatically calculates the percentages and I can see them in the tooltip when I hover over a pie segment. If you need to see them more obviously and all together, then you can calculate them yourself and add them to the name field like this:

index=_* 
| stats count BY component 
| eventstats sum(count) AS total_count
| eval pct = round((100 * count / total_count), 3)
| eval component = component . "(" . pct . "%)"
| table component count

[vader13]

That works for pct of count, but that's not what I'm looking for. Let me show the data it might make more sense.

So the TOP.sh script (*NIX Add-on) runs every 1 minute and returns the processes running and the pctCPU. In our server, we have 44 cores, so you will see process grab 20 or 30 cores at times.

Example:
Time Interval #1

COMMAND pctCPU
processA 103.8%

processB 27.1%
processC 3103.9%

Time Interval #2
COMMAND pctCPU
processA 431.8
processB 89.1
processC 300.9

I wrote a chart search to sum by each process, which combines looks like this:

PieChart
processA 535.6

processB 116.2
processC 3404.8

Splunk creates the piechart and that works fine. It also creates the pie slice percentages. I want this in a statistics table.

So, all I need to do is add a new field to piechart results that totals all of the process percentages together and then divided each process by the total in a new column. Then, return the Process and PCT.

COMMAND pctCPU totalCPU totalPCT
processA 535.6 4056.6 13.2
processB 116.2 4056.6 2.8
processC 3404.8 4056.6 83.9

FINAL
COMMAND totalPCT
processA 13.2
processB 2.8
processC 83.9

I just keep getting a logic error with the syntax above. My subsearch works fine by itself. When I put it in the EVAL, it appears to return a boolean value. The search by itself returns a number.

[woodcock]

You need to ditch the subsearch anyway....

[vader13]

I agree. The pie chart shows it. I want the equivalent data in a table. I'll try your logic and report back.