Splunk Query to list top CPU consuming process when utilisation is greater than 70%

[aparnaa]

Hello

I have 2 queries, one to find top 10 CPU utilising process and 1 more for finding the avg CPU utilisation but I am not sure how to combine both of them in 1 query

My requirement is, if CPU utilisation is greater than 70% then we must get the top 10 running process for those servers so that it will help us in troubleshooting in 1 report

Query-1 : To get list of Top process

index= infra earliest=-15m source="Perfmon:Process" counter="% Processor Time" (instance!="_Total" AND instance!="Idle" AND instance!="System") | eventstats avg(Value) as AvgValue by host,instance | top instance by AvgValue,host limit=10 showperc=f showcount=f| sort -host,-AvgValue

Query-2: Get the list of servers whose CPU utilisation is greater than 70%

| loadjob savedsearch="nobody:cdfs-infg:infra_saved_search"|stats latest(CPU) as CPU,latest(Memory) as Memory, latest(Swap) as Swap by Server | lookup inventory "Server Name" as Server OUTPUT "Application Name"
|table "Application Name",Server,CPU,Memory,Swap
|eval CPU=round(CPU,2)| eval total_memory=round((Memory+Swap),2) |eval Swap=round(Swap,2)| dedup Server,CPU,Memory,Swap | where CPU>70 OR total_memory>70 | sort - total_memory

Can you please help ?

Thank you so much !
aparna

[gokadroid]

In order to form a query for this we should have a way to get all the servers which have greater than 70% CPU utilization and these servers should be searchable in index where you have process listed. If that is so you can proceed as follows:

outer search to get the host and process  [ inner search which will return the hosts which have greater than 70% CPU Utilization and will be used as search strings in outer query ]
| completing the outer search to get the top processes

In your example, I am thinking the field Server is what has hostname and should be searchable in outer query as a host:

index= infra earliest=-15m source="Perfmon:Process" counter="% Processor Time" (instance!="_Total" AND instance!="Idle" AND instance!="System")  
[| loadjob savedsearch="nobody:cdfs-infg:infra_saved_search"|stats latest(CPU) as CPU by Server
|eval CPU=round(CPU,2)
| dedup Server,CPU
| where CPU>70
| table Server  ]
| eventstats avg(Value) as AvgValue by host,instance | top instance by AvgValue,host limit=10 showperc=f showcount=f| sort -host,-AvgValue

NOTE In the inner query I have used all the calculations which were sufficient to calculate | where CPU>70. If you need to calculate total_memory>70 as well then some tweaks might be needed.

[aparnaa]

I tried but I am getting no results found 😞

[gokadroid]

Check these:

1) Are really servers running at 70% now? Can you try to lower that value to | where CPU > 5 and see if it returns something.

2) Run inner query separately and see if you get values for Server fields.

3) Is value of host field in outer query and value of Server from inner query similar?? Like if inner Server field has values like "abc.domain.com", outer index field host should have values like "abc.domain.com" as well. Or at least the string "abc.domain.com" should be present in outer index events for outer query to search it and return events.

[aparnaa]

1) host and server have matching values, when i try them separately its working
2) its not working even when i take way the condition "where"

[gokadroid]

Ok, interesting, maybe we are missing something...can you tweak this part of inner query as follows:

| dedup Server,CPU
| where CPU>70
 | table Server  

Change above lines to below lines to see if it actually returns something...and then complete your outer query.. this should work

| dedup Server,CPU
| where CPU>70
| return 10000 $Server