Splunk Search

Splunk Query to list top CPU consuming process when utilisation is greater than 70%

aparnaa
Path Finder

Hello

I have 2 queries, one to find top 10 CPU utilising process and 1 more for finding the avg CPU utilisation but I am not sure how to combine both of them in 1 query

My requirement is, if CPU utilisation is greater than 70% then we must get the top 10 running process for those servers so that it will help us in troubleshooting in 1 report

Query-1 : To get list of Top process

index= infra earliest=-15m source="Perfmon:Process" counter="% Processor Time" (instance!="_Total" AND instance!="Idle" AND instance!="System") | eventstats avg(Value) as AvgValue by host,instance | top instance by AvgValue,host limit=10 showperc=f showcount=f| sort -host,-AvgValue

Query-2: Get the list of servers whose CPU utilisation is greater than 70%

| loadjob savedsearch="nobody:cdfs-infg:infra_saved_search"|stats latest(CPU) as CPU,latest(Memory) as Memory, latest(Swap) as Swap by Server | lookup inventory "Server Name" as Server OUTPUT "Application Name"
|table "Application Name",Server,CPU,Memory,Swap
|eval CPU=round(CPU,2)| eval total_memory=round((Memory+Swap),2) |eval Swap=round(Swap,2)| dedup Server,CPU,Memory,Swap | where CPU>70 OR total_memory>70 | sort - total_memory

Can you please help ?

Thank you so much !
aparna

0 Karma

gokadroid
Motivator

In order to form a query for this we should have a way to get all the servers which have greater than 70% CPU utilization and these servers should be searchable in index where you have process listed. If that is so you can proceed as follows:

outer search to get the host and process  [ inner search which will return the hosts which have greater than 70% CPU Utilization and will be used as search strings in outer query ]
| completing the outer search to get the top processes

In your example, I am thinking the field Server is what has hostname and should be searchable in outer query as a host:

index= infra earliest=-15m source="Perfmon:Process" counter="% Processor Time" (instance!="_Total" AND instance!="Idle" AND instance!="System")  
[| loadjob savedsearch="nobody:cdfs-infg:infra_saved_search"|stats latest(CPU) as CPU by Server
|eval CPU=round(CPU,2)
| dedup Server,CPU
| where CPU>70
| table Server  ]
| eventstats avg(Value) as AvgValue by host,instance | top instance by AvgValue,host limit=10 showperc=f showcount=f| sort -host,-AvgValue

NOTE In the inner query I have used all the calculations which were sufficient to calculate | where CPU>70. If you need to calculate total_memory>70 as well then some tweaks might be needed.

0 Karma

aparnaa
Path Finder

I tried but I am getting no results found 😞

gokadroid
Motivator

Check these:

1) Are really servers running at 70% now? Can you try to lower that value to | where CPU > 5 and see if it returns something.

2) Run inner query separately and see if you get values for Server fields.

3) Is value of host field in outer query and value of Server from inner query similar?? Like if inner Server field has values like "abc.domain.com", outer index field host should have values like "abc.domain.com" as well. Or at least the string "abc.domain.com" should be present in outer index events for outer query to search it and return events.

0 Karma

aparnaa
Path Finder

1) host and server have matching values, when i try them separately its working
2) its not working even when i take way the condition "where"

gokadroid
Motivator

Ok, interesting, maybe we are missing something...can you tweak this part of inner query as follows:

| dedup Server,CPU
| where CPU>70
 | table Server  

Change above lines to below lines to see if it actually returns something...and then complete your outer query.. this should work

| dedup Server,CPU
| where CPU>70
| return 10000 $Server  
0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...