Splunk Search

Set an alert for 3 hosts when their CPU Utilization exceeds 80%

hishamjan
Explorer

Hi,

I have Splunk Add-on for Unix and Linux installed on my 3 hosts sending data to an Indexer.

 

I have created a dashboard of CPU utilization of each host using the vmstat command.

 

However, I would like to create an alert that will send an alert message every time any of those three forwarders exceed 80% of CPU utilization.

 

The query for CPU utilization is as follows:

 

 

 

index=os  sourcetype=df  
            | dedup host 
            | multikv 
            | table host Filesystem Size Used Avail UsePct

 

 

Is there any way I can achieve this? 

If so, I'd like to know and edit my search above to cater for my needs.

 

Thanks in advance to anyone willing to help.

Regards

Labels (5)
Tags (3)
0 Karma

ITWhisperer
Legend

Is the stat already extracted into a field in vmstat? If not, is it contained in the _raw event? What does this look like?

0 Karma

hishamjan
Explorer

Hi,

No, vmstat is a sourcetype that I explicitly defined in the ~/local/inputs.conf file

 

Regards

0 Karma

ITWhisperer
Legend

By CPU utilisation, do you mean memory utilisation and not processor utilisation?

0 Karma

hishamjan
Explorer

Hi, 

I mean memory utilization

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @hishamjan,

you have to use the same search to idewntify the threshold value, something like this:

source="vmstat"
| dedup host 
| eval host=upper(host), FreeGBs=FreeMBytes/1024, TotalGBs=TotalMBytes/1024, UsedGBs=UsedMBytes/1024
| eval perc=UsedGBs/TotalMBytes*100
| where perc>80
| table host TotalGBs UsedGBs FreeGBs memFreePct memUsedPct
| sort host

The you can save this search as an alert.

Only one additional hint: in the main search use always the index, the search will be quicker!

Ciao.

Giuseppe

0 Karma

hishamjan
Explorer

Hi @gcusello 

i'm sorry for the mistake, the code sample I attached was from another post. 

The one I'm looking for is attached below:

index=os  sourcetype=df  
            | dedup host 
            | multikv 
            | table host Filesystem Size Used Avail UsePct

I need to have an alert attached to this query when the CPU utilization exceeds 80%.

 

I'm sorry if I caused any inconvenience.

 

Regards

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @hishamjan,

the best approach is to install the Splunk App for Linux and Unix (https://splunkbase.splunk.com/app/273/) where you can find all the searches you need.

Anyway, try this:

index=os
| multikv 
| stats max(pctCPU) as maxCPU values(Size) AS Size max(Used) AS used max(Avail) AS Avail BY host Filesystem 
| where maxCPU>80

Ciao.

Giuseppe

0 Karma

hishamjan
Explorer

Hi @gcusello 

Thanks for the reply but it shows no results found when I ran your query.

 

Is there any way, by chance, to run timechart instead of a table? 

 

Regards

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @hishamjan,

at first find the correct frequency and timeframe to run your search, to be sure to take the data.

Then, to check your alert, try the search with a different threshold and see if it matches the condition.

At least, don't use the condition and see what are the results, to analyze them and eventually correct the search.

Ciao.

Giuseppe

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.