Set an alert for 3 hosts when their CPU Utilization exceeds 80%

[hishamjan]

Hi,

I have Splunk Add-on for Unix and Linux installed on my 3 hosts sending data to an Indexer.

I have created a dashboard of CPU utilization of each host using the vmstat command.

However, I would like to create an alert that will send an alert message every time any of those three forwarders exceed 80% of CPU utilization.

The query for CPU utilization is as follows:

index=os  sourcetype=df  
            | dedup host 
            | multikv 
            | table host Filesystem Size Used Avail UsePct

Is there any way I can achieve this? 

If so, I'd like to know and edit my search above to cater for my needs.

Thanks in advance to anyone willing to help.

Regards

[ITWhisperer]

Is the stat already extracted into a field in vmstat? If not, is it contained in the _raw event? What does this look like?

[hishamjan]

Hi,

No, vmstat is a sourcetype that I explicitly defined in the ~/local/inputs.conf file

Regards

[ITWhisperer]

By CPU utilisation, do you mean memory utilisation and not processor utilisation?

[hishamjan]

Hi, 

I mean memory utilization

[gcusello]

Hi @hishamjan,

you have to use the same search to idewntify the threshold value, something like this:

source="vmstat"
| dedup host 
| eval host=upper(host), FreeGBs=FreeMBytes/1024, TotalGBs=TotalMBytes/1024, UsedGBs=UsedMBytes/1024
| eval perc=UsedGBs/TotalMBytes*100
| where perc>80
| table host TotalGBs UsedGBs FreeGBs memFreePct memUsedPct
| sort host

The you can save this search as an alert.

Only one additional hint: in the main search use always the index, the search will be quicker!

Ciao.

Giuseppe

[hishamjan]

Hi @gcusello 

i'm sorry for the mistake, the code sample I attached was from another post. 

The one I'm looking for is attached below:

index=os  sourcetype=df  
            | dedup host 
            | multikv 
            | table host Filesystem Size Used Avail UsePct

I need to have an alert attached to this query when the CPU utilization exceeds 80%.

I'm sorry if I caused any inconvenience.

Regards

[gcusello]

Hi @hishamjan,

the best approach is to install the Splunk App for Linux and Unix (https://splunkbase.splunk.com/app/273/) where you can find all the searches you need.

Anyway, try this:

index=os
| multikv 
| stats max(pctCPU) as maxCPU values(Size) AS Size max(Used) AS used max(Avail) AS Avail BY host Filesystem 
| where maxCPU>80

Ciao.

Giuseppe

[hishamjan]

Hi @gcusello 

Thanks for the reply but it shows no results found when I ran your query.

Is there any way, by chance, to run timechart instead of a table? 

Regards

[gcusello]

Hi @hishamjan,

at first find the correct frequency and timeframe to run your search, to be sure to take the data.

Then, to check your alert, try the search with a different threshold and see if it matches the condition.

At least, don't use the condition and see what are the results, to analyze them and eventually correct the search.

Ciao.

Giuseppe