topic Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80% in Splunk Search

Set an alert for 3 hosts when their CPU Utilization exceeds 80%

hishamjan — Tue, 02 Mar 2021 08:31:46 GMT

Hi,

I have Splunk Add-on for Unix and Linux installed on my 3 hosts sending data to an Indexer.

I have created a dashboard of CPU utilization of each host using the vmstat command.

However, I would like to create an alert that will send an alert message every time any of those three forwarders exceed 80% of CPU utilization.

The query for CPU utilization is as follows:

index=os sourcetype=df | dedup host | multikv | table host Filesystem Size Used Avail UsePct

Is there any way I can achieve this?

If so, I'd like to know and edit my search above to cater for my needs.

Thanks in advance to anyone willing to help.

Regards

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

gcusello — Tue, 02 Mar 2021 08:02:32 GMT

Hi @hishamjan,

you have to use the same search to idewntify the threshold value, something like this:

The you can save this search as an alert.

Only one additional hint: in the main search use always the index, the search will be quicker!

Ciao.

Giuseppe

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

ITWhisperer — Tue, 02 Mar 2021 08:03:18 GMT

Is the stat already extracted into a field in vmstat? If not, is it contained in the _raw event? What does this look like?

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

hishamjan — Tue, 02 Mar 2021 08:13:10 GMT

Hi @gcusello

Thanks for the reply but it shows no results found when I ran your query.

Is there any way, by chance, to run timechart instead of a table?

Regards

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

hishamjan — Tue, 02 Mar 2021 08:14:10 GMT

Hi,

No, vmstat is a sourcetype that I explicitly defined in the ~/local/inputs.conf file

Regards

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

ITWhisperer — Tue, 02 Mar 2021 08:19:54 GMT

By CPU utilisation, do you mean memory utilisation and not processor utilisation?

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

hishamjan — Tue, 02 Mar 2021 08:30:57 GMT

Hi,

I mean memory utilization

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

gcusello — Tue, 02 Mar 2021 08:32:13 GMT

Hi @hishamjan,

at first find the correct frequency and timeframe to run your search, to be sure to take the data.

Then, to check your alert, try the search with a different threshold and see if it matches the condition.

At least, don't use the condition and see what are the results, to analyze them and eventually correct the search.

Ciao.

Giuseppe

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

hishamjan — Tue, 02 Mar 2021 08:33:42 GMT

Hi @gcusello

i'm sorry for the mistake, the code sample I attached was from another post.

The one I'm looking for is attached below:

index=os sourcetype=df | dedup host | multikv | table host Filesystem Size Used Avail UsePct

I need to have an alert attached to this query when the CPU utilization exceeds 80%.

I'm sorry if I caused any inconvenience.

Regards

Re: Set an alert for 3 hosts when their CPU Utilization exceeds 80%

gcusello — Tue, 02 Mar 2021 09:24:56 GMT

Hi @hishamjan,

the best approach is to install the Splunk App for Linux and Unix (https://splunkbase.splunk.com/app/273/) where you can find all the searches you need.

Anyway, try this:

index=os | multikv | stats max(pctCPU) as maxCPU values(Size) AS Size max(Used) AS used max(Avail) AS Avail BY host Filesystem | where maxCPU>80

Ciao.

Giuseppe