Alerting

Reducing Noise of Alerts

ankitagm
New Member

Hi,

I want to implement splunk alert in below way : Please let me know how it can achieve?

I am querying the CPU utilization in near real time i.e every 5 minutes

If CPU Utilization crosses 90% then trigger an alert mail/Log a JIRA And It should not create any further repetitive alert mail till it reaches its specified below threshold value (Here it should not depend on certain amount of time by specifying throttleing, but it should depend on the values i.e if it has come back to threshold trigger a mail saying CPU has come down now till then no further email alert should generate )

Thanks,

Tags (1)
0 Karma
1 Solution

FrankVl
Ultra Champion

You should be able to create a search that returns servername, CPU usage over past 5 minutes, CPU usage over previous 5 minutes (10-5 min ago).

Then from that search create two flavors: one that filters out servers that had below 90 in previous and >90 in current period. And one that filters servers that had >90 in previous period and <90 in current. Then create alerts off of each of these.

Quick example based on CPU stats from introspection logs (replace that with your own source of CPU stats)

index="_introspection" sourcetype=splunk_resource_usage component=Hostwide earliest=-10m@m latest=-5m@m 
| stats avg(data.cpu_idle_pct) as previousCPU by host 
| eval previousCPU=100-previousCPU 
| append 
    [| search index="_introspection" sourcetype=splunk_resource_usage component=Hostwide earliest=-5m@m latest=now() 
    | stats avg(data.cpu_idle_pct) as currentCPU by host 
    | eval currentCPU=100-currentCPU
        ]
| stats max(previousCPU) as previousCPU max(currentCPU) as currentCPU by host

View solution in original post

0 Karma

ankitagm
New Member

Hi

Thnx,Yes query shows previous and current cpu.
But how can i control the action of sending an email until cpu falls down to below threshold value.
We do not want to spam our mails, so requirement is once cpu goes high send only one alert mail till it reaches to threshold value again.
And once it reaches below threshold trigger a mail saying it has come down.
Again if goes high then send an alert mail and no further mails till it falls down again , continue in this way.
Thanks !

0 Karma

FrankVl
Ultra Champion

The example I gave is indeed only the base of your search. As I already mentioned: from this you need to create 2 alert searches: one that filters for previous below 90 and current >90. And another alert search that filters for previous >90 and current <90.

PS: you might want to convert your "answer" into a comment to my answer 🙂

0 Karma

FrankVl
Ultra Champion

You should be able to create a search that returns servername, CPU usage over past 5 minutes, CPU usage over previous 5 minutes (10-5 min ago).

Then from that search create two flavors: one that filters out servers that had below 90 in previous and >90 in current period. And one that filters servers that had >90 in previous period and <90 in current. Then create alerts off of each of these.

Quick example based on CPU stats from introspection logs (replace that with your own source of CPU stats)

index="_introspection" sourcetype=splunk_resource_usage component=Hostwide earliest=-10m@m latest=-5m@m 
| stats avg(data.cpu_idle_pct) as previousCPU by host 
| eval previousCPU=100-previousCPU 
| append 
    [| search index="_introspection" sourcetype=splunk_resource_usage component=Hostwide earliest=-5m@m latest=now() 
    | stats avg(data.cpu_idle_pct) as currentCPU by host 
    | eval currentCPU=100-currentCPU
        ]
| stats max(previousCPU) as previousCPU max(currentCPU) as currentCPU by host
0 Karma

ankitagm
New Member

Ok Thanks, will try an let you know.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...