Reducing Noise of Alerts

[ankitagm]

Hi,

I want to implement splunk alert in below way : Please let me know how it can achieve?

I am querying the CPU utilization in near real time i.e every 5 minutes

If CPU Utilization crosses 90% then trigger an alert mail/Log a JIRA And It should not create any further repetitive alert mail till it reaches its specified below threshold value (Here it should not depend on certain amount of time by specifying throttleing, but it should depend on the values i.e if it has come back to threshold trigger a mail saying CPU has come down now till then no further email alert should generate )

Thanks,

[ankitagm]

Hi

Thnx,Yes query shows previous and current cpu.
But how can i control the action of sending an email until cpu falls down to below threshold value.
We do not want to spam our mails, so requirement is once cpu goes high send only one alert mail till it reaches to threshold value again.
And once it reaches below threshold trigger a mail saying it has come down.
Again if goes high then send an alert mail and no further mails till it falls down again , continue in this way.
Thanks !

[FrankVl]

The example I gave is indeed only the base of your search. As I already mentioned: from this you need to create 2 alert searches: one that filters for previous below 90 and current >90. And another alert search that filters for previous >90 and current <90.

PS: you might want to convert your "answer" into a comment to my answer 🙂

[FrankVl · Accepted solution]

You should be able to create a search that returns servername, CPU usage over past 5 minutes, CPU usage over previous 5 minutes (10-5 min ago).

Then from that search create two flavors: one that filters out servers that had below 90 in previous and >90 in current period. And one that filters servers that had >90 in previous period and <90 in current. Then create alerts off of each of these.

Quick example based on CPU stats from introspection logs (replace that with your own source of CPU stats)

index="_introspection" sourcetype=splunk_resource_usage component=Hostwide earliest=-10m@m latest=-5m@m 
| stats avg(data.cpu_idle_pct) as previousCPU by host 
| eval previousCPU=100-previousCPU 
| append 
    [| search index="_introspection" sourcetype=splunk_resource_usage component=Hostwide earliest=-5m@m latest=now() 
    | stats avg(data.cpu_idle_pct) as currentCPU by host 
    | eval currentCPU=100-currentCPU
        ]
| stats max(previousCPU) as previousCPU max(currentCPU) as currentCPU by host

[ankitagm]

Ok Thanks, will try an let you know.