topic Re: Trigger specific custom alert in Alerting

Trigger specific custom alert

alekseisaiko — Wed, 30 Sep 2020 04:23:46 GMT

Hi there!
I'm using this query

index="dev" |eval raw_len=len(_raw) | eval raw_len_gb = raw_len/1024/1024/1024 | stats sum(raw_len_gb) as GB by kubernetes_namespace | bin _time span=1d

To get the amount of received data from K8S logs, and I want to trigger alert using this query, if the amount will be more than 0.5 gig a day.
How to define it in alert?
Right now, when I'm defining alert that triggers when - "eval raw_len_gb > 0.5", it's not triggered

Thanks,

ALeksei

Re: Trigger specific custom alert

richgalloway — Tue, 03 Mar 2020 13:28:07 GMT

Add | where raw_len_gb > 0.5 to the end of your query and change the alert to trigger if the number of results is not zero.

Re: Trigger specific custom alert

alekseisaiko — Tue, 03 Mar 2020 14:11:26 GMT

Thanks for you answer!
Right now, I want to test that this query works.

If I'm running my query (without the | where raw_len_gb > 0.5) for last 15 mins, I'm getting results, with 0.029 GB for example in one of the results.
But if I want to check the same query with | where raw_len_gb > 0.01 just to see that it will show results, it doesn't. Though it must show the one with 0.029 GB and more.

Re: Trigger specific custom alert

alekseisaiko — Tue, 03 Mar 2020 14:58:34 GMT

Ok, it worked liked this -
index="dev" |eval raw_len=len(_raw) | eval raw_len_gb = raw_len/1024/1024/1024 | stats sum(raw_len_gb) as GB by kubernetes_namespace | bin _time span=1d | where GB > 0.0004
And I'm getting relevant results now, but still , trigger is not working from some reason

Re: Trigger specific custom alert

alekseisaiko — Tue, 03 Mar 2020 15:03:02 GMT

My bad, it worked! 🙂
Thanks