topic Re: Create a recurring alert based on message strings in Alerting

How to create a recurring alert based on message strings?

chaitanyaaiops — Tue, 26 Jul 2022 14:33:16 GMT

Dear experts,
I've created an alert based on a message string to identify closed connections . However, alert gets triggered only once although the problem doesn't get fixed until we bounce.

Looking for a query to have an recurring alert, until I see success message string as "*reconfigured with 'RabbitMQ' bean*" as the latest in comparison to the failed strings across all events.

Failed messages: *com.rabbitmq.client.ShutdownSignalException* OR "*"channel shutdown*"
Success message: "*reconfigured with 'RabbitMQ' bean*"

Current Alert query that occurs only once:

Re: Create a recurring alert based on message strings

richgalloway — Tue, 26 Jul 2022 13:29:32 GMT

Alerts are triggered each time the search criteria are met (unless throttled). If the shutdown event is only received once then the alert will only be triggered once. If you want the alert to repeat then the search must be written and scheduled to find the triggering event (or canceling event) each time it runs.

Re: Create a recurring alert based on message strings

chaitanyaaiops — Tue, 26 Jul 2022 13:56:57 GMT

Thank you Rich - however, I don't want to create noise of recurring alert unless there is a need. i.e., only if the reconfigured message is not the latest in comparison to other strings - i want alerts to recur

Re: Create a recurring alert based on message strings

richgalloway — Tue, 26 Jul 2022 15:31:52 GMT

That requirement can be included in the search.

index IN ("devcf","devsc") cf_org_name IN(xxxx,yyyy) cf_app_name=* "rabbit*" AND ("channel shutdown*" OR "*com.rabbitmq.client.ShutdownSignalException*" OR "*rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error*" OR "*reconfigured with 'RabbitMQ' bean*") | dedup <<field with message>> | where NOT match(<<field with message>>, "reconfigured with 'RabbitMQ' bean") | stats count by cf_app_name, cf_foundation

Re: Create a recurring alert based on message strings

chaitanyaaiops — Wed, 27 Jul 2022 13:30:21 GMT

Thank you once again Rich.
To add more details:
Failed condition comes as a different fields in the event compared to reconfigured which comes at a different position of the event. In short, if i extract this is how it would look

msg1 = "channel shutdown"
msg2 = "com.rabbitmq.client.ShutdownSignalException"
msg3 ="*rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error*"
msg4= "*reconfigured with 'RabbitMQ' bean*"

Alert should be kept triggering until msg 4 is latest in comparison to all other 3 messages irrespective of even it occurring only once.

Re: Create a recurring alert based on message strings

richgalloway — Wed, 27 Jul 2022 14:18:22 GMT

If the failure and success messages are in different fields, then we can use the coalesce function to combine them for dedup.

index IN ("devcf","devsc") cf_org_name IN(xxxx,yyyy) cf_app_name=* "rabbit*" AND ("channel shutdown*" OR "*com.rabbitmq.client.ShutdownSignalException*" OR "*rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error*" OR "*reconfigured with 'RabbitMQ' bean*") | eval alert_field = coalesce(<<msg1 field>>, <<msg2 field>>, <<msg3 field>>, <<msg4 field>>) | dedup alert_field | where NOT match(alert_field, "reconfigured with 'RabbitMQ' bean") | stats count by cf_app_name, cf_foundation

Re: Create a recurring alert based on message strings

chaitanyaaiops — Wed, 27 Jul 2022 16:42:06 GMT

Thanks Rich. However, the challenge being alert is set to run for every 15 mins and events occur only once. How can it recur every 15 mins since the failure event won't occur?

Thank you for patience

Re: Create a recurring alert based on message strings

richgalloway — Wed, 27 Jul 2022 18:43:53 GMT

Yes, that's the tricky bit and goes back to the part of my first reply that said "the search must be written and scheduled to find the triggering event". Rather than search back 15 minutes, it will be necessary for the alert to search back as far as necessary to find the events of interest.

Re: Create a recurring alert based on message strings

chaitanyaaiops — Thu, 28 Jul 2022 01:08:15 GMT

That's right - however, if i use dedup, and if the failed message has occurred after the success message, would that removes duplicates for failed messages

Only bit I don't get is, how do i compare timestamp for msg 4 (success) to be latest among all messages?

Re: Create a recurring alert based on message strings

richgalloway — Thu, 28 Jul 2022 12:11:46 GMT

There's no need to compare timestamps. The dedup command keeps the most recent event so whatever result you get must be the latest message.