topic Re: scom: how to remove resolved events that are monitored using dedup in Splunk Search

Scom: How to remove resolved events that are monitored using dedup?

nathanluke86 — Wed, 30 Mar 2022 22:25:56 GMT

We are ingesting scom events

When an alert is triggered it is assigned an id (the earliest event pictured) and we have created a dashboard of alerts that are in status new.

This issue we have is some of the alerts have actually been resolved but the logs that show an alert as resolved show the id as "monitoringalertid" not "id" so the dedup "id" isn't working

We are having issues joining these alerts to get the latest status and remove alert if it has been solved.

The only value to match these events is the id/monitoringalertid.

Anyone know a way to match these events.

TIA

Re: scom: how to remove resolved events that are monitored using dedup

manjunathmeti — Thu, 25 Feb 2021 15:41:25 GMT

hi @nathanluke86,

You can use coalesce to set id=monitoringalertid where id is null.

| eval id=coalesce(id, monitoringalertid) | dedup id

More info on coalesce here: https://docs.splunk.com/Documentation/Splunk/8.1.2/SearchReference/ConditionalFunctions#coalesce.28X.2C....29

If this reply helps you, an upvote/like would be appreciated.

Re: scom: how to remove resolved events that are monitored using dedup

nathanluke86 — Fri, 26 Feb 2021 08:47:16 GMT

Thanks @manjunathmeti ,

Tried using coalesce but this doesn't seem to work.

I have noticed that the logs that have monitoringalertid field that matches the original id also have a field id which does not match the original id.

This might be causing the issue with coalesce.

Is there another way to match id with monitoringalertid and if latest status is closed then ignore.

TIA.

Re: scom: how to remove resolved events that are monitored using dedup

maciep — Fri, 26 Feb 2021 12:47:42 GMT

you may be able to just swap the order in the coalesce statement - so that the monitoring id will be used if it exists, then id would be checked for next.

Re: scom: how to remove resolved events that are monitored using dedup

nathanluke86 — Fri, 26 Feb 2021 15:19:22 GMT

Thanks @maciep @manjunathmeti ,

I have this working now using:

| eval logID = coalesce('monitoringalertid','id') |transaction logID.

Thanks for all the help

Re: scom: how to remove resolved events that are monitored using dedup

johnrbhancock — Wed, 30 Mar 2022 21:23:47 GMT

I have been brainstorming the same issue. The issue is when the alert is closed then reopened it wont recongnise the same event id as the id seems to change, so if it flaps youll get alert storms. My solution is to use the monitoringobjectid and the name together. The monitoringobjectid is unique to each object but can have multiple different alerts raised against it. So using the name you ensure you only get unique events for that object. This will help with Maintenance Mode and alert storms. Even with disk alerts that seem to continue raising events at certain intervals of disk usage have different ids.