Scom: How to remove resolved events that are monitored using dedup?

[nathanluke86]

We are ingesting scom events

When an alert is triggered it is assigned an id (the earliest event pictured) and we have created a dashboard of alerts that are in status new.

This issue we have is some of the alerts have actually been resolved but the logs that show an alert as resolved show the id as "monitoringalertid" not "id" so the dedup "id" isn't working

We are having issues joining these alerts to get the latest status and remove alert if it has been solved.

The only value to match these events is the id/monitoringalertid.

Anyone know a way to match these events.

TIA

[johnrbhancock]

I have been brainstorming the same issue. The issue is when the alert is closed then reopened it wont recongnise the same event id as the id seems to change, so if it flaps youll get alert storms. My solution is to use the monitoringobjectid and the name together. The monitoringobjectid is unique to each object but can have multiple different alerts raised against it. So using the name you ensure you only get unique events for that object. This will help with Maintenance Mode and alert storms. Even with disk alerts that seem to continue raising events at certain intervals of disk usage have different ids.

[nathanluke86 · Accepted solution]

Thanks @maciep @manjunathmeti ,

I have this working now using:

| eval logID = coalesce('monitoringalertid','id') |transaction logID.

Thanks for all the help

[nathanluke86]

Thanks @manjunathmeti ,

Tried using coalesce but this doesn't seem to work.

I have noticed that the logs that have monitoringalertid field that matches the original id also have a field id which does not match the original id.

This might be causing the issue with coalesce.

Is there another way to match id with monitoringalertid and if latest status is closed then ignore.

TIA.

[maciep]

you may be able to just swap the order in the coalesce statement - so that the monitoring id will be used if it exists, then id would be checked for next.

[manjunathmeti]

hi @nathanluke86,

You can use coalesce to set id=monitoringalertid where id is null.

| eval id=coalesce(id, monitoringalertid)
| dedup id

 More info on coalesce here: https://docs.splunk.com/Documentation/Splunk/8.1.2/SearchReference/ConditionalFunctions#coalesce.28X...

If this reply helps you, an upvote/like would be appreciated.