Splunk Search

Scom: How to remove resolved events that are monitored using dedup?

nathanluke86
Communicator

We are ingesting scom events

When an alert is triggered it is assigned an id (the earliest event pictured) and we have created a dashboard of alerts that are in status new.

This issue we have is some of the alerts have actually been resolved but the logs that show an alert as resolved show the id as "monitoringalertid" not "id" so the dedup "id" isn't working

We are having issues joining these alerts to get the latest status and remove alert if it has been solved.

The only value to match these events is the id/monitoringalertid.

Anyone know a way to match these events.

TIA

issue.png

Labels (5)
Tags (2)
0 Karma
1 Solution

nathanluke86
Communicator

Thanks @maciep @manjunathmeti ,

 

I have this working now using:

| eval logID = coalesce('monitoringalertid','id') |transaction logID.

 

Thanks for all the help

View solution in original post

0 Karma

johnrbhancock
Engager

I have been brainstorming the same issue. The issue is when the alert is closed then reopened it wont recongnise the same event id as the id seems to change, so if it flaps youll get alert storms. My solution is to use the monitoringobjectid and the name together. The monitoringobjectid is unique to each object but can have multiple different alerts raised against it. So using the name you ensure you only get unique events for that object. This will help with Maintenance Mode and alert storms. Even with disk alerts that seem to continue raising events at certain intervals of disk usage have different ids.

0 Karma

nathanluke86
Communicator

Thanks @maciep @manjunathmeti ,

 

I have this working now using:

| eval logID = coalesce('monitoringalertid','id') |transaction logID.

 

Thanks for all the help

0 Karma

nathanluke86
Communicator

Thanks @manjunathmeti ,

Tried using coalesce but this doesn't seem to work.

I have noticed that the logs that have monitoringalertid field that matches the original id also have a field id which does not match the original id.

This might be causing the issue with coalesce.

Is there another way to match id with monitoringalertid and if latest status is closed then ignore.

TIA.

 

0 Karma

maciep
Champion

you may be able to just swap the order in the coalesce statement - so that the monitoring id will be used if it exists, then id would be checked for next.

manjunathmeti
SplunkTrust
SplunkTrust

hi @nathanluke86,

You can use coalesce to set id=monitoringalertid where id is null.

| eval id=coalesce(id, monitoringalertid)
| dedup id

 More info on coalesce here: https://docs.splunk.com/Documentation/Splunk/8.1.2/SearchReference/ConditionalFunctions#coalesce.28X...

If this reply helps you, an upvote/like would be appreciated.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...