Alerting

How to create and trigger an alert when I receive a trap for a specific OID that wasn't received for the past 24 hours?

noybin
Communicator

Hello,

I am receiving SNMP traps. I need to create an alert when I receive a trap for a specific OID (.1.3.6.1.4.1.2620.1.3000.5.2.1), but only If I didn't receive any traps for that OID for a day.

I am receiving some traps very frequently (every 20 seconds), but I don't want the alert to be triggered every 20 seconds. I want to trigger the alert if I receive a trap that didn't come for 24 hours.

I.E:
I can search the traps with the following: index="snmp-trap-cust" snmp_index=.1.3.6.1.4.1.2620.1.3000.5.2.1

How can I achieve this?

Thank you very much.

0 Karma

woodcock
Esteemed Legend

Schedule your alert for something like every hour with a time frame from -24h@h to @hand trigger when # of events is equal to 1:

index="snmp-trap-cust" snmp_index=.1.3.6.1.4.1.2620.1.3000.5.2.1
0 Karma

noybin
Communicator

If more than one trap is been triggered when the alert is evaluating the conditions, It won't trigger.

Thanks again!!

0 Karma

woodcock
Esteemed Legend

Are you saying that this will work or that there is still a problem?

0 Karma

noybin
Communicator

There's still a problem. Because I can not be sure that when the alert condition is evaluated, only 1 trap has been received. Maybe I received more then one and I still need to trigger the alert.

Thank you.

0 Karma

woodcock
Esteemed Legend

Only if the two are separated by the hour boundary, which I admit is possible. Now you are talking about a real-time alert based on a 24-hour window. This is VERY DANGEROUS to your cluster but you can set the timerange from rt-24h to rt. You will probably need to throttle this to 1 notification in 24-hours, but I am not sure as I do not run realtime alerts.

0 Karma

noybin
Communicator

I was thinkning on non rt alerts. But scheduled every 5 minutes instead.
In that case if more than one trap arrives during those five minutes, # of events will be greater than 1

0 Karma

woodcock
Esteemed Legend

Yes, it is a question of granularity of concern for consecutive searches. You have a solution, now you just need to make a decision on your risk/granularity.

0 Karma

somesoni2
Revered Legend

Try like this

index="snmp-trap-cust" snmp_index=.1.3.6.1.4.1.2620.1.3000.5.2.1 earliest=-25h latest=@m | streamstats current=f window=1 values(_time) as prev_time | where prev_time-_time>86400
0 Karma

woodcock
Esteemed Legend

When you save your search as an Alert, there is a throttling mechanism that you can set for "Do not trigger again until 24 hours later".

0 Karma

noybin
Communicator

Hi,

But I don't want to supress the alert for 24 hours.
I want to trigger the alert ONLY if there's been 24 hours without an event with the trap appears.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...