Alerting

How to produce an alert invocations report?

danielbb
Motivator

We are not sure what's going on with our cyber alerts and @gcusello assisted at Is there a way to inspect an alert?

Is there a way to produce an alerts invocations report? A report that would show how many times each alert was fired.

alt text

In the Searches, Reports, and Alerts page, we see the Alerts count - for which time period is it?

Tags (2)
0 Karma
1 Solution

jacobpevans
Motivator

Based on your screenshot, these are saved as triggered alerts (since the Alerts column is not zero). This query will identify all triggered alerts:

index=_audit action=alert_fired

It will also give you the fields ss_app and ss_name (name of the Alert) if that is useful for you.

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.

View solution in original post

jacobpevans
Motivator

Based on your screenshot, these are saved as triggered alerts (since the Alerts column is not zero). This query will identify all triggered alerts:

index=_audit action=alert_fired

It will also give you the fields ss_app and ss_name (name of the Alert) if that is useful for you.

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.

danielbb
Motivator

Thank you @jacobevans.

Using your query I found out that there were 91 invocations of a certain alert during a two hour span.

The alert type is Real-time
The Trigger alert when is Per-Result
and Throttle is not checked out

I wonder what we do wrong here. I guess throttling should help - Throttle alerts

However, on 7/30 we had 91 such alerts. I searched the data for that day using the query from the alert and only 8 events came back.

It doesn't make much sense....

0 Karma

jacobpevans
Motivator

I'm guessing here, but I think you're seeing 91 emails (that's what you mean by alerts?), but there were only 8 events because it was only triggered 8 times, but there were many more emails due to the per-result part of the alert.

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

danielbb
Motivator

I opened a bug report with Support. You see, the moment I changed it to index_earliest=-15m _index_latest=now index=your index | rest of the stuff on a 15 minute cron job, It works perfectly fine.

At Why are we getting excessive number of alerts?

0 Karma

danielbb
Motivator
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...