topic Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them? in Splunk Search

How to search the names of triggered alerts, their trigger time, and the events that triggered them?

LuiesCui — Sun, 06 Sep 2015 04:25:58 GMT

Hi guys,

I have a problem with the triggered alerts and I really need your help!
Now, I have some alerts working great in my Splunk. To count the number of the alerts triggered in a period of time, I made a report with the search string like:

index=_audit action="alert_fired" | timechart count by ss_name

From this search string, I get the time and name of the triggered alerts as below:

But I also want to know the original events which triggered these alerts. All I want is a table with 3 columns. Column 1 shows the trigger time of the alerts, Column 2 shows the names of the triggered alerts, and Column 3 shows the events which triggered the alerts. How can I do that?

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

MuS — Sun, 06 Sep 2015 23:29:34 GMT

Hi LuiesCui,

I must admit, this makes a really good use case for join since the alert threshold is hold in the savedsearch.conf and is accessible over the REST endpoint /services/saved/searches.
You will first need to get the alerts_threshold from REST and join it with the triggered alerts by the title:

index=_audit action="alert_fired" 
| rename ss_name AS title 
| join title [ | rest /services/saved/searches | table title, alert_threshold ] 
| timechart values(alert_threshold) AS alert_threshold count by title

Hope this helps to get you started ...

cheers, MuS

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

LuiesCui — Mon, 07 Sep 2015 10:37:57 GMT

Hi, thank you for your reply! I kind of worked this out but got a new problem. Could you help me with this please? Thank you again. http://answers.splunk.com/answers/305369/why-the-results-from-triggered-alert-is-different.html

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

iqbalintouch — Tue, 26 Sep 2017 11:54:51 GMT

Hi MuS,

I have same issue and looking for resolution. I followed the same query but it didn't work for me. Can you please help me out.
FYI: I am using enterprise splunk, version 6.3.2
I was able to get the list of all the enabled alert from here: | rest /servicesNS/-/-/saved/searches

But I need to get the details of how many times the alert was triggered in particular time duration, what was the alert and what time(when) ?

Thank you!

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

MuS — Thu, 28 Sep 2017 01:28:00 GMT

I will check that we I have access to an instance that has alerts enabled. Just one thing for now, rest calls are ignoring time but I'm sure there is some information available to see when an alert was triggered.

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

iqbalintouch — Tue, 29 Sep 2020 16:03:48 GMT

Hi MuS ,

Yes, I am quite sure there must be a place where we capture the details of when the alert crossed the threshold value and an email was sent out. Using that value we can pull the data when this alert was triggered, how many times (let's say how many times in last 15 days) and whom the email notification was sent.

I use below query for all the enabled alerts:
| rest /servicesNS/-/-/saved/searches splunk_server=local | search disabled=0 actions=* is_scheduled=1

so using the above query can you please help me out here?

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

MuS — Mon, 02 Oct 2017 19:01:53 GMT

Okay, can you please explain what is not working for you? You need to have the admin role assigned to use the search posted in the original answer.

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

iqbalintouch — Tue, 29 Sep 2020 16:04:29 GMT

Hi MuS,
the query itself not returning means i is not giving any result:

though I can get the desired output by this:
| rest /servicesNS/-/-/saved/searches splunk_server=local | search disabled=0 actions=* is_scheduled=1

so my request is simple I need to pull a report OR data where how many times an alert was triggered during a given time period, when it was triggered and whom the email was sent.

I do have power user access on splunk but not an admin level.

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

MuS — Tue, 03 Oct 2017 18:57:14 GMT

Well your request is indeed very simple, though there is not much I or anyone can do to help. Since you have the power user role and not admin you cannot access the index _audit and therefore will not be able to get the search working.

Ask your friendly Splunk admin to create a report for you or to schedule a saved search which feeds into a summary index so you can run the reports on the summary index.

cheers, MuS

Re: How to search the names of triggered alerts, their trigger time, and the events that triggered them?

iqbalintouch — Wed, 04 Oct 2017 04:05:23 GMT

Hi MuS,

Got it, thank you 🙂