How to search the names of triggered alerts, their trigger time, and the events that triggered them?

[LuiesCui]

Hi guys,

I have a problem with the triggered alerts and I really need your help!
Now, I have some alerts working great in my Splunk. To count the number of the alerts triggered in a period of time, I made a report with the search string like:

index=_audit action="alert_fired" | timechart count by ss_name

From this search string, I get the time and name of the triggered alerts as below:

But I also want to know the original events which triggered these alerts. All I want is a table with 3 columns. Column 1 shows the trigger time of the alerts, Column 2 shows the names of the triggered alerts, and Column 3 shows the events which triggered the alerts. How can I do that?

[MuS · Accepted solution]

Hi LuiesCui,

I must admit, this makes a really good use case for join since the alert threshold is hold in the savedsearch.conf and is accessible over the REST endpoint /services/saved/searches.
You will first need to get the alerts_threshold from REST and join it with the triggered alerts by the title:

index=_audit action="alert_fired" 
| rename ss_name AS title 
| join title [ | rest /services/saved/searches | table title, alert_threshold ] 
| timechart values(alert_threshold) AS alert_threshold count by title

Hope this helps to get you started ...

cheers, MuS

[iqbalintouch]

Hi MuS,

I have same issue and looking for resolution. I followed the same query but it didn't work for me. Can you please help me out.
FYI: I am using enterprise splunk, version 6.3.2
I was able to get the list of all the enabled alert from here: | rest /servicesNS/-/-/saved/searches

But I need to get the details of how many times the alert was triggered in particular time duration, what was the alert and what time(when) ?

Thank you!

[MuS]

I will check that we I have access to an instance that has alerts enabled. Just one thing for now, rest calls are ignoring time but I'm sure there is some information available to see when an alert was triggered.

[iqbalintouch]

Hi MuS ,

Yes, I am quite sure there must be a place where we capture the details of when the alert crossed the threshold value and an email was sent out. Using that value we can pull the data when this alert was triggered, how many times (let's say how many times in last 15 days) and whom the email notification was sent.

I use below query for all the enabled alerts:
| rest /servicesNS/-/-/saved/searches splunk_server=local | search disabled=0 actions=* is_scheduled=1

so using the above query can you please help me out here?

[MuS]

Okay, can you please explain what is not working for you? You need to have the admin role assigned to use the search posted in the original answer.

[iqbalintouch]

Hi MuS,
the query itself not returning means i is not giving any result:

index=_audit action="alert_fired"
| rename ss_name AS title
| join title [ | rest /services/saved/searches | table title, alert_threshold ]
| timechart values(alert_threshold) AS alert_threshold count by title

though I can get the desired output by this:
| rest /servicesNS/-/-/saved/searches splunk_server=local | search disabled=0 actions=* is_scheduled=1

so my request is simple I need to pull a report OR data where how many times an alert was triggered during a given time period, when it was triggered and whom the email was sent.

I do have power user access on splunk but not an admin level.

[MuS]

Well your request is indeed very simple, though there is not much I or anyone can do to help. Since you have the power user role and not admin you cannot access the index _audit and therefore will not be able to get the search working.

Ask your friendly Splunk admin to create a report for you or to schedule a saved search which feeds into a summary index so you can run the reports on the summary index.

cheers, MuS

[iqbalintouch]

Hi MuS,

Got it, thank you 🙂

[LuiesCui]

Hi, thank you for your reply! I kind of worked this out but got a new problem. Could you help me with this please? Thank you again. http://answers.splunk.com/answers/305369/why-the-results-from-triggered-alert-is-different.html