Cant query triggered alerts from Splunk

[ridutta]

I have few alerts enabled in Splunk and they are added to triggered actions. Now today an alert triggered and its showing under the alert but i cant search the alert metadata using the following query:

index=_audit action="alert_fired"

App: Search and Reporting
Alert Configurede in the same app only. Am i doing someting wrong?

Thanks and Regards,
Rishav

[ridutta]

Got it what i wanted. Thanks all!!

[mayurr98]

hey @ridutta

This makes a really good use case for join since the alert threshold is hold in the savedsearch.conf and is accessible over the REST endpoint /services/saved/searches.
You will first need to get the alerts_threshold from REST and join it with the triggered alerts by the title:

index=_audit action="alert_fired" 
 | rename ss_name AS title 
 | join title [ | rest /services/saved/searches | table title, alert_threshold ] 
 | timechart values(alert_threshold) AS alert_threshold count by title

I hope this works for you!

[nikita_p · Accepted solution]

Hi @ridutta,
Can you try rest api search for this?
|rest /services/search/jobs | search author=admin AND id=*scheduler AND (label= OR )| stats values(earliestTime) AS Time BY label

[ridutta]

Thanks a lot Nikita.... This kind of worked and I can tweek further. But there are so many columns but I want the specific ones.... How to remove the rest?

[nikita_p]

You can remove rest using NOT label=

[p_gurav]

Hi ridutta,

Could you please tell what information you want regarding alert?

[ridutta]

So what i want is actually to fetch the triggered alert metadata and add it as a dashboard panel for the last 24 hours.
Alert Name and Triggered Time and count will do.