Alerting

Alerts falsely show up as reports

zapping575
Explorer

Hi,

I have a bunch of alerts in my savedsearches.conf.

I would like to configure the alert action "Add to triggered alerts" (as is offered when you add the alert using the ui). I am doing this programmatically.

After restarting splunk, the alerts do not show up as alerts, but rather as reports (in the reports tab). Is this intended behaviour by splunk or am I missing out on something?

An example alert can be found below

 

 

[generic-alert-name]                                                                                 
alert.expires = 120d                                                                                                   
alert.severity = 2                                                                                                     
alert.suppress = 0                                                                                                     
alert.track = 1                                                                                                        
counttype = number of events                                                                                           
cron_schedule = * * * * *                                                                                              
description = 
dispatch.earliest_time = rt-30d                                                                                        
dispatch.latest_time = rt-0d                                                                                           
display.general.type = statistics                                                                                      
display.page.search.tab = statistics                                                                                   
enablesched = 1                                                                                                        
quantity = 0                                                                                                           
relation = greater than                                                                                                
request.ui_dispatch_app = my_app                                                                     
request.ui_dispatch_view = my_app
search = eventtype = "some-eventtype" | stats count by id | search count >= 4711 

 

 

 

Labels (1)
Tags (2)
0 Karma
1 Solution

zapping575
Explorer

So I was using the exact same name for my alerts as I was using for the eventtypes that were used to generate them.
Whats more is that because of the large number of alerts, splunk stated the following:

The number of search artifacts in the dispatch directory is higher than recommended

Thus I changed the alert type from real-type to planned.

They are now appearing in the Alerts section as expected.

Thanks @richgalloway for the help.

 

View solution in original post

0 Karma

zapping575
Explorer

So I was using the exact same name for my alerts as I was using for the eventtypes that were used to generate them.
Whats more is that because of the large number of alerts, splunk stated the following:

The number of search artifacts in the dispatch directory is higher than recommended

Thus I changed the alert type from real-type to planned.

They are now appearing in the Alerts section as expected.

Thanks @richgalloway for the help.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Compare your "alert" to one created using the UI and the difference should be clear.

---
If this reply helps you, Karma would be appreciated.
0 Karma

zapping575
Explorer

Thank you for the response.

I did what you suggested (in a separate app for testing).

The entry in savedsearches that you can find below will show up as an alert. However, I can see no difference to the entry in my first post.

[test1]
                                                                                                                       alert.expires = 120d                                                                                                                       alert.suppress = 0                                                                                                                       alert.track = 1                                                                                                                      counttype = number of events                                                                                                                       cron_schedule = * * * * *                                                                                                                       description = test1                                                                                                                       dispatch.earliest_time = rt-30d                                                                                                                       dispatch.latest_time = rt-0d                                                                                                                       display.general.type = statistics                                                                                                                       display.page.search.tab = statistics                                                                                                                       enableSched = 1                                                                                                                       quantity = 0                                                                                                                       relation = greater than                                                                                                                       request.ui_dispatch_app = test1                                                                                                                       request.ui_dispatch_view = search                                                                                                                       search = index="_internal" | stats count by action, host | search count > 1

 I did also compare the default.meta files. They are identical, so permissions shouldnt be an issue.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

In my experience, the counttype field makes the difference.  For reports, counttype is "always" (the default).

---
If this reply helps you, Karma would be appreciated.
0 Karma

zapping575
Explorer

Thank you very much for the tip.

For a single alert in savedsearches.conf, I changed counttype to always and restarted splunk.

Unfortunately, the selected alert still doesnt show up where it should.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If counttype is "always" then the saved search is a report, not an alert.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...