Splunk Search

SPlunk not sending alerts

SS1
Loves-to-Learn Everything

Hi,

My splunk instance is not sending email alerts for a new alert th Can soat i just setup. I am getting other alert emails from the same splunk instance but the new alert isn't sending the alerts although it generates a stats table. I have set the alert to trigger per result. Same alert trigger condition works for other results. Any help is appreciated.

Labels (5)
0 Karma

SS1
Loves-to-Learn Everything

Hi,

@ITWhisperer 

We are running on 8.0.2

Below is the non working alert (working alert setup is same but the query doesnt have 'eval' )

Below is the query (it is generating stats table) ,

(sourcetype="AppV-User" *PUT /package*) OR (sourcetype=sql_appv_latest)
| rex "\/packages\/\w+\/(?<Id>\w+)\s+-\s+\d+\s+(?<User>[^ ]+)"
| eval Id=coalesce(Id, ID)
| stats dc(sourcetype) AS dc_sourcetype values(PkgName) as PkgName values(user) as user values(Status) as Status by Id
| where dc_sourcetype=2

ALert Setup,

 

SS1_1-1610693924845.png

 

0 Karma

ITWhisperer
Super Champion

Have you tried other trigger conditions such as number of results greater than 0?

0 Karma

ITWhisperer
Super Champion

Can you share the alerts, particularly, how the queries and triggers are set up for working and non-working alerts? Also, which version of splunk are you using? We have had issues with custom triggers not firing correctly so perhaps it is related to that, but seeing what you have working and not working would be helpful.

0 Karma

burwell
SplunkTrust
SplunkTrust

Hello. This sounds like the bug  we hit in 8.0.2

When we had an alert and then used custom criteria for alerting, if we had.. 

 

search count > 0

Splunk stored search count &gt; 0 in the savedsearches.conf and then the alert never fired.

This bug was fixed in 8.0.5: https://docs.splunk.com/Documentation/Splunk/8.0.5/ReleaseNotes/Fixedissues
SPL-189917

If you do an advanced edit, I believe you can see the bug. 

If this is your case, the solution would be to upgrade to 8.0.8.

I would not recommend 8.0.5 as it has this nasty bug 

2020-07-15SPL-192057, SPL-188608Realtime and in-progress adhoc searches shows "Job terminated unexpectedly" on members of SHC other than the SH from which the search originated

 

We upgraded to 8.0.2 then 8.0.5 and then 8.0.6.

0 Karma