Alerting

Alert doesn't work, but clone does?

vince88917
Explorer

I have an alert with a "Send email" trigger action when the number of results is greater than zero. The aim is to send a table of results inline in the email.

This isn't currently working - no email is being received when there are valid qualifying events in the search period (previous day).

The alert was deployed using the SHC deployer, and is owned by "nobody".

If I "Open in Search" I see results.

If I clone the alert so a local version is run under my user context, I get the alert email sent.

 

Looking into _internal events, I can see that when the "nobody" search runs, no results are returned, and hence no email is sent - this isn't an issue with email configuration. Why does this search in this context give me no results?

Labels (2)
0 Karma
1 Solution

vince88917
Explorer

For future reference, the alert did not specify an index so I rectified this. However, this did not solve the problem.

As mentioned, I could clone the alert and this would run fine. I could also then change the ownership of the cloned alert to "nobody" and it would also run fine. 

I wasn't content with this solution though, as centrally deployed configuration is strongly preferred to local changes in my environment.

I've ended up deploying from SHC Deployer, making a simple local change (i.e. disabling), then undoing this local change (so, re-enabling) in order that no significant local change continues to exist. Doing this, I find the alert runs fine.

Weird.

View solution in original post

vince88917
Explorer

For future reference, the alert did not specify an index so I rectified this. However, this did not solve the problem.

As mentioned, I could clone the alert and this would run fine. I could also then change the ownership of the cloned alert to "nobody" and it would also run fine. 

I wasn't content with this solution though, as centrally deployed configuration is strongly preferred to local changes in my environment.

I've ended up deploying from SHC Deployer, making a simple local change (i.e. disabling), then undoing this local change (so, re-enabling) in order that no significant local change continues to exist. Doing this, I find the alert runs fine.

Weird.

PickleRick
SplunkTrust
SplunkTrust

There are three different levels at which this could fail

1. The search could have not been run. You can check last runs of the alert-generating search in settings ->saved searches, reports and alerts. There you can find your alert and click on "view recent" or something like that to see when it was last run. If it wasn't run at all, there might have been a problem with the scheduler being unable to find resources or user exceeding quota. You can also inspect last jobs and see the job logs

2. The search could have been dispatched but might have not returned the relevant resukts. Here, as @richgalloway already pointed out, the user who owns the search might not be allowed to search for the events (most typically - index restrictions on user's role). In the last runs view you can see how many events were returned.

3. The search might have been run, returned results but the email wasn't sent. The user's role needs a list_settings privilege to be able to send email via predefined server.

richgalloway
SplunkTrust
SplunkTrust

Perhaps your role has access to the data being searched for, but nobody's role does not.  If so, consider creating a service account with the proper role and giving ownership of the alert to that account.

---
If this reply helps you, Karma would be appreciated.

vince88917
Explorer

It feels like this is the right area. I have many other alerts and reports in the same app with owner=nobody that run with no problem. How do I debug this one further?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Start by comparing your role to that of nobody.  Confirm each can access the same indexes.  If the alert doesn't specify an index (a Bad Practice) then also make sure each role has the same set of default indexes.

Do the same for any lookup tables and other KOs the alert might use.

Check the Job Inspector and search log for each query to see if they shed any light on the matter.

---
If this reply helps you, Karma would be appreciated.

vince88917
Explorer

How do I ascertain the details of the role of “nobody”?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Check in the settings -> roles. Or see the output of

| rest /services/authorization/roles

 

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...