Sendemail does not work if selected in the Alert config, but sendemail does work!

[apietersen]

Created a supportticket:

Sendemail does not work if selected and set in the Alert config. But Sendemail function is working OK!? 

Business Impact: Can not respond on any "System_down/System_offline" situation . Happens not very often but very critical to respond to.

Product Version : 9.2.0.1 / I assume that it might not work since Splunk Enterprise v9.1.2 either (not sure)

Area: Search/Index - Splunk Enterprise

Deployment Type: On Prem / Small instance with only indexer, kV Search-head active 
OS:  Windows 2019 server

When did you first notice the issue? Somewhere 1/26/2024 (noticed a system_down situation on dashboard but was not notified by email)

Did you make any changes recently?: I upgrade last week to v9.2.0.1 on our test server. Later I found that our production server (v9.1.2) has the same issue

Steps Reproduce: You can created an Alert_Trigger_Test: (zie code below)

| makeresults
| eval ATT=4
| stats max(ATT) as mincount

Then test it every 5 minutes (cron-schedule) by: search mincount < 7

======================Alert Code : in savedsearch.conf (.../search/local) =====================

[Alert_trigger_1v1]
action.email = 1
action.email.cc = <your@email_address_2>
action.email.include.search = 1
action.email.inline = 1
action.email.priority = 2
action.email.sendresults = 1
action.email.to = <your@email_address_1>
action.email.useNSSubject = 1
action.lookup = 0
action.lookup.append = 1
action.lookup.filename = alerttrigger.csv
alert.digest_mode = 0
alert.expires = 1h
alert.suppress = 0
alert.track = 1
alert_condition = search mincount >7
allow_skew = 5m
counttype = custom
cron_schedule = */5 * * * *
dispatch.earliest_time = -5m
dispatch.latest_time = now
display.general.type = statistics
display.page.search.mode = verbose
display.page.search.tab = statistics
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = | makeresults \
| eval ATT=3\
| stats max(ATT) as mincount

=======================END Code ===================

Is there any one else suffering from the same issues?

regards AshleyP

[apietersen]

Hi PickleRick,

Thanks, 

1) Yes, I have tried many thing, also by using the opposite condition: alert_condition = search mincount < 7

2) In all the logs I have checked (including sendemail.py, which does not log anything in my opinion) and searched, it is/was successful, status: success, not skipped or otherwise found to be delayed or ???

3) example spl: 
index=_* AND alert_actions="email"
| stats count by status
Output: 42 Success over last 24h 

02-17-2024 12:12:17.264 +0100 INFO SavedSplunker - savedsearch_id="nobody;search;Alert_trigger_1v1", search_type="scheduled", search_streaming=0, user="admin", app="search", savedsearch_name="Alert_trigger_1v1", priority=default, status=success, digest_mode=0, durable_cursor=0, scheduled_time=1708168200, window_time=0, dispatch_time=1708168333, run_time=0.142, result_count=1, alert_actions="email", sid="scheduler__admin__search__RMD5ea1ed26b5154d33f_at_1708168200_47", suppressed=0, fired=1, skipped=0, action_time_ms=3528, thread_id="AlertNotifierWorker-0", message="", workload_pool=""

So no error message to be found sofar?? 

What do I miss here. Although I also searched in python.log and I found part of many messages:

2024-02-17 12:32:17,265 +0100 ERROR sendemail:572 - (554, b'5.2.252 SendAsDenied; ticket@eremote.nl not allowed to send as Splunk_eRemote@uBDC01; STOREDRV.Submission.Exception:SendAsDeniedException.MapiExceptionSendAsDenied; Failed to process message due to a permanent exception with message [BeginDiagnosticData]Cannot submit message.

....>>

Splunk_eRemote@uBDC01;  Splunk_eRemote is a textfield in SMPT-settings. Python code seems to append "@uBDC01" to it (is the hostname of my testserver ???)

I  can not make any sense out of it (same setting used in past 6 year?!) on our production server
Again: Sendemail is working fine, in dashboard-search (form) and when using it for testing by running manual SPL code in searches

Any thoughts?

regards AshleyP

[PickleRick]

This is a message returned by the other end of the smtp transaction so it can contain anything. Next thing I'd check the smtp server's logs for the reason for rejection because it's the mail server that's not accepting messages from you.

[apietersen]

Yes it does when using:

| makeresults
| sendemail to="aaa@ccc", from="bbb@eee.nl", subject="Testing Email from Splunk / from: uBDC01", use_ssl=false, use_tls=true

or as form in a Dashboard we use to create a ticket for our customer. Only alerting has stop working recently?

BTW: we use OFFICE365 so this should be occuring elsewhere too, do not you think?.
And no errors-essaging is showing in Splunk (as user or as admin) when this happens.

So you suggest to look in the O365 log of the exchange environment?

[PickleRick]

The sendemail.py is apparently trying to send but the server is rejecting the email. It's easiest to check at the denying party's logs why it's happening. If really nothing has changed recently on either side (are you absolutely sure there was no change in policies in the mail environment?), maybe it's simply a case of over-quota recipient mailbox.

[apietersen]

I have already send a support request to MS Office365. Hope that they can give a better or more detailed reason then "SendAsDenied; ticket@eremote.nl not allowed to send as Splunk_eRemote@uBDC01;"

In the mean time I will follow also my Splunk support-case route...

Will post my findings here, later....

[apietersen]

Update:

Had a session today with O365 support about message in python.log: (posted before)

SendAsDenied; ticket@eremote.nl not allowed to send as Splunk_eRemote@uBDC01;"

Answer: As discussed on the call, the bounceback (NDR report) you received shows that your office 365 account (ticket@eremote.nl) is not allowed to send email as Splunk_eRemote@uBDC01. This shows that there is a setting in Splunk that is preventing this action. And like I mentioned on the call, from Microsoft 365 perspective, email relayed is allowed and that is why you can send normal email from the application, and it delivers.

De 'Splunk_eRemote@uBDC01'  comes from the SMTP-server settings in Splunk:

Note:

1) We use O365 and some 4-5 months ago we stepped over from a personal account to a shared account (but did not notice the defect in the Alerting_email_function, just last week :-(. As Sendemail was working in Dasboards and in SPL code as expected.

2) We now use a shared mailbox under O365, which means you can not use any Alias. Before we could.

3) Using my personal account (with Alias's) is 2FA and is not possible according to O365 support.

4) Last week tried many things -including different fields A en B, without succes. Also tested it with use of a GMAIL account, no succes either.

5) Yesterday (Sunday) rebooted our W2029 server as part of our weekly maintenance schedule, 

Today I found out:

Note: Field A (username) must be same as SendAS (in the past we used only the word "Splunk" in field B)
Also note that keeping the B feidl empty it does a discovery and comes up with "Splunk@uBDC01"  so it appears. 

Now it is working again (in test) with field-A = field-B! (what is de use? apperantly only working in combination with personal-account and proper Alias, I conclude)

I will close this post now and thank you for the repsonse PickeRick :-)!

AshleyP

[PickleRick]

What I would start with when debugging such problems would be narrowing down the source of the problem. If you have an alert based on a search which obviously always matches (which is not obvious because you wrote that it should trigger when the value is less than 7 and your .conf dump shows a condition for higher than 7), I'd start by verifying:

1. If the search is indeed run at all? (or maybe your environment has some problems scheduling the search and skips the searches or delays them significantly)

2. If the search is indeed being run, is the alert action being dispatched.

3. If the action is being dispatched - do you see any errors from the sendemail.py in _internal.