Why is Microsoft Office 365 Reporting Add-on for Splunk not pulling data and exiting with 403 Client Error?

[freddy_Guo]

I have installed Microsoft Office 365 Reporting Add-on for Splunk and configured with AD app with correct permission. But it keeps quite with 403. Below is the error that we are getting from /opt/splunk/var/log/splunk/ta_ms_o365_reporting_ms_o365_message_trace_oauth.log

2022-08-15 14:38:06,042 ERROR pid=17034 tid=MainThread file=base_modinput.py:log_error:316 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/lib/splunktaucclib/modinput_wrapper/base_modinput.py", line 140, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace_oauth.py", line 355, in collect_events
    get_events_continuous(helper, ew)
  File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace_oauth.py", line 96, in get_events_continuous
    message_response = get_messages(helper, microsoft_trace_url)
  File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace_oauth.py", line 74, in get_messages
    raise e
  File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace_oauth.py", line 66, in get_messages
    r.raise_for_status()
  File "/opt/splunk/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error:  for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2022-08-10T14:38:05.092475Z'%20and%20EndDate%20eq%20datetime'2022-08-10T15:38:05.092475Z'

[jconger · Accepted solution]

403 is a permissions error code.  Did you add the Azure AD app registration to the Azure AD Exchange Administrator role?

Here is a link to the Microsoft documentation about assigning the role => https://docs.microsoft.com/azure/active-directory/roles/manage-roles-portal

Also, here is a cheat sheet for add-on permissions => http://bit.ly/Splunk_Azure_Permissions 

[henrikh]

@jconger Have you definitively confirmed with Microsoft that the Exchange Administrator role is 100% required for this? Exchange Administrator is a fairly highly privileged role, and it seems absurd to be casually handing out such a role to an app registration that is only used to fetch Message Trace report.

[jconger]

Update: the originally required permissions were either Global Administrator or Exchange Administrator.  However, Microsoft has changed that to now allow the Global Reader role.

[x3ncrypt]

Hi jconger, would it be possible for me to reach out to you via email? Is there a way I can contact you directly? I am experiencing the same issue and require some assistance. Cheers!

[henrikh]

Thanks for the update! I suppose Global Reader is an improvement. Hopefully they will add a more appropriate role (or proper service principal permissions) in the future. (Or even better: a new API for Reporting/MessageTrace!)

[freddy_Guo]

That will be the dream 

[freddy_Guo]

Hi guys,

Thank you so much for the help so far! That was the discussion I had with my internal team yesterday as well.

My understanding is that we only grant Exchange Admin role to the Azure AD app, then the App has minimum advantage to check message trace report. So it's not as scary as granting Exchange Admin to the Add-On so it can do everything. 

Please correct me if I'm wrong. 

[freddy_Guo]

Hi guys,

I have assigned the app Exchange Administrator role and the log now is coming.

[VatsalJagani]

@freddy_Guo - The account does not have enough permission to access the email tracing.

Here I'm reading a guide about permission:

• The account you use to access the reports must have administrative permissions in the Office 365 organization. This report requires the user to be assigned to the View-Only Recipients role.
• If using new OAuth (added on 1st August 2022)
  • Exchange Administrator

Read about required permissions here - https://splunkbase.splunk.com/app/3720/#/details

I hope this helps!!!

[freddy_Guo]

Thank you so much for the response. 

I shall give a try on this one today. Just like I replied the thread above. I need to explain to our internal team that the exchange admin is only granted the Azure AD app, not to the Splunk Add-on itself. 

[freddy_Guo]

@jconger  Hi Jason, I have been reading all your answers about this TA. It would be wonderful if you could please point me to the right direction. Much appreciated.