I'm getting the same results as well - the official Splunk version of this ingest app is working for me. I didn't adjust anything or update to the just-released newer version (4.3) - I have 4.2.1 installed. A brief test in my non-production worked and pulled in a bunch of trace logs. ... View more

I got an update today on the ticket I have open: Thank you for your patience while we have been working to resolve the issue you reported. We would like to assure you that our engineering team and Microsoft team have been conducting a thorough investigation into the problem. We are pleased to inform you that the MS team have successfully reproduced the issue locally and is currently implementing patches to prevent reported issues. As per the latest update from the Microsoft side, They have a new patch being tested and will be rolled out today or the day by tomorrow to Production. ETA for this fix is to be available very soon. If there are any delays, we will update you as soon as possible. We kindly request your patience as we work to implement this fix. Rest assured that we will update you as soon as we have further information from our engineering team. Thank you for your understanding and support.   In addition, my sales engineer indicated that they are having potential success by rolling back to the updated, OAuth version of the Microsoft add-on versus using the Splunk add-on: https://splunkbase.splunk.com/app/3720 instead of https://splunkbase.splunk.com/app/4055 3720 actually indicates that users should migrate to 4055, but perhaps that's bad advice at the moment. I'm optimistic that they'll fix the 4055 add-on as well though. For now, I'm going to test version 2.0.1 of that first add-on link and I'll report back with my findings. ... View more

By way of a further update - I logged a case with Splunk support today and got the following response:   Thank you for submitting the case. We are aware of this issue and I want to let you know that we have received many cases of the same issue from other customers as well. We have reproduced & encountered the same error and suspect an issue with the API, not with the add-on. I would request that you allow us some time to validate the issue from the Microsoft Azure end to know the API behaviour. In order to expedite the case, we have escalated the issue to our internal team, and the add-on engineering team has started the conversation with Microsoft about the 401 client error (message trace failure). Rest assured that I will keep you informed of any further updates on this matter. I have also associated the internal Jira ticket for this issue with your support case so now even your account owner can check the status for any update from our internal add-on engineering team regarding this. If you have any other questions, kindly let me know. ... View more

They did, the only direct reference they provided was a Jira case: ADDON-61818 ... View more

Update from me - I just met with my Splunk support team for something else and made a mention about this. Apparently Splunk is aware and has an internal Jira open. It would appear MS changed something on their side that broke Splunk's TA. I'm going to open a case with Splunk anyway to help with visibility, but this might be a case of having to wait for the two companies to sort it out and fix it >.< ... View more

My messaging team said that MS got back to us and the issue is the deprecation of Basic Auth in our tenant. We didn't get any heads up that I was aware of, so it took me by surprise. Unfortunately I've since installed the Splunk Add-on for Microsoft Office 365 (https://splunkbase.splunk.com/app/4055) and it doesn't work. Using an Azure application with OAuth I still get a 401 error. I've tested the exact same credentials with Postman and they work fine and message trace logs download, so there's something else going on. I have a little more troubleshooting to do and then I'll open a ticket with Splunk as well... ... View more

I saw the same thing yesterday when I went to reference the status. My organization has a ticket open with Microsoft and our O365 team forwarded me the update from a Microsoft engineer that they could replicate the issue. If our ticket bears fruit and I have a worthwhile update for everyone I'll post it here. ... View more

@LukaszZeszko  - are you currently consuming message trace logs successfully? I implemented this add-on and sorted out my permissions issues and I still get the same 401 error that I got with basic auth: Basic auth: 2023-04-10 08:53:50,836 DEBUG pid=2504 tid=MainThread file=connectionpool.py:_make_request:437 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-03-31T15:51:52.522026Z'%20and%20EndDate%20eq%20datetime'2023-03-31T16:51:52.522026Z' HTTP/1.1" 401 1293 2023-04-10 08:53:50,838 ERROR pid=2504 tid=MainThread file=base_modinput.py:log_error:309 | HTTP Request error: 401 Client Error: for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-03-31T15:51:52.522026Z'%20and%20EndDate%20eq%20datetime'2023-03-31T16:51:52.522026Z' 2023-04-10 08:53:50,839 ERROR pid=2504 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events. OAuth: 2023-04-10 16:36:55,126 level=DEBUG pid=22685 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_process_messages:298 | datainput=b'XXXX_message_trace' start_time=1681159011 | message="nextLink URL (@odata.nextLink): https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-09T11%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-09T12%3A00%3A00Z'&$skiptoken=1999" 2023-04-10 16:36:55,424 level=ERROR pid=22685 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_get_messages:253 | datainput=b'XXXX_message_trace' start_time=1681159011 | message="HTTP Request error: 401 Client Error: for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-09T11%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-09T12%3A00%3A00Z'&$skiptoken=1999" stack_info=True   ... View more

@LukaszZeszko- would you be able to share the permissions you set up to avoid the 403 error? I've gotten stuck there. I have "Office 365 Exchange Online --> ReportingWebService.Read.All" permissions for my application. It seems I need more permissions for this to work but it's unclear to me from the documentation what permissions that should be. Thanks in advance! ... View more

That's great information. I had reviewed that update to the app and to using OAuth but never got around to implementing. I'm going to test that in my non-production environment today and I'll report back. ... View more

I was glad to find your post - we're having the exact same issue. My first instance of receiving a 401 error for that integration was on 2023-03-31 13:22:20 (Eastern). My O365 administration team pointed me to this posting from Microsoft (looks like you'll need a certain level of access in your tenant to view this) https://admin.microsoft.com/#/servicehealth/:/alerts/EX537209. It would seem that this is a Microsoft issue but any kind of information is hard to come by. I hope that's of some use. We're working on reporting it to MS but we're basically in a holding pattern until this works again. ... View more

I never did, no, but I went forward with configuring this myself. I run a test environment for Splunk, so I was able to confirm that there didn't seem to be any adverse affects from running the sensor on my hosts. Since deploying it in production I've had zero issues and zero detections of any kind. I created a dedicated host group with tags to manage my Splunk environment separately (if necessary) but I haven't had to. I have pretty default linux sensor settings and it has been working fine. I'm happy to share any specific configurations I have in place if you need any guidance. ... View more