We are currently using the Splunk Add-on for Microsoft Cloud Services but it doesn't support importing of message tracking logs. These logs are critical to our SOC so we need to find a way to export/import them. I can export them from Office365 via PowerShell but this will be cumbersome. Has anyone else solved this issue? Thanks for the help!
Thank you all for the info. We implemented a Powershell based approach to gather message trace logs but I'm looking at replacing that with this add-on now. The confusion came from 2 different add-on's in splunkbase.
Here is the one that I'm currently using:
This app pulls audit logs, etc. but it does not pull message trace.
The app that you make a reference to is:
I was not aware of this app or it didn't exist when we attempted to implement this.
Thank you for the update!
This is fantastic. I was also surprised to find the other TA only pulling Audit logs (missing sender and message_id)--so thank you for sharing this app.
Does anyone know how well the "Microsoft Office 365 Reporting Add-on for Splunk" scales? For example, what is the max number of emails per day they have seen pulled in? Or the size of the o365 instance (number of users)?
This question may be better responded to as a new answers post. But to start the conversation, I think there's a lot of "it depends". It's more of a question on what will be the bottleneck in the system. IIRC, the add on you referenced will use some modular inputs which pull from O365. The things to consider are the size of the host running the collection, what other things you have it doing, the specs of the server (how fast can it write the results), any networking bottlenecks, etc...
It might be more fruitful to work with your account time to learn how to best enable the feature while monitoring the performance in the Monitoring Console. You can then more easily identify when it's time to scale out for more resources.
In terms of the powershell approach:
Just got word of another potential solution from the PM.
Apparently, we were told that this is achieved by enabling extra level of auditing as per https://technet.microsoft.com/en-us/library/ff461937(v=exchg.160).aspx although what you're looking for might be more of the stuff not exposed over REST api as per https://technet.microsoft.com/en-us/library/jj200712(v=exchg.150).aspx
If it's not available on the REST API, I'd go back to exploring the AddOn Builder to collect and index the data.
I see there is also a webhooks approach. If you prefer to go with a webhooks approach, you could have the data be posted to an HTTP Event Collector.
Update on using a webhook to get this data:
You can configure the O365 Management API to send data to a webhook, but that data would be limited to what’s available in the API which doesn’t include message tracking.
To answer your question about configuring the webhook.
- To configure the Office365 Management API to send data to a webhook, you would have to make a one-time REST POST call to the API that will start a subscription and specify the webhook properties (URL, credentials, etc).
- After that, Office365 will send a HTTP POST call to the webhook when new content is available in the service you subscribed to.
- The webhook is going to be on the application side, so you would need the ability to configure a webhook listener in Splunk, or utilize Azure Automation to process the webhook data.
All that being said, Office 365 Reporting Web Services is what you would need to utilize to programmatically pull message trace logs and I’m not aware if it can be configured to send data to a webhook.
Link to the Office 365 Reporting Web Services: https://msdn.microsoft.com/library/office/jj984325.aspx
To clarify "configure a webhook listener in Splunk" -> that is addressed with the HTTP Event Collector feature of Splunk.
But it doesn't matter here because O365 won't send the data you need on a webhook, right? If that is correct, then I'd suggest the powershell approach even before looking into writing directly to a log file.
Potentially related thread: https://answers.splunk.com/answers/470197/splunk-add-on-for-microsoft-cloud-services-how-do.html
Sounds like the add-on currently collects the Exchange Online Audit Logs but not message-tracking logs. Hence this gap here.