All Apps and Add-ons
Highlighted

Import Office365 message tracking logs into Splunk

Explorer

We are currently using the Splunk Add-on for Microsoft Cloud Services but it doesn't support importing of message tracking logs. These logs are critical to our SOC so we need to find a way to export/import them. I can export them from Office365 via PowerShell but this will be cumbersome. Has anyone else solved this issue? Thanks for the help!

Highlighted

Re: Import Office365 message tracking logs into Splunk

Ultra Champion

Potentially related thread: https://answers.splunk.com/answers/470197/splunk-add-on-for-microsoft-cloud-services-how-do.html

Sounds like the add-on currently collects the Exchange Online Audit Logs but not message-tracking logs. Hence this gap here.

0 Karma
Highlighted

Re: Import Office365 message tracking logs into Splunk

Ultra Champion

In terms of the powershell approach:

  • It appears the data comes in over REST API and there are some query parameters that could be used on such a REST call. As such, I imagine you could use the Add-On Builder and add some cursor management to the result so as to keep track of what’s been brought it vs what’s new
  • For the approach of writing to a file, wouldn’t you need some limits on what’s collected so as not to be trying to write massive files? I imagine that years from now, each REST call could be huge and you wouldn’t want to rewrite the entire file every time. I imagine you’d have to use the REST filters (search https://msdn.microsoft.com/en-us/office-365/office-365-management-activity-api-reference for ‘startTime’ for example).

Just got word of another potential solution from the PM.

Apparently, we were told that this is achieved by enabling extra level of auditing as per https://technet.microsoft.com/en-us/library/ff461937(v=exchg.160).aspx although what you're looking for might be more of the stuff not exposed over REST api as per https://technet.microsoft.com/en-us/library/jj200712(v=exchg.150).aspx

If it's not available on the REST API, I'd go back to exploring the AddOn Builder to collect and index the data.

I see there is also a webhooks approach. If you prefer to go with a webhooks approach, you could have the data be posted to an HTTP Event Collector.

0 Karma
Highlighted

Re: Import Office365 message tracking logs into Splunk

Explorer

Update on using a webhook to get this data:

You can configure the O365 Management API to send data to a webhook, but that data would be limited to what’s available in the API which doesn’t include message tracking.

To answer your question about configuring the webhook.
- To configure the Office365 Management API to send data to a webhook, you would have to make a one-time REST POST call to the API that will start a subscription and specify the webhook properties (URL, credentials, etc).
- After that, Office365 will send a HTTP POST call to the webhook when new content is available in the service you subscribed to.
- The webhook is going to be on the application side, so you would need the ability to configure a webhook listener in Splunk, or utilize Azure Automation to process the webhook data.

All that being said, Office 365 Reporting Web Services is what you would need to utilize to programmatically pull message trace logs and I’m not aware if it can be configured to send data to a webhook.

Link to the Office 365 Reporting Web Services: https://msdn.microsoft.com/library/office/jj984325.aspx

0 Karma
Highlighted

Re: Import Office365 message tracking logs into Splunk

Ultra Champion

To clarify "configure a webhook listener in Splunk" -> that is addressed with the HTTP Event Collector feature of Splunk.

But it doesn't matter here because O365 won't send the data you need on a webhook, right? If that is correct, then I'd suggest the powershell approach even before looking into writing directly to a log file.

0 Karma
Highlighted

Re: Import Office365 message tracking logs into Splunk

Splunk Employee
Splunk Employee

The Microsoft Office 365 Reporting Add-on for Splunk pulls message trace logs -> https://splunkbase.splunk.com/app/3720/

Highlighted

Re: Import Office365 message tracking logs into Splunk

Ultra Champion

As per the latest release right? That's new functionality in like 1.0.1 or was it always there?

0 Karma
Highlighted

Re: Import Office365 message tracking logs into Splunk

Splunk Employee
Splunk Employee

Pulling message trace logs has was the main use case for the O365 reporting add-on, so that functionality has always been there.

0 Karma
Highlighted

Re: Import Office365 message tracking logs into Splunk

Ultra Champion

Thanks for clarifying sir!

0 Karma
Highlighted

Re: Import Office365 message tracking logs into Splunk

Explorer

Thank you all for the info. We implemented a Powershell based approach to gather message trace logs but I'm looking at replacing that with this add-on now. The confusion came from 2 different add-on's in splunkbase.

Here is the one that I'm currently using:
https://splunkbase.splunk.com/app/3110/#/overview
This app pulls audit logs, etc. but it does not pull message trace.

The app that you make a reference to is:
https://splunkbase.splunk.com/app/3720/#/overview
I was not aware of this app or it didn't exist when we attempted to implement this.

Thank you for the update!

0 Karma