All Apps and Add-ons

Import Office365 message tracking logs into Splunk

bandras
Explorer

We are currently using the Splunk Add-on for Microsoft Cloud Services but it doesn't support importing of message tracking logs. These logs are critical to our SOC so we need to find a way to export/import them. I can export them from Office365 via PowerShell but this will be cumbersome. Has anyone else solved this issue? Thanks for the help!

jconger
Splunk Employee
Splunk Employee

The Microsoft Office 365 Reporting Add-on for Splunk pulls message trace logs -> https://splunkbase.splunk.com/app/3720/

sloshburch
Splunk Employee
Splunk Employee

As per the latest release right? That's new functionality in like 1.0.1 or was it always there?

0 Karma

jconger
Splunk Employee
Splunk Employee

Pulling message trace logs has was the main use case for the O365 reporting add-on, so that functionality has always been there.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Thanks for clarifying sir!

0 Karma

bandras
Explorer

Thank you all for the info. We implemented a Powershell based approach to gather message trace logs but I'm looking at replacing that with this add-on now. The confusion came from 2 different add-on's in splunkbase.

Here is the one that I'm currently using:
https://splunkbase.splunk.com/app/3110/#/overview
This app pulls audit logs, etc. but it does not pull message trace.

The app that you make a reference to is:
https://splunkbase.splunk.com/app/3720/#/overview
I was not aware of this app or it didn't exist when we attempted to implement this.

Thank you for the update!

0 Karma

TonyLeeVT
Builder

This is fantastic. I was also surprised to find the other TA only pulling Audit logs (missing sender and message_id)--so thank you for sharing this app.

Does anyone know how well the "Microsoft Office 365 Reporting Add-on for Splunk" scales? For example, what is the max number of emails per day they have seen pulled in? Or the size of the o365 instance (number of users)?

Thanks again

0 Karma

sloshburch
Splunk Employee
Splunk Employee

This question may be better responded to as a new answers post. But to start the conversation, I think there's a lot of "it depends". It's more of a question on what will be the bottleneck in the system. IIRC, the add on you referenced will use some modular inputs which pull from O365. The things to consider are the size of the host running the collection, what other things you have it doing, the specs of the server (how fast can it write the results), any networking bottlenecks, etc...
It might be more fruitful to work with your account time to learn how to best enable the feature while monitoring the performance in the Monitoring Console. You can then more easily identify when it's time to scale out for more resources.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

In terms of the powershell approach:

  • It appears the data comes in over REST API and there are some query parameters that could be used on such a REST call. As such, I imagine you could use the Add-On Builder and add some cursor management to the result so as to keep track of what’s been brought it vs what’s new
  • For the approach of writing to a file, wouldn’t you need some limits on what’s collected so as not to be trying to write massive files? I imagine that years from now, each REST call could be huge and you wouldn’t want to rewrite the entire file every time. I imagine you’d have to use the REST filters (search https://msdn.microsoft.com/en-us/office-365/office-365-management-activity-api-reference for ‘startTime’ for example).

Just got word of another potential solution from the PM.

Apparently, we were told that this is achieved by enabling extra level of auditing as per https://technet.microsoft.com/en-us/library/ff461937(v=exchg.160).aspx although what you're looking for might be more of the stuff not exposed over REST api as per https://technet.microsoft.com/en-us/library/jj200712(v=exchg.150).aspx

If it's not available on the REST API, I'd go back to exploring the AddOn Builder to collect and index the data.

I see there is also a webhooks approach. If you prefer to go with a webhooks approach, you could have the data be posted to an HTTP Event Collector.

0 Karma

bandras
Explorer

Update on using a webhook to get this data:

You can configure the O365 Management API to send data to a webhook, but that data would be limited to what’s available in the API which doesn’t include message tracking.

To answer your question about configuring the webhook.
- To configure the Office365 Management API to send data to a webhook, you would have to make a one-time REST POST call to the API that will start a subscription and specify the webhook properties (URL, credentials, etc).
- After that, Office365 will send a HTTP POST call to the webhook when new content is available in the service you subscribed to.
- The webhook is going to be on the application side, so you would need the ability to configure a webhook listener in Splunk, or utilize Azure Automation to process the webhook data.

All that being said, Office 365 Reporting Web Services is what you would need to utilize to programmatically pull message trace logs and I’m not aware if it can be configured to send data to a webhook.

Link to the Office 365 Reporting Web Services: https://msdn.microsoft.com/library/office/jj984325.aspx

0 Karma

sloshburch
Splunk Employee
Splunk Employee

To clarify "configure a webhook listener in Splunk" -> that is addressed with the HTTP Event Collector feature of Splunk.

But it doesn't matter here because O365 won't send the data you need on a webhook, right? If that is correct, then I'd suggest the powershell approach even before looking into writing directly to a log file.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Potentially related thread: https://answers.splunk.com/answers/470197/splunk-add-on-for-microsoft-cloud-services-how-do.html

Sounds like the add-on currently collects the Exchange Online Audit Logs but not message-tracking logs. Hence this gap here.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...