Splunk Add-on for Microsoft Cloud Services: How do I get Message Tracking Logs from Exchange Online into Splunk?

[user12345a_2]

Hello.

At the moment my organization uses MS Exchange on-premise. We index our Message Tracking Logs for our Information Security team who use that information in searches / panels for message recipient lists, etc. We are moving our on-premise Exchange to Office 365. Will the Splunk-Add on for MS Cloud Services enable me to give them similar information to what they used to get from the Message Track. Logs?

Thanks in advance for any advice / pointers.

[user12345a_2 · Accepted solution]

So I've been running the following Powershell script for the past several months without issues. It downloads up to 5 million messages and runs in about 40 seconds. I run it via task scheduler every 10 minutes. It's proven very useful to our Incident Response to have the message trace logs in Splunk.

...
<SESSION CONNECTION STUFF>
...
[DateTime]$DateEnd = Get-Date -format g
$DateEnd = $DateEnd.ToUniversalTime()
[DateTime]$DateStart = $DateEnd.Addminutes(-10)
$Outfile = "c:\O365\logs\MessageTrace_" + (get-date -Format "MM-dd-yyy-hh-mm-ss") + ".csv"

$FoundCount = 0

For($i = 1; $i -le 1000; $i++)  # Maximum allowed pages is 1000
{
    $Messages = Get-MessageTrace -StartDate $DateStart -EndDate $DateEnd -PageSize 5000 -Page $i

    If($Messages.count -gt 0)
    {
        $Status = $Messages[-1].Received.ToString("MM/dd/yyyy HH:mm") + " - " + $Messages[0].Received.ToString("MM/dd/yyyy HH:mm") + "  [" + ("{0:N0}" -f ($i*5000)) + " Searched | " + $FoundCount + " Found]"

        Write-Progress -activity "Checking Messages (Up to 5 Million)..." -status $Status

        $Entries = $Messages | Select Received, SenderAddress, RecipientAddress, Subject, Status, FromIP, Size, MessageId
        $Entries | Export-Csv $Outfile -NoTypeInformation -Append

        $FoundCount += $Entries.Count
    }
    Else
    {
        Break
    }
}  

Write-Host $FoundCount "Entries Found & Logged In" $Outfile

# (Get-Content $Outfile) | Foreach-Object {$_ -replace '"', ""} | out-file -FilePath $Outfile -Force -Encoding ascii

###################################################
# Delete all Files in C:\O365\ older than 2 days  #
###################################################
$Path = "C:\O365\logs"
$Daysback = "-2"

$CurrentDate = Get-Date
$DatetoDelete = $CurrentDate.AddDays($Daysback)
Get-ChildItem $Path | Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item

[mbenwell]

I've previously asked for message tracking on the add-on too. Would be really handy

In the mean time it's possible to use the 'Get-MessageTrace' powershell command in a simple script. Don't expect the below to work by copying it verbatim as it's not complete but it should give you an idea what I've done to deal with this for the interim. It's not perfect but it does work. If anyone wants to offer suggests to wrap some more smarts around it go your hardest, feedback is more than welcome 🙂

...
<session stuff>
...
For ($i=1; $i -le 10; $i++) {  
  Get-MessageTrace -PageSize 5000 -Page $i -StartDate ([DateTime]::Now.AddMinutes(-30)) -EndDate ([DateTime]::Now.AddMinutes(-15)) | Select @{Name="time";Expression={$_."Received"}},
  @{Name="message_id";Expression={$_."MessageID"}},
  @{Name="recipient";Expression={$_."RecipientAddress"}},
  @{Name="sender";Expression={$_."SenderAddress"}},
  @{Name="src_ip";Expression={$_."FromIP"}},
  @{Name="dest_ip";Expression={$_."ToIP"}},
  @{Name="subject";Expression={$_."Subject"}},
  @{Name="status";Expression={$_."Status"}},
  @{Name="size";Expression={$_."Size"}} | Export-Csv "c:\temp\message_trace\$((get-date ([DateTime]::Now) -Format yyyyMMddTHHmm))_page_$i.csv" -NoTypeInformation
}

[ehaddad_splunk]

In the MS Cloud Services Add-on, you have the ability to ingest data from the MS O365 management activity API:
https://msdn.microsoft.com/en-us/office-365/office-365-management-activity-api-reference

This should give you access to all sorts of audit events related Exchange online.

[marycordova]

I downvoted this post because it wasn't in the least bit helpful, lmgtfy does not count as an answer.

[GregMefford]

I believe this add-on (currently) only collects Exchange Online Audit Logs, not message-tracking logs. This would cover things like administrative access to a user's mailbox, sending on behalf of someone else, etc. You can configure it to audit more detailed information like the creation of different kinds of items, but I haven't seen a way yet to use this type of log to track all messages being sent (as opposed to mail items being created or modified).

Microsoft maintains message tracking logs in a searchable form for some period of time, and I'd also be interested to know how to collect them yourself for longer-term retention and analysis in Splunk. There's probably a way, but I haven't found it yet either.

[ehaddad_splunk]

does enabling extra logging allow you to get the full message using the same API call in the addon?

https://technet.microsoft.com/en-us/library/ff461937(v=exchg.160).aspx.