All Apps and Add-ons

Splunk Add-on for Microsoft Cloud Services: How do I get Message Tracking Logs from Exchange Online into Splunk?

user12345a_2
Explorer

Hello.

At the moment my organization uses MS Exchange on-premise. We index our Message Tracking Logs for our Information Security team who use that information in searches / panels for message recipient lists, etc. We are moving our on-premise Exchange to Office 365. Will the Splunk-Add on for MS Cloud Services enable me to give them similar information to what they used to get from the Message Track. Logs?

Thanks in advance for any advice / pointers.

1 Solution

user12345a_2
Explorer

So I've been running the following Powershell script for the past several months without issues. It downloads up to 5 million messages and runs in about 40 seconds. I run it via task scheduler every 10 minutes. It's proven very useful to our Incident Response to have the message trace logs in Splunk.

...
<SESSION CONNECTION STUFF>
...
[DateTime]$DateEnd = Get-Date -format g
$DateEnd = $DateEnd.ToUniversalTime()
[DateTime]$DateStart = $DateEnd.Addminutes(-10)
$Outfile = "c:\O365\logs\MessageTrace_" + (get-date -Format "MM-dd-yyy-hh-mm-ss") + ".csv"

$FoundCount = 0

For($i = 1; $i -le 1000; $i++)  # Maximum allowed pages is 1000
{
    $Messages = Get-MessageTrace -StartDate $DateStart -EndDate $DateEnd -PageSize 5000 -Page $i

    If($Messages.count -gt 0)
    {
        $Status = $Messages[-1].Received.ToString("MM/dd/yyyy HH:mm") + " - " + $Messages[0].Received.ToString("MM/dd/yyyy HH:mm") + "  [" + ("{0:N0}" -f ($i*5000)) + " Searched | " + $FoundCount + " Found]"

        Write-Progress -activity "Checking Messages (Up to 5 Million)..." -status $Status

        $Entries = $Messages | Select Received, SenderAddress, RecipientAddress, Subject, Status, FromIP, Size, MessageId
        $Entries | Export-Csv $Outfile -NoTypeInformation -Append

        $FoundCount += $Entries.Count
    }
    Else
    {
        Break
    }
}  

Write-Host $FoundCount "Entries Found & Logged In" $Outfile

# (Get-Content $Outfile) | Foreach-Object {$_ -replace '"', ""} | out-file -FilePath $Outfile -Force -Encoding ascii

###################################################
# Delete all Files in C:\O365\ older than 2 days  #
###################################################
$Path = "C:\O365\logs"
$Daysback = "-2"

$CurrentDate = Get-Date
$DatetoDelete = $CurrentDate.AddDays($Daysback)
Get-ChildItem $Path | Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item

View solution in original post

0 Karma

user12345a_2
Explorer

So I've been running the following Powershell script for the past several months without issues. It downloads up to 5 million messages and runs in about 40 seconds. I run it via task scheduler every 10 minutes. It's proven very useful to our Incident Response to have the message trace logs in Splunk.

...
<SESSION CONNECTION STUFF>
...
[DateTime]$DateEnd = Get-Date -format g
$DateEnd = $DateEnd.ToUniversalTime()
[DateTime]$DateStart = $DateEnd.Addminutes(-10)
$Outfile = "c:\O365\logs\MessageTrace_" + (get-date -Format "MM-dd-yyy-hh-mm-ss") + ".csv"

$FoundCount = 0

For($i = 1; $i -le 1000; $i++)  # Maximum allowed pages is 1000
{
    $Messages = Get-MessageTrace -StartDate $DateStart -EndDate $DateEnd -PageSize 5000 -Page $i

    If($Messages.count -gt 0)
    {
        $Status = $Messages[-1].Received.ToString("MM/dd/yyyy HH:mm") + " - " + $Messages[0].Received.ToString("MM/dd/yyyy HH:mm") + "  [" + ("{0:N0}" -f ($i*5000)) + " Searched | " + $FoundCount + " Found]"

        Write-Progress -activity "Checking Messages (Up to 5 Million)..." -status $Status

        $Entries = $Messages | Select Received, SenderAddress, RecipientAddress, Subject, Status, FromIP, Size, MessageId
        $Entries | Export-Csv $Outfile -NoTypeInformation -Append

        $FoundCount += $Entries.Count
    }
    Else
    {
        Break
    }
}  

Write-Host $FoundCount "Entries Found & Logged In" $Outfile

# (Get-Content $Outfile) | Foreach-Object {$_ -replace '"', ""} | out-file -FilePath $Outfile -Force -Encoding ascii

###################################################
# Delete all Files in C:\O365\ older than 2 days  #
###################################################
$Path = "C:\O365\logs"
$Daysback = "-2"

$CurrentDate = Get-Date
$DatetoDelete = $CurrentDate.AddDays($Daysback)
Get-ChildItem $Path | Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item
0 Karma

mbenwell
Communicator

I've previously asked for message tracking on the add-on too. Would be really handy

In the mean time it's possible to use the 'Get-MessageTrace' powershell command in a simple script. Don't expect the below to work by copying it verbatim as it's not complete but it should give you an idea what I've done to deal with this for the interim. It's not perfect but it does work. If anyone wants to offer suggests to wrap some more smarts around it go your hardest, feedback is more than welcome 🙂

...
<session stuff>
...
For ($i=1; $i -le 10; $i++) {  
  Get-MessageTrace -PageSize 5000 -Page $i -StartDate ([DateTime]::Now.AddMinutes(-30)) -EndDate ([DateTime]::Now.AddMinutes(-15)) | Select @{Name="time";Expression={$_."Received"}},
  @{Name="message_id";Expression={$_."MessageID"}},
  @{Name="recipient";Expression={$_."RecipientAddress"}},
  @{Name="sender";Expression={$_."SenderAddress"}},
  @{Name="src_ip";Expression={$_."FromIP"}},
  @{Name="dest_ip";Expression={$_."ToIP"}},
  @{Name="subject";Expression={$_."Subject"}},
  @{Name="status";Expression={$_."Status"}},
  @{Name="size";Expression={$_."Size"}} | Export-Csv "c:\temp\message_trace\$((get-date ([DateTime]::Now) -Format yyyyMMddTHHmm))_page_$i.csv" -NoTypeInformation
}
0 Karma

ehaddad_splunk
Splunk Employee
Splunk Employee

In the MS Cloud Services Add-on, you have the ability to ingest data from the MS O365 management activity API:
https://msdn.microsoft.com/en-us/office-365/office-365-management-activity-api-reference

This should give you access to all sorts of audit events related Exchange online.

marycordova
SplunkTrust
SplunkTrust

I downvoted this post because it wasn't in the least bit helpful, lmgtfy does not count as an answer.

@marycordova
0 Karma

GregMefford
Explorer

I believe this add-on (currently) only collects Exchange Online Audit Logs, not message-tracking logs. This would cover things like administrative access to a user's mailbox, sending on behalf of someone else, etc. You can configure it to audit more detailed information like the creation of different kinds of items, but I haven't seen a way yet to use this type of log to track all messages being sent (as opposed to mail items being created or modified).

Microsoft maintains message tracking logs in a searchable form for some period of time, and I'd also be interested to know how to collect them yourself for longer-term retention and analysis in Splunk. There's probably a way, but I haven't found it yet either.

0 Karma

ehaddad_splunk
Splunk Employee
Splunk Employee

does enabling extra logging allow you to get the full message using the same API call in the addon?

https://technet.microsoft.com/en-us/library/ff461937(v=exchg.160).aspx.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...