TA-Exchange-ClientAccess - Object reference not set to an instance of an object ( Event ID 4100 and 4104)

[corti77]

Hi,

I have configured IT Essential Works (4.9.2) with Exchange content pack (1.4.3) and  TA-Exchange-ClientAccess (4.0.3).
By chance I was checking PowerShell event logs in our exchange server and I saw the error bellow.

Log Name:      Microsoft-Windows-PowerShell/Operational
Source:        Microsoft-Windows-PowerShell
Date:          29/11/2021 11:34:13
Event ID:      4100
Task Category: Executing Pipeline
Level:         Warning
Keywords:      None
User:          SYSTEM
Computer:      XXXX.YYYY.ZZZZ
Description:
Error Message = Object reference not set to an instance of an object.
Fully Qualified Error ID = System.NullReferenceException,Microsoft.Exchange.Management.SystemConfigurationTasks.SearchAdminAuditLog


Context:
        Severity = Warning
        Host Name = ConsoleHost
        Host Version = 5.1.14393.4583
        Host ID = 644d49a8-7f8f-4b1e-9250-959ff1a8b7b4
        Host Application = Powershell -PSConsoleFile E:\Program Files\Microsoft\Exchange Server\V15\\bin\exshell.psc1 -command . 'C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1'
        Engine Version = 5.1.14393.4583
        Runspace ID = 10a2c198-89dd-47c1-99c1-4d493d35a837
        Pipeline ID = 1
        Command Name = Search-AdminAuditLog
        Command Type = Cmdlet
        Script Name = C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1
        Command Path = 
        Sequence Number = 19
        User = XXXXX\SYSTEM
        Connected User = 
        Shell ID = Microsoft.PowerShell


User Data:


Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-PowerShell" Guid="{A0C1853B-5C40-4B15-8766-3CF1C58F985A}" />
    <EventID>4100</EventID>
    <Version>1</Version>
    <Level>3</Level>
    <Task>106</Task>
    <Opcode>19</Opcode>
    <Keywords>0x0</Keywords>
    <TimeCreated SystemTime="2021-11-29T10:34:13.679546900Z" />
    <EventRecordID>5879670</EventRecordID>
    <Correlation ActivityID="{2DF7DE6F-E0AC-000E-036D-F92DACE0D701}" />
    <Execution ProcessID="62888" ThreadID="41736" />
    <Channel>Microsoft-Windows-PowerShell/Operational</Channel>
    <Computer>XXXXXX.YYYY.ZZZZ</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="ContextInfo">        Severity = Warning
        Host Name = ConsoleHost
        Host Version = 5.1.14393.4583
        Host ID = 644d49a8-7f8f-4b1e-9250-959ff1a8b7b4
        Host Application = Powershell -PSConsoleFile E:\Program Files\Microsoft\Exchange Server\V15\\bin\exshell.psc1 -command . 'C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1'
        Engine Version = 5.1.14393.4583
        Runspace ID = 10a2c198-89dd-47c1-99c1-4d493d35a837
        Pipeline ID = 1
        Command Name = Search-AdminAuditLog
        Command Type = Cmdlet
        Script Name = C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1
        Command Path = 
        Sequence Number = 19
        User = XXXXXXX\SYSTEM
        Connected User = 
        Shell ID = Microsoft.PowerShell
</Data>
    <Data Name="UserData">
    </Data>
    <Data Name="Payload">Error Message = Object reference not set to an instance of an object.
Fully Qualified Error ID = System.NullReferenceException,Microsoft.Exchange.Management.SystemConfigurationTasks.SearchAdminAuditLog
</Data>
  </EventData>
</Event>

Log Name:      Microsoft-Windows-PowerShell/Operational
Source:        Microsoft-Windows-PowerShell
Date:          29/11/2021 11:34:09
Event ID:      4104
Task Category: Execute a Remote Command
Level:         Warning
Keywords:      None
User:          SYSTEM
Computer:      XXXXX.YYYY.ZZZZ
Description:
Creating Scriptblock text (1 of 1):
########################################################
#
# Splunk for Microsoft Exchange
# Exchange 2010/2013 Mailbox Store Data Definition
# 
# Copyright (C) 2005-2021 Splunk Inc. All Rights Reserved.
# All Rights Reserved
#
########################################################
#
# This returns the filename of the audit database - due to some
# funkiness with permissions, deployment server and the local
# directory, we're using the %TEMP% as the location.  For the
# NT Authority\SYSTEM account, this is normally C:\Windows\Temp
#
$AuditTempFile = $ENV:Temp | Join-Path -ChildPath "splunk-msexchange-mailboxauditlogs.clixml"
[Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8

$AuditDetails = @{}
if (Test-Path $AuditTempFile) {
	$AuditDetails = Import-CliXml $AuditTempFile
}

#
# Given a single audit record from the Search-MailboxAuditLog 
function Output-AuditRecord($Record) {
	$O = New-Object System.Collections.ArrayList
	$D = Get-Date $Record.LastAccessed -format 'yyyy-MM-ddTHH:mm:sszzz'
	[void]$O.Add($D)
	
	foreach ($p in $Record.PSObject.Properties) {
		[void]$O.Add("$($p.Name)=`"$($Record.PSObject.Properties[$p.Name].Value)`"")
	}
	
	Write-Host ($O -join " ")
}

function Output-AuditLog($Mailbox) {
	$Identity = $Mailbox.Identity
    $IdentityStr = $Identity.ToDNString()
	$LastSeen = (Get-Date).AddMonths(-1)
    if ($AuditDetails.ContainsKey($Identity)) {
        $LastSeen = $AuditDetails[$Identity]
        $AuditDetails.Remove($Identity)
        $AuditDetails[$IdentityStr] = $LastSeen
    } elseif ($AuditDetails.ContainsKey($IdentityStr)) {
        $LastSeen = $AuditDetails[$IdentityStr]
    }
	
	$LastRecord = $LastSeen
	Search-MailboxAuditLog -Identity $Identity -LogonTypes Owner,Delegate,Admin -ShowDetails -StartDate $LastSeen -EndDate (Get-Date)| sort LastAccessed | Foreach-Object {
		if ($_.LastAccessed -gt $LastSeen) {
			Output-AuditRecord($_)
		}
		$LastRecord = $_.LastAccessed
	}
	
	$AuditDetails[$IdentityStr] = $LastRecord
}

$Mailboxes = Get-Mailbox -Filter { AuditEnabled -eq $true } -Server $Env:ComputerName -ResultSize Unlimited
$Mailboxes | Foreach-Object { If($_ -ne $null)
{ Output-AuditLog($_) }}

#
# Now that we have done the work, save off the Audit Temp File
$AuditDetails | Export-CliXml $AuditTempFile


ScriptBlock ID: 1e79b015-daf6-40c2-8c87-fedde7b4a866
Path: C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-Mailbox\bin\powershell\read-mailbox-audit-logs_2010_2013.ps1
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-PowerShell" Guid="{A0C1853B-5C40-4B15-8766-3CF1C58F985A}" />
    <EventID>4104</EventID>
    <Version>1</Version>
    <Level>3</Level>
    <Task>2</Task>
    <Opcode>15</Opcode>
    <Keywords>0x0</Keywords>
    <TimeCreated SystemTime="2021-11-29T10:34:09.300998400Z" />
    <EventRecordID>5879669</EventRecordID>
    <Correlation ActivityID="{2DF7DE6F-E0AC-0010-CA3C-F92DACE0D701}" />
    <Execution ProcessID="38808" ThreadID="54116" />
    <Channel>Microsoft-Windows-PowerShell/Operational</Channel>
    <Computer>XXXX.YYYY.ZZZ</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="MessageNumber">1</Data>
    <Data Name="MessageTotal">1</Data>
    <Data Name="ScriptBlockText">########################################################
#
# Splunk for Microsoft Exchange
# Exchange 2010/2013 Mailbox Store Data Definition
# 
# Copyright (C) 2005-2021 Splunk Inc. All Rights Reserved.
# All Rights Reserved
#
########################################################
#
# This returns the filename of the audit database - due to some
# funkiness with permissions, deployment server and the local
# directory, we're using the %TEMP% as the location.  For the
# NT Authority\SYSTEM account, this is normally C:\Windows\Temp
#
$AuditTempFile = $ENV:Temp | Join-Path -ChildPath "splunk-msexchange-mailboxauditlogs.clixml"
[Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8

$AuditDetails = @{}
if (Test-Path $AuditTempFile) {
	$AuditDetails = Import-CliXml $AuditTempFile
}

#
# Given a single audit record from the Search-MailboxAuditLog 
function Output-AuditRecord($Record) {
	$O = New-Object System.Collections.ArrayList
	$D = Get-Date $Record.LastAccessed -format 'yyyy-MM-ddTHH:mm:sszzz'
	[void]$O.Add($D)
	
	foreach ($p in $Record.PSObject.Properties) {
		[void]$O.Add("$($p.Name)=`"$($Record.PSObject.Properties[$p.Name].Value)`"")
	}
	
	Write-Host ($O -join " ")
}

function Output-AuditLog($Mailbox) {
	$Identity = $Mailbox.Identity
    $IdentityStr = $Identity.ToDNString()
	$LastSeen = (Get-Date).AddMonths(-1)
    if ($AuditDetails.ContainsKey($Identity)) {
        $LastSeen = $AuditDetails[$Identity]
        $AuditDetails.Remove($Identity)
        $AuditDetails[$IdentityStr] = $LastSeen
    } elseif ($AuditDetails.ContainsKey($IdentityStr)) {
        $LastSeen = $AuditDetails[$IdentityStr]
    }
	
	$LastRecord = $LastSeen
	Search-MailboxAuditLog -Identity $Identity -LogonTypes Owner,Delegate,Admin -ShowDetails -StartDate $LastSeen -EndDate (Get-Date)| sort LastAccessed | Foreach-Object {
		if ($_.LastAccessed -gt $LastSeen) {
			Output-AuditRecord($_)
		}
		$LastRecord = $_.LastAccessed
	}
	
	$AuditDetails[$IdentityStr] = $LastRecord
}

$Mailboxes = Get-Mailbox -Filter { AuditEnabled -eq $true } -Server $Env:ComputerName -ResultSize Unlimited
$Mailboxes | Foreach-Object { If($_ -ne $null)
{ Output-AuditLog($_) }}

#
# Now that we have done the work, save off the Audit Temp File
$AuditDetails | Export-CliXml $AuditTempFile
</Data>
    <Data Name="ScriptBlockId">1e79b015-daf6-40c2-8c87-fedde7b4a866</Data>
    <Data Name="Path">C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-Mailbox\bin\powershell\read-mailbox-audit-logs_2010_2013.ps1</Data>
  </EventData>
</Event>

Any idea on what could be happening? I found some answer related to the user that is running the universal splunk forwarder in the server. It is currently configured as local SYSTEM and in some answers, found on internet, it was mentioned to use a domain account with exchange permissions. But I check the official documentation and I could not find any mention to it.

Thanks.