All Apps and Add-ons

Splunk App for Microsoft Exchange: Why are we not receiving admin audit events?

Contributor

Still a pretty new Splunker here -- We are testing the Splunk App for Exchange in our dev environment. I've followed the documentation pretty closely (we did not install a Universal Forwarder on our Hub Transport device, we aren't interested in message tracking). We've installed UFs on our MailboxStore and CAS boxes. This is an Exchange 2010 environment.

We are currently only getting events for the following sourcetypes from this configuration:
1. MSExchange:2010:Folder-Usage
2. MSexchange:2010:Topology
3. MSExchange:2010:Mailbox-Usage
4. MSExchange:2010:PublicFolder-Stats
5. MSExchange:2010:Database-Stats
6. MSExchange:2010:RPCClientAccess
7. MSEXchange:2010:ThrottlingPolicy

We are looking to track admin accesses and changes to mailboxes and non-owner mailbox access in our environment. I figure the powershell scripts "read-audit-logs" and "read-mailbox-audit-logs" are responsible for pulling these events into Splunk, but they don't seem to be running. The Exchange admin can run Exchange PowerShell cmdlets on the server itself such as Search-MailboxAuditLog and we can see the events for the actions he's making. But these events are not sending over to Splunk. inputs.conf has been left default and the stanzas for the sourcetypes MSExchange:2010:AdminAudit and MSExchange:2010:MailboxAudit both have the setting disabled=false.

Any ideas on where to look to get these logs into Splunk?

Explorer

We do have the same issue with an exchange server 2013. Other powershell scripts run great, but the following scripts fail:

MSExchange:2013:AdminAudit (read-audit-logs20102013.ps1)
MSExchange:2013:MailboxAudit (read-mailbox-audit-logs20102013.ps1)

When we check the logs in the Splunk forwarder we see following entry repeating every 5min:
Location of the log file: C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log

09-23-2016 15:54:40.918 +0200 ERROR ExecProcessor - message from 
""C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd" v15 read-audit-logs_2010_2013.ps1"

Search-AdminAuditLog : The attempt to search the administrator audit log 
failed. Please try again later.
At C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\
bin\powershell\read-audit-logs_2010_2013.ps1:81 char:12
+ $Records = Search-AdminAuditLog -StartDate $LastSeen
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Search-AdminAuditLog], AdminA 
   uditLogSearchException
    + FullyQualifiedErrorId : [Server=SRRZ2EXC01,RequestId=22ee77d8-ec52-42f2- 
   a1b5-5a9f69f832cb,TimeStamp=23.09.2016 13:54:39] [FailureCategory=Cmdlet-A  
  dminAuditLogSearchException] 5C16D6DF,Microsoft.Exchange.Management.System   
 ConfigurationTasks.SearchAdminAuditLog

We just setup the TA and corresponding App. Is there anyone else that faced this issue and could resolve it?

0 Karma

Contributor

I want to mention that we had issues when deploying more than one TA to a single host which housed multiple Exchange roles. We're now testing in an Exchange 2013 environment and got the audit logs working on ONE server, but I am working with support to identify why it isn't working on 4 or 5 other hosts.

If you have an Exchange host with mixed roles and you plan on deploying (for instance) TA-Exchange-Mailbox and TA-Exchange-ClientAccess, both TAs have the same scripted input (read-audit-logs20102013.ps1), and each script writes a CliXml output to to a single file to "C:\Windows\Temp\splunk-msexchange-auditfile.clixml". If both TAs are deployed, and both inputs are enabled, you'll come across issues while each TA attempts to run the script and write to that file that the other TA has locked. Since they're both the same script, disable the scripted input on one TA and leave it enabled on the other.

I would attempt to run "Search-AdminAuditLog" manually in the Exchange Powershell console to see if it's working as intended. Perhaps you can enable DEBUG logging for ExecProcessor on that host to try and identify more information. It's helped me a little in resolving my issue (which is not yet completely resolved :)).

0 Karma

Explorer

I also enabled debug logging for the ExecProcessor in the meantime - but I cannot get more information out of the logfile - maybe someone can assist here?

Unfortunatly I am not allowed to upload the logfile since I do not have enough karma...

0 Karma

Contributor

Unfortunately I still have an open ticket with Splunk for this issue. Have you opened one? We've been working on it for a couple of months off and on, but if I end up getting a resolution then I will for sure pass it on.

0 Karma

Explorer

No, we are not allowed to open a case since we are in the middle of a POC and do not yet have a valid contract.

So what, I found out in the meantime, that everytime one of the following script is running

  • read-audit-logs20102013.ps1
  • read-mailbox-audit-logs20102013.ps1
  • get-inboxrules20102013.ps1

I do receive an Audit Failure directly in the Windows Security log containing this information:

An account failed to log on.

Subject:
    Security ID:        DOMAIN\USERNAME
    Account Name:       USERNAME
    Account Domain:     DOMAIN
    Logon ID:       0xEE89D66

Logon Type:         5

Account For Which Logon Failed:
    Security ID:        NULL SID
    Account Name:       -
    Account Domain:     -

Failure Information:
    Failure Reason:     An Error occured during Logon.
    Status:         0xC0000022
    Sub Status:     0xC0000022

Process Information:
    Caller Process ID:  0x6ccc
    Caller Process Name:    C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

Network Information:
    Workstation Name:   -
    Source Network Address: -
    Source Port:        -

Detailed Authentication Information:
    Logon Process:      Advapi  
    Authentication Package: Negotiate
    Transited Services: -
    Package Name (NTLM only):   -
    Key Length:     0

Where DOMAIN and USERNAME stands for the Domainservice Account which is running the SplunkUniversalFowareder Service.

But I am 110% sure that it is not a permission issue for the following reasons:

  • When I logon with the Service Account I can run all commands which are not working in the Scripts
  • Some of the Powershell Scripts are working

Maybe you can see something similar in your environment?

0 Karma

Explorer

we have the same issue. Were you able to resolve it and if yes, what is the solution?

0 Karma

Contributor

Not yet unfortunately, we put off implementing this until February 😞

0 Karma

Explorer

As you mentioned our Exchange Servers have multiple roles - so we had the issue you said, that there was an conflict between the TA's. In the meantime we made sure that there are no more conflicts by disabling the scripts where required but that did not change a lot - since we will see no data comming into Splunk.

What I can see in the TEMP directory is two XML Files

  • splunk-msexchange-auditfile.clixml
  • splunk-msexchange-mailboxauditlogs.clixml

but both of them do just have a basic XML structure with no content.

I also tested to run "Search-AdminAuditLog" and "Search-MailboxAuditLog" manually from Exchange Powershell console - both commands give me the output that we want to see.

Do you may have any other ideas?

0 Karma

New Member

LOGbinder for Exchange can handle this for you. It will get both the admin and mailbox audit logs in to Splunk for you. There is also a Splunk App for LOGbinder that they have created. There is a Splunk page on their site here: https://www.logbinder.com/Partners/Splunk

0 Karma

Splunk Employee
Splunk Employee

I downvoted this post because this is borderline spam, and the post does not add value besides the link to a 3rd party solution to which the poster has a relationship (which would be fine if the answer actually added more value). poster has 1 post, and has not been active since 2015.

To be clear, we love the ecosystem. I would not consider this a black mark against the vendor in question, they may have some compelling stuff to offer.

Contributor

Is this not a feature of the Splunk App for Exchange?

0 Karma

New Member

Exchange has a variety of logs. I believe the Splunk app for Exchange 3.0 will get you loads of info it gathers from the environment. Unfortunately, with Exchange, the security audit logs for Mailbox auditing are stored within Exchange, inaccessible to Splunk. This is where LOGbinder bridges that gap. LOGbinder gets the mailbox audit logs from Exchange where they are stored in each users mailbox, correlates guids and other useless unless translated data and outputs it to a location where Splunk can access it. Either in the Security log, application log, a file share or to a Syslog receiver. LOGbinder does the same for the admin audit log.

Editted

0 Karma