Still a pretty new Splunker here -- We are testing the Splunk App for Exchange in our dev environment. I've followed the documentation pretty closely (we did not install a Universal Forwarder on our Hub Transport device, we aren't interested in message tracking). We've installed UFs on our MailboxStore and CAS boxes. This is an Exchange 2010 environment.
We are currently only getting events for the following sourcetypes from this configuration:
We are looking to track admin accesses and changes to mailboxes and non-owner mailbox access in our environment. I figure the powershell scripts "read-audit-logs" and "read-mailbox-audit-logs" are responsible for pulling these events into Splunk, but they don't seem to be running. The Exchange admin can run Exchange PowerShell cmdlets on the server itself such as Search-MailboxAuditLog and we can see the events for the actions he's making. But these events are not sending over to Splunk. inputs.conf has been left default and the stanzas for the sourcetypes MSExchange:2010:AdminAudit and MSExchange:2010:MailboxAudit both have the setting disabled=false.
Any ideas on where to look to get these logs into Splunk?
We do have the same issue with an exchange server 2013. Other powershell scripts run great, but the following scripts fail:
When we check the logs in the Splunk forwarder we see following entry repeating every 5min:
Location of the log file: C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log
09-23-2016 15:54:40.918 +0200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\exchangepowershell.cmd" v15 read-audit-logs_2010_2013.ps1" Search-AdminAuditLog : The attempt to search the administrator audit log failed. Please try again later. At C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\ bin\powershell\read-audit-logs_2010_2013.ps1:81 char:12 + $Records = Search-AdminAuditLog -StartDate $LastSeen + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Search-AdminAuditLog], AdminA uditLogSearchException + FullyQualifiedErrorId : [Server=SRRZ2EXC01,RequestId=22ee77d8-ec52-42f2- a1b5-5a9f69f832cb,TimeStamp=23.09.2016 13:54:39] [FailureCategory=Cmdlet-A dminAuditLogSearchException] 5C16D6DF,Microsoft.Exchange.Management.System ConfigurationTasks.SearchAdminAuditLog
We just setup the TA and corresponding App. Is there anyone else that faced this issue and could resolve it?
I want to mention that we had issues when deploying more than one TA to a single host which housed multiple Exchange roles. We're now testing in an Exchange 2013 environment and got the audit logs working on ONE server, but I am working with support to identify why it isn't working on 4 or 5 other hosts.
If you have an Exchange host with mixed roles and you plan on deploying (for instance) TA-Exchange-Mailbox and TA-Exchange-ClientAccess, both TAs have the same scripted input (read-audit-logs_2010_2013.ps1), and each script writes a CliXml output to to a single file to "C:\Windows\Temp\splunk-msexchange-auditfile.clixml". If both TAs are deployed, and both inputs are enabled, you'll come across issues while each TA attempts to run the script and write to that file that the other TA has locked. Since they're both the same script, disable the scripted input on one TA and leave it enabled on the other.
I would attempt to run "Search-AdminAuditLog" manually in the Exchange Powershell console to see if it's working as intended. Perhaps you can enable DEBUG logging for ExecProcessor on that host to try and identify more information. It's helped me a little in resolving my issue (which is not yet completely resolved :)).
I also enabled debug logging for the ExecProcessor in the meantime - but I cannot get more information out of the logfile - maybe someone can assist here?
Unfortunatly I am not allowed to upload the logfile since I do not have enough karma...
Unfortunately I still have an open ticket with Splunk for this issue. Have you opened one? We've been working on it for a couple of months off and on, but if I end up getting a resolution then I will for sure pass it on.
No, we are not allowed to open a case since we are in the middle of a POC and do not yet have a valid contract.
So what, I found out in the meantime, that everytime one of the following script is running
I do receive an Audit Failure directly in the Windows Security log containing this information:
An account failed to log on. Subject: Security ID: DOMAIN\USERNAME Account Name: USERNAME Account Domain: DOMAIN Logon ID: 0xEE89D66 Logon Type: 5 Account For Which Logon Failed: Security ID: NULL SID Account Name: - Account Domain: - Failure Information: Failure Reason: An Error occured during Logon. Status: 0xC0000022 Sub Status: 0xC0000022 Process Information: Caller Process ID: 0x6ccc Caller Process Name: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Network Information: Workstation Name: - Source Network Address: - Source Port: - Detailed Authentication Information: Logon Process: Advapi Authentication Package: Negotiate Transited Services: - Package Name (NTLM only): - Key Length: 0
Where DOMAIN and USERNAME stands for the Domainservice Account which is running the SplunkUniversalFowareder Service.
But I am 110% sure that it is not a permission issue for the following reasons:
Maybe you can see something similar in your environment?
As you mentioned our Exchange Servers have multiple roles - so we had the issue you said, that there was an conflict between the TA's. In the meantime we made sure that there are no more conflicts by disabling the scripts where required but that did not change a lot - since we will see no data comming into Splunk.
What I can see in the TEMP directory is two XML Files
but both of them do just have a basic XML structure with no content.
I also tested to run "Search-AdminAuditLog" and "Search-MailboxAuditLog" manually from Exchange Powershell console - both commands give me the output that we want to see.
Do you may have any other ideas?
LOGbinder for Exchange can handle this for you. It will get both the admin and mailbox audit logs in to Splunk for you. There is also a Splunk App for LOGbinder that they have created. There is a Splunk page on their site here: https://www.logbinder.com/Partners/Splunk
I downvoted this post because this is borderline spam, and the post does not add value besides the link to a 3rd party solution to which the poster has a relationship (which would be fine if the answer actually added more value). poster has 1 post, and has not been active since 2015.
To be clear, we love the ecosystem. I would not consider this a black mark against the vendor in question, they may have some compelling stuff to offer.
Exchange has a variety of logs. I believe the Splunk app for Exchange 3.0 will get you loads of info it gathers from the environment. Unfortunately, with Exchange, the security audit logs for Mailbox auditing are stored within Exchange, inaccessible to Splunk. This is where LOGbinder bridges that gap. LOGbinder gets the mailbox audit logs from Exchange where they are stored in each users mailbox, correlates guids and other useless unless translated data and outputs it to a location where Splunk can access it. Either in the Security log, application log, a file share or to a Syslog receiver. LOGbinder does the same for the admin audit log.