Hi, we would like to monitor authentication attempts in our SMTP server (Exchange 2016) but I could not find a way to do so. We already have IT Essentials with the Exchange Content Pack but SMTP does not seem to be covered. Then I took a look into the "Splunk Add-on for Microsoft Exchange" which contains the TA-Exchange-HubTransport which monitors the following path but I checked those logs on my server and they did not contain any authentication event.  [installation drive] :\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking Have anyone managed to find those events and ingest them in Splunk somehow? Our final goal is to find public IPs trying to brute force our SMTP server. Thanks a lot. ... View more

Hi, I have Splunk 8.1.4 with Splunk Add-on for CISCO ESA 1.5.0. I also have the old  app Cisco Secuirty Suite that even though it does not support Splunk 8.1.4, it shows results so I planned to get inspired by its query for Message tracking--> Transaction details.   index=cisco eventtype=cisco-esa | transaction keepevicted=true icid mid | search host="*" sender="*" recipient="*" dest_interface="*" policy_direction="*" subject="*TEST*"   I think the ESA events are getting correctly to Splunk, I use Syslog Connector for Splunk.  The test I performed is the following : 1. send an email from my corporate email to GMAIL with the subject TEST 2. simply reply from gmail. With the above query I would expect to see two events but I only see the outgoing event. I tried to filter by recipient and it thrown zero results.   index=cisco eventtype=cisco-esa | transaction keepevicted=true icid mid | search host="*" sender="*" recipient="xxxx@yyy.zz" dest_interface="*" policy_direction="*"   If I do a simpler query without the transaction command, I can see an event with the right internal recipient which corresponds to the incoming email that I could not find previously. But in that event there is no field subject.   index=cisco eventtype=cisco-esa recipient="xxxx@yyy.zz"   Could someone help me out with some query that consolidate inbound /outbound emails with filtering capabilities? thanks ... View more

Hi, I went to the search of my own app I created a extracted field using the wizard.  Once created, I go to Settings--> Fields extractions I can see the extracted field , type inline, assigned to my app , enabled and with permissions on the App for everyone read and write. then I go to my app once again and I perform a simple query in verbose mode. To be sure I also click on All fields to be sure that all fields are actually shown    index=cisco sourcetype="cisco:esa:amp"   Unfortunately the extracted field does not show on the list. any idea what I am missing? many thanks   ... View more

Good morning, I have an app that is currently deployed in several servers via deployment manager. Recently we install a new server and I add it to a specific server class in order to receive a small app. For some reason, the app does not get deployed but others server classes do get deployed. The error in the splunkd.log file is the following 06-20-2022 12:08:43.952 +0200 INFO DeployedApplication [7180 HttpClientPollingThread_37C53C58-43DD-4D11-87C4-6C43DC08B1BB] - Checksum mismatch 0 <> 6534619572757127978 for app=Splunk_TA_windows - Process terminated. Will reload from='splunk.xxx.yyy.zzz:8089/services/streams/deployment?name=default:Process%20Termination:Splunk_TA_windows%20-%20Process%20terminated' 06-20-2022 12:08:44.154 +0200 ERROR HttpClientRequest [7180 HttpClientPollingThread_37C53C58-43DD-4D11-87C4-6C43DC08B1BB] - HTTP client error=Connection closed by peer while accessing server=https://splunk.xxx.yyy.zzz:8089 for request=https://splunk.xxx.yyy.zzz:8089/services/streams/deployment?name=default:Process%20Termination:Splunk_TA_windows%20-%20Process%20terminated. 06-20-2022 12:08:44.154 +0200 WARN HTTPClient [7180 HttpClientPollingThread_37C53C58-43DD-4D11-87C4-6C43DC08B1BB] - Download of file C:\Program Files\SplunkUniversalForwarder\var\run\Process Termination\Splunk_TA_windows - Process terminated-1655717532.bundle failed with status 502 06-20-2022 12:08:44.154 +0200 WARN DeployedApplication [7180 HttpClientPollingThread_37C53C58-43DD-4D11-87C4-6C43DC08B1BB] - Problem downloading from uri=splunk.xxx.yyy.zzz:8089 to path='/services/streams/deployment?name=default:Process%20Termination:Splunk_TA_windows%20-%20Process%20terminated' 06-20-2022 12:08:44.155 +0200 ERROR DeployedServerclass [7180 HttpClientPollingThread_37C53C58-43DD-4D11-87C4-6C43DC08B1BB] - name=Process Termination Failed to download app=Splunk_TA_windows - Process terminated 06-20-2022 12:08:44.174 +0200 WARN DC:DeploymentClient [7180 HttpClientPollingThread_37C53C58-43DD-4D11-87C4-6C43DC08B1BB] - Restarting Splunkd... could someone tell me what is happening? the same app is already deployed in other servers without any issue. thanks   ... View more

Hi, Trying to correlate failed logon attempts (event 4776) with the IIS OWA logs, I realized that the OWA logs are in UTC by default and I am in CEST time (Madrid). According to the official documentation    To configure time zone settings, edit the props.conf file in $FORWARDER_HOME/etc/system/local/ or in your own custom application directory in $FORWARDER_HOME/etc/apps/.   https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Applytimezoneoffsetstotimestamps I deployed several apps in the exchange server but onle one app is reporting wrongly , called TA-Windows-Exchange-IIS. So I only need to change the timezone in that specific app if I understood correctly. And this is what I did, creating the file props.conf in the local path of the app. C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Windows-Exchange-IIS\local   [monitor://C:\inetpub\logs\LogFiles\W3SVC1\*.log] TZ = UTC [monitor://E:\Program Files\Microsoft\Exchange Server\V15\Logging\Ews] TZ = UTC      I restarted the splunkforwarder service just in case. The result is that the time is still wrongly taken from those exchange events, in UTC. Any idea on what I am doing wrong? thanks a lot. ... View more

Hi, I am trying to create a simple app to onboard data from THOR application. First I deployed the UF in my W10 and I see the client in the forwarder management console. phone home 1 second ago. After I create a new app (barebones) and I create a simple inputs.conf in the local folder. Then, that I moved the folder from apps folder to deployment-apps folder. finally I created a server class containing only my computer and assigned the new app. The final result is that the application is never deployed. the UI shows a deployment error but not other information is shown. Another weird issue is that I dont find the UF service on my workstation, I have the path and I can open a CMD and restart splunk but for some reason there is no visible service in the services console. Could be that both issues are related? Could someone tell me what could be the reason? Thanks a lot.   ... View more

Hi, I am trying to configure PaloAlto logs via the Splunk Connect for Syslog. I followed the instructions here  https://splunk.github.io/splunk-connect-for-syslog/main/sources/PaloaltoNetworks/ I configured the syslog at PaloAlto according the instructions. I also c I can see the syslog connections arriving to the host from the firewall using the command tcpdump port 514. Add the following lines to splunk_metadata.csv   pan_config,index,test pan_correlation,index,test pan_globalprotect,index,test pan_hipmatch,index,test pan_log,index,test pan_system,index,test pan_threat,index,test pan_traffic,index,test pan_userid,index,test       And restart sc4s   systemctl restart sc4s   I checked the index test and it is empty. I enabled the debug by adding with the line in the env_file   SC4S_DEST_GLOBAL_ALTERNATES=d_hec_debug   and it seems like the index defined in the spplunk_metadata.csv is not taken, instead osnix is used.   curl -k -u "sc4s HEC debug:$SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN" "https://splunk.XX.XXX.XXu:8088/services/collector/event" -d '{"time":"1643726324.000","sourcetype":"nix:syslog","source":"program:","index":"osnix","host":"atlas-fw-01.XXX.XX.XX","fields":{"sc4s_vendor_product":"nix_syslog","sc4s_syslog_severity":"info","sc4s_syslog_format":"rfc5424_strict","sc4s_syslog_facility":"user","sc4s_proto":"UDP","sc4s_loghost":"xxxxxxxxxx","sc4s_fromhostip":"192.168.10.100","sc4s_destport":"514","sc4s_container":"xxxxxxxx"},"event":"2022-02-01T14:38:44.000+00:00 atlas-fw-01.xxx.xxx.xxx - - - - 1,2022/02/01 15:38:43,011901021137,TRAFFIC,end,2561,2022/02/01 15:38:43,192.168.20.63,157.240.27.54,154.14.118.254,157.240.27.54,Normal traffic,xxx\\yyy,,quic,vsys1,Internal,External,ae1,ae2.6,Splunk,2022/02/01 15:38:43,113676,1,56081,443,49985,443,0x400019,udp,allow,7358,2250,5108,19,2022/02/01 15:36:43,0,any,,7030011678692056750,0x0,192.168.0.0-192.168.255.255,Germany,,7,12,aged-out,0,0,0,0,,atlas-fw-01,from-policy,,,0,,0,,N/A,0,0,0,0,c8250554-4ccd-46e3-8498-e74cfe9cdd10,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2022-02-01T15:38:44.130+01:00,,,infrastructure,networking,browser-based,1,tunnel-other-application,,quic,no,no,0"}'    I already check and the HEC token is allowed to index test. Could someone tell me what is happening? thanks ... View more

Hi, I have configured IT Essential Works (4.9.2) with Exchange content pack (1.4.3) and  TA-Exchange-ClientAccess (4.0.3). By chance I was checking PowerShell event logs in our exchange server and I saw the error bellow. Log Name: Microsoft-Windows-PowerShell/Operational Source: Microsoft-Windows-PowerShell Date: 29/11/2021 11:34:13 Event ID: 4100 Task Category: Executing Pipeline Level: Warning Keywords: None User: SYSTEM Computer: XXXX.YYYY.ZZZZ Description: Error Message = Object reference not set to an instance of an object. Fully Qualified Error ID = System.NullReferenceException,Microsoft.Exchange.Management.SystemConfigurationTasks.SearchAdminAuditLog Context: Severity = Warning Host Name = ConsoleHost Host Version = 5.1.14393.4583 Host ID = 644d49a8-7f8f-4b1e-9250-959ff1a8b7b4 Host Application = Powershell -PSConsoleFile E:\Program Files\Microsoft\Exchange Server\V15\\bin\exshell.psc1 -command . 'C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1' Engine Version = 5.1.14393.4583 Runspace ID = 10a2c198-89dd-47c1-99c1-4d493d35a837 Pipeline ID = 1 Command Name = Search-AdminAuditLog Command Type = Cmdlet Script Name = C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1 Command Path = Sequence Number = 19 User = XXXXX\SYSTEM Connected User = Shell ID = Microsoft.PowerShell User Data: Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-PowerShell" Guid="{A0C1853B-5C40-4B15-8766-3CF1C58F985A}" /> <EventID>4100</EventID> <Version>1</Version> <Level>3</Level> <Task>106</Task> <Opcode>19</Opcode> <Keywords>0x0</Keywords> <TimeCreated SystemTime="2021-11-29T10:34:13.679546900Z" /> <EventRecordID>5879670</EventRecordID> <Correlation ActivityID="{2DF7DE6F-E0AC-000E-036D-F92DACE0D701}" /> <Execution ProcessID="62888" ThreadID="41736" /> <Channel>Microsoft-Windows-PowerShell/Operational</Channel> <Computer>XXXXXX.YYYY.ZZZZ</Computer> <Security UserID="S-1-5-18" /> </System> <EventData> <Data Name="ContextInfo"> Severity = Warning Host Name = ConsoleHost Host Version = 5.1.14393.4583 Host ID = 644d49a8-7f8f-4b1e-9250-959ff1a8b7b4 Host Application = Powershell -PSConsoleFile E:\Program Files\Microsoft\Exchange Server\V15\\bin\exshell.psc1 -command . 'C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1' Engine Version = 5.1.14393.4583 Runspace ID = 10a2c198-89dd-47c1-99c1-4d493d35a837 Pipeline ID = 1 Command Name = Search-AdminAuditLog Command Type = Cmdlet Script Name = C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-ClientAccess\bin\powershell\read-audit-logs_2010_2013.ps1 Command Path = Sequence Number = 19 User = XXXXXXX\SYSTEM Connected User = Shell ID = Microsoft.PowerShell </Data> <Data Name="UserData"> </Data> <Data Name="Payload">Error Message = Object reference not set to an instance of an object. Fully Qualified Error ID = System.NullReferenceException,Microsoft.Exchange.Management.SystemConfigurationTasks.SearchAdminAuditLog </Data> </EventData> </Event>   Log Name: Microsoft-Windows-PowerShell/Operational Source: Microsoft-Windows-PowerShell Date: 29/11/2021 11:34:09 Event ID: 4104 Task Category: Execute a Remote Command Level: Warning Keywords: None User: SYSTEM Computer: XXXXX.YYYY.ZZZZ Description: Creating Scriptblock text (1 of 1): ######################################################## # # Splunk for Microsoft Exchange # Exchange 2010/2013 Mailbox Store Data Definition # # Copyright (C) 2005-2021 Splunk Inc. All Rights Reserved. # All Rights Reserved # ######################################################## # # This returns the filename of the audit database - due to some # funkiness with permissions, deployment server and the local # directory, we're using the %TEMP% as the location. For the # NT Authority\SYSTEM account, this is normally C:\Windows\Temp # $AuditTempFile = $ENV:Temp | Join-Path -ChildPath "splunk-msexchange-mailboxauditlogs.clixml" [Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8 $AuditDetails = @{} if (Test-Path $AuditTempFile) { $AuditDetails = Import-CliXml $AuditTempFile } # # Given a single audit record from the Search-MailboxAuditLog function Output-AuditRecord($Record) { $O = New-Object System.Collections.ArrayList $D = Get-Date $Record.LastAccessed -format 'yyyy-MM-ddTHH:mm:sszzz' [void]$O.Add($D) foreach ($p in $Record.PSObject.Properties) { [void]$O.Add("$($p.Name)=`"$($Record.PSObject.Properties[$p.Name].Value)`"") } Write-Host ($O -join " ") } function Output-AuditLog($Mailbox) { $Identity = $Mailbox.Identity $IdentityStr = $Identity.ToDNString() $LastSeen = (Get-Date).AddMonths(-1) if ($AuditDetails.ContainsKey($Identity)) { $LastSeen = $AuditDetails[$Identity] $AuditDetails.Remove($Identity) $AuditDetails[$IdentityStr] = $LastSeen } elseif ($AuditDetails.ContainsKey($IdentityStr)) { $LastSeen = $AuditDetails[$IdentityStr] } $LastRecord = $LastSeen Search-MailboxAuditLog -Identity $Identity -LogonTypes Owner,Delegate,Admin -ShowDetails -StartDate $LastSeen -EndDate (Get-Date)| sort LastAccessed | Foreach-Object { if ($_.LastAccessed -gt $LastSeen) { Output-AuditRecord($_) } $LastRecord = $_.LastAccessed } $AuditDetails[$IdentityStr] = $LastRecord } $Mailboxes = Get-Mailbox -Filter { AuditEnabled -eq $true } -Server $Env:ComputerName -ResultSize Unlimited $Mailboxes | Foreach-Object { If($_ -ne $null) { Output-AuditLog($_) }} # # Now that we have done the work, save off the Audit Temp File $AuditDetails | Export-CliXml $AuditTempFile ScriptBlock ID: 1e79b015-daf6-40c2-8c87-fedde7b4a866 Path: C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-Mailbox\bin\powershell\read-mailbox-audit-logs_2010_2013.ps1 Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-PowerShell" Guid="{A0C1853B-5C40-4B15-8766-3CF1C58F985A}" /> <EventID>4104</EventID> <Version>1</Version> <Level>3</Level> <Task>2</Task> <Opcode>15</Opcode> <Keywords>0x0</Keywords> <TimeCreated SystemTime="2021-11-29T10:34:09.300998400Z" /> <EventRecordID>5879669</EventRecordID> <Correlation ActivityID="{2DF7DE6F-E0AC-0010-CA3C-F92DACE0D701}" /> <Execution ProcessID="38808" ThreadID="54116" /> <Channel>Microsoft-Windows-PowerShell/Operational</Channel> <Computer>XXXX.YYYY.ZZZ</Computer> <Security UserID="S-1-5-18" /> </System> <EventData> <Data Name="MessageNumber">1</Data> <Data Name="MessageTotal">1</Data> <Data Name="ScriptBlockText">######################################################## # # Splunk for Microsoft Exchange # Exchange 2010/2013 Mailbox Store Data Definition # # Copyright (C) 2005-2021 Splunk Inc. All Rights Reserved. # All Rights Reserved # ######################################################## # # This returns the filename of the audit database - due to some # funkiness with permissions, deployment server and the local # directory, we're using the %TEMP% as the location. For the # NT Authority\SYSTEM account, this is normally C:\Windows\Temp # $AuditTempFile = $ENV:Temp | Join-Path -ChildPath "splunk-msexchange-mailboxauditlogs.clixml" [Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8 $AuditDetails = @{} if (Test-Path $AuditTempFile) { $AuditDetails = Import-CliXml $AuditTempFile } # # Given a single audit record from the Search-MailboxAuditLog function Output-AuditRecord($Record) { $O = New-Object System.Collections.ArrayList $D = Get-Date $Record.LastAccessed -format 'yyyy-MM-ddTHH:mm:sszzz' [void]$O.Add($D) foreach ($p in $Record.PSObject.Properties) { [void]$O.Add("$($p.Name)=`"$($Record.PSObject.Properties[$p.Name].Value)`"") } Write-Host ($O -join " ") } function Output-AuditLog($Mailbox) { $Identity = $Mailbox.Identity $IdentityStr = $Identity.ToDNString() $LastSeen = (Get-Date).AddMonths(-1) if ($AuditDetails.ContainsKey($Identity)) { $LastSeen = $AuditDetails[$Identity] $AuditDetails.Remove($Identity) $AuditDetails[$IdentityStr] = $LastSeen } elseif ($AuditDetails.ContainsKey($IdentityStr)) { $LastSeen = $AuditDetails[$IdentityStr] } $LastRecord = $LastSeen Search-MailboxAuditLog -Identity $Identity -LogonTypes Owner,Delegate,Admin -ShowDetails -StartDate $LastSeen -EndDate (Get-Date)| sort LastAccessed | Foreach-Object { if ($_.LastAccessed -gt $LastSeen) { Output-AuditRecord($_) } $LastRecord = $_.LastAccessed } $AuditDetails[$IdentityStr] = $LastRecord } $Mailboxes = Get-Mailbox -Filter { AuditEnabled -eq $true } -Server $Env:ComputerName -ResultSize Unlimited $Mailboxes | Foreach-Object { If($_ -ne $null) { Output-AuditLog($_) }} # # Now that we have done the work, save off the Audit Temp File $AuditDetails | Export-CliXml $AuditTempFile </Data> <Data Name="ScriptBlockId">1e79b015-daf6-40c2-8c87-fedde7b4a866</Data> <Data Name="Path">C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-Exchange-Mailbox\bin\powershell\read-mailbox-audit-logs_2010_2013.ps1</Data> </EventData> </Event>   Any idea on what could be happening? I found some answer related to the user that is running the universal splunk forwarder in the server. It is currently configured as local SYSTEM and in some answers, found on internet, it was mentioned to use a domain account with exchange permissions. But I check the official documentation and I could not find any mention to it. Thanks.   ... View more

Hi, recently we deployed IT Essential Works with latest Exchange Content Pack. we also deployed the three addons for the Exchange  in the exchange nodes (including IIS and OWA logs). Now we are in the process of validation of the ITSI dashboards, External Logins Map is one of them, and we realized that the extracted source IP (c_ip field) corresponds to our load balancer (XXX.XXX.XXX.XXX) instead of the remote host (IP shown at the end of the event). below an example of exchange event that reach our splunk infra.     2021-10-08 12:22:31 XXX.XXX.XXX.XXX POST /Microsoft-Server-ActiveSync/default.eas Cmd=Ping&User=---%5n---&DeviceId=-------&DeviceType=Outlook&CorrelationID=<empty>;&cafeReqId=c586f22d-14cd-4449-be95-fe666b30c92e; 443 -------\----- 192.168.X.X Outlook-iOS-Android/1.0 - 200 0 0 181382 52.98.193.109     we use the official TA-Exchange-2013-Mailbox, TA-Exchange-ClientAccess and Ta-Windows-Exchange-IIS addons. I found the definition of c_ip field in the transform.conf and props.conf in  the TA-Windows-Exchange-IIS, but I dont see any specific regex for its correct extraction.  could someone tell me how to proceed to fix this parsing issue so the dashboards can show correct information? many thanks ... View more