Did you happen to share that addon anywhere? ... View more

I saw your mention of dnslib and I skimmed the project but haven't had time to see what they are doing to decode. The dns-packet also looked promising -but neither can be called directly from Powershell. This article has no code, but does seem to explain the packet layout: Dissecting DNS Communications. ... View more

We can get the logs into Splunk. I'm particularly asking about the peculiar log format of the audit data. Audit event: audit_schema_version:1 event_time:2019-10-23 20:57:13.1337990 sequence_number:1 action_id:LGIF succeeded:false is_column_permission:false session_id:0 server_principal_id:0 database_principal_id:0 target_server_principal_id:0 target_database_principal_id:0 object_id:0 user_defined_event_id:0 transaction_id:0 class_type:LX duration_milliseconds:0 response_rows:0 affected_rows:0 client_ip:10.99.99.999 permission_bitmask:00000000000000000000000000000000 sequence_group_id:954EDDBB-F5A2-45B2-AAC5-2E9797EC3859 session_server_principal_name: server_principal_name:iamfake server_principal_sid: ... View more

Same challenge here, I had this add-on would have parsing rules for SQL audit data in Windows Event Logs. https://docs.splunk.com/Documentation/AddOns/released/MSSQLServer/Datatypes ... View more

I can understand why that might be the issue if I used the sendresults command inline, but I even see this error during a scheduled report. I would have expected it to use the same logic as the base "send email action", which I can schedule. from index=_internal sourcetype=sendresults:log 2019-08-07 12:50:21,859 ERROR invocation_id=1565182221.77:80484 invocation_type="action" msg="Could not send email" rcpt="a@xyz.com,h@xyz.com" error="(530, 'Authentication required', u'splunk@xyz.com')" ... View more

Very clever, @dstaulcu. I expanded on the SPL a bit to include a multi-value field with pipe delimiters. | stats count as field1 | eval field1="host1,1111,A|b;host2,2222,c|d;host3,3333," | eval field1=split(field1,";") | mvexpand field1 | rex field=field1 "(?<host>.*),(?<serial>.*),(?<other>.*)" | eval other=split(other,"|") | table host serial other ... View more

We had the same issue (and a few others improvements we wanted with that Add On) so we ended up bundling all of our fixes here: https://splunkbase.splunk.com/app/4300/ ... View more

The props.conf has regexes (written by the original author) to extract the fields. EXTRACT-win-dns-analytical-if-dest = InterfaceIP=(?<src>[^;]+);\s+Destination=(?<dest>[^;]+); EXTRACT-win-dns-analytical-if-src = InterfaceIP=(?<dest>[^;]+);\s+Source=(?<src>[^;]+); EXTRACT-win-dns-analytical-src-if = Source=(?<src>[^;]+);\s+InterfaceIP=(?<dest>[^;]+); .......... ... View more

I've pushed a "whitelist" (zone ignore) feature to the GitHub repo as well. At least in our environment, I'm burning up several GB per day indexing local and/or low-risk DNS queries, e.g. microsoft.com. ... View more

I'll try to share it via Splunkbase once I get one more tweak checked in. We burn up a lot of log volume on A lookups for microsoft.com (4x more than for our internal/authoritative zones). I want to get some sort of filter in place for one or more of: zones hosted on the DNS server an Alexa 100-like list (open to suggestions here) manual entries in the UI or script ... View more

I'm a novice at pushing to Github, but here goes. https://github.com/hkelley/TA-windnsanalytical/ Warning - I added an "init" script to run at UF startup. This may not meet your needs. ... View more

Here's a sample of the results from that approach. The two TimeCreated entries are at the front and back of the logs. As you can see, now that I'm using an "offline" copy of the file, there's ~100ms of pause but that seems to be the only data loss. TimeCreated : 11/1/2018 5:43:31 PM TimeCreated : 11/1/2018 5:43:43 PM LoggingPausedMs : 69 DataRetrievalMs : 1237 LogFileMaxBytes : 1073741824 LoggedBytes : 737280 LoggedRecs : 2088 LoggedTimespanSecs : 11.8507947 ScriptRunSecs : 5.8446792 TimeCreated : 11/1/2018 5:43:43 PM TimeCreated : 11/1/2018 5:45:09 PM LoggingPausedMs : 114 DataRetrievalMs : 22110 LogFileMaxBytes : 1073741824 LoggedBytes : 6139904 LoggedRecs : 16581 LoggedTimespanSecs : 85.8057772 ScriptRunSecs : 26.4272838 ... View more

I think/fear we're missing that whole window. My next move (in what seems like a never-ending saga of optimization) is to switch to the following method for clearing the log and then read/parse from that backup. This will probably drive up the "paused" counter but should increase my visibility overall. ClearLog (string logName, string backupPath); I'm open to other ideas. One other thing I've observed, these logs get really slow to read once they get too large. I'd like to have a "clear" command that runs whenever the UF starts. Otherwise, the script simply can't handle the pile of data that might be sitting there in the log. The original author of this TA doesn't seem to have it in GitHub/similar so I'm not sure how best to get all of our ideas back into the original work. ... View more

We currently run it every minute - and it sometimes runs as long as 20s. I think we're going to filter further (possibly not logging results that were returned from authoritative zones). In addition to the CPU load on our DCs we're chewing up quite a bit of license volume. ... View more

Good find. For what it's worth, we ultimately found that the performance of Get-WinEvent was terrible. The time spent in FormatDescription() seems to be the issue. We ultimately rewrote the script a bit to bypass it and do the formatting ourselves. We don't send the raw packet data to Splunk (didn't have time to work out the formatting for it). $logName = 'Microsoft-Windows-DNSServer/Analytical' $filterXPath = "*[System[EventID!=280] and EventData[Data[@Name='InterfaceIP']!='127.0.0.1']]" $eventlogSettings = get-winevent -ListLog $logName $logFile = [System.Environment]::ExpandEnvironmentVariables($eventlogSettings.LogFilePath) # expand the variables in the file path # Extract the DNS "message types" from the event provider into a sparse array (element number == event ID). This is used later for our lightweight message generation $prov = Get-WinEvent -ListProvider $eventlogSettings.OwningProviderName $messageTypes= new-object string[] 999 # no four-digit event IDs so we should be fine with 999 $prov.Events | %{ $description = $_.Description -replace ";\s+PacketData=%\d+", "" # remove packetdata (for now, too complicated to parse) $description = $description -replace "%(?<token>\d{1,2})", "{`${token}}" # convert for PS-based tokens $messageTypes[$_.Id] = $description } # Do not use Get-WinEvent to avoid performance overhead of FormatDescription() $events = @() $query = New-Object System.Diagnostics.Eventing.Reader.EventLogQuery($logFile,[System.Diagnostics.Eventing.Reader.PathType]::FilePath , $filterXPath); $reader = New-Object System.Diagnostics.Eventing.Reader.EventLogReader($query) while(($record = $reader.ReadEvent()) -ne $null) { # convert the raw data to the format, without relying on EventLogRecord.FormatDescription () $propVals=@($null) foreach($prop in $record.psbase.Properties) { $propVals += $prop.value } $record | Add-Member -MemberType NoteProperty -Name Message -Value ($messageTypes[$record.Id] -f $propVals) $events += $record } # emit for Splunk UF to parse $events | fl # Clear the log $logSize = $eventlogSettings.Filesize # before clearing $swLogPaused = [Diagnostics.Stopwatch]::StartNew() $eventlogSettings.IsEnabled = $false $eventlogSettings.SaveChanges() try { [System.Diagnostics.Eventing.Reader.EventLogSession]::GlobalSession.ClearLog($eventlogSettings.LogName) } catch [System.Management.Automation.MethodException] { # eat this exception. It says "The process cannot access the file because it is being used by another process" but it lies, the log is cleared } $eventlogSettings.IsEnabled = $true $eventlogSettings.SaveChanges() $swLogPaused.Stop() # calculate the run time $elapsedTimeSecs = (New-TimeSpan -Start (Get-Process -Id $pid).StartTime -End (Get-Date)).Seconds # Emit some performance stats [pscustomobject]@{ LoggingPausedMs=$swLogPaused.Elapsed.Milliseconds; LogFileMaxBytes=$eventlogSettings.MaximumSizeInBytes LoggedBytes=$logSize LoggedRecs=$events.Count LoggedTimespanSecs=(New-TimeSpan -Start $events[0].timecreated -End $events[-1].timecreated).Seconds ScriptRunSecs=$elapsedTimeSecs } | fl ... View more

Are you able to read the logs via PS without getting the "enabled channel" error? How/where does the CSV get purged? I took a similar approach with my mods (my log is set for retention) but I read the log in place and then pause it just long enough to clear it (and write the logs to STDOUT). Separate issue - performance of the Get-WinEvent cmdlet was terrible for us ( a few thousand records every minute) so I had to resort to reading via .NET classes and formatting my own string. Is there a github site for this TA so that we could merge back some of these fixes? ... View more

I haven't tested fully but I think this code will avoid the "requested operation cannot be performed over an enabled direct channel" issue. I would replace the code in get_dns_analytics.ps1 with this. Adapted from a Technet post $logName = 'Microsoft-Windows-DNSServer/Analytical' $eventlogSettings = get-winevent -ListLog $logName $logFileRoot = Split-Path $eventlogSettings.LogFilePath -Parent $logFileA = [System.Environment]::ExpandEnvironmentVariables($eventlogSettings.LogFilePath) # need to expand variables for later use by Get-WinEvent $logFileB = "{0}\Microsoft-Windows-DNSServer%4Analytical.{1:yyyyMMddhhmmss}.etl" -f $logFileRoot,(Get-Date).ToUniversalTime() $filterXPath = "*[System[EventID!=280] and EventData[Data[@Name='InterfaceIP']!='127.0.0.1']]" # Rotate the log file Write-Host "rotating out " $eventlogSettings.LogFilePath Set-Service DNS -Status Paused $eventlogSettings.IsEnabled = $false $eventlogSettings.LogFilePath = $logFileB $eventlogSettings.SaveChanges() $eventlogSettings.IsEnabled = $true $eventlogSettings.SaveChanges() Set-Service DNS -Status Running # Read events to STDOUT Get-WinEvent -Oldest -Path $logFileA -FilterXPath $filterXPath | fl # TODO delete $logFileA ... View more

Did you find a solution for reading the Microsoft-Windows-DNSServer/Analytical logs? It's my understanding from this article that the analytical log can't be read "online" if circular logging is enabled. Error when enabling Analytic or Debug event log: "The requested operation cannot be performed over an enabled direct channel One solution might be to switch the event log to manual clearing and configure the Splunk add-on to do that log clearing. I'm not sure if that's a feature of the add-on. ... View more

Does this work for "remote pulled" event logs as well? I've put the following in inputs.conf but it does not seem to be doing lookups. [default] evt_dc_name = evt_dns_name = [script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path] disabled = 0 index = eventlog_filtering_test evt_resolve_ad_obj = 1 # resolved GUIDs and SIDs in the event data ... View more

I'm still exploring this theory, but it seems like this doesn't match the events [source::WMI:WinEventLog:Security] but this does (specifying sourcetype, as opposed to source). [WMI:WinEventLog:Security] Does that sound right? ... View more