Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

[coltwanger]

I've deployed the Windows DNS Analytical and Diagnostic Logs add-on to our DNS servers, but the PowerShell script returns the following error:

ERROR ExecProcessor - message from "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command "& 'C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-windnsanalytical\bin\get_dns_analytics.ps1'"" Get-WinEvent : The requested operation cannot be performed over an enabled direct channel. The channel must first be disabled before performing the requested operation.

I suppose this is normal with ETL logs. The fix appears to be to let the log fill up and clear it manually, or let it roll off into archived files, which will eventually need to be deleted manually. This doesn't help the fact that Splunk still wouldn't be able to monitor the active ETL analytics log while it's being written to.

So I guess my option is to roll the log into archive files, and monitor the archive files (a batch input maybe)? How is everyone else monitoring the Analytics log? We really don't want to enable debug logging on our DNS servers due to the performance hit. It seems like this add-on can only work if the retention on the DNS Analytics log is configured to "Do not automatically overwrite events".

[mpasha]

So here is how i do it:
first of all i have configured the log for rotation, although you can not actively see the logs in the event viewer but it is perfectly fine and it is still collecting logs.
i have a powershell script that goes over the log "scheduled to run every 5 minutes" and dump/append the logs for the past 5 minute to a CSV file. I am only looking for event ID 257 and 256 i also have a regex built into the script that will discard the "packet data" which is part of the event message. this will help dramatically controlling the amount of data fed into splunk.
the rest is easy, create an index and feed the CSV to Splunk!!
here is the powershell script i am using :
i am not sure why but when adding the code here; all "$_" are replaced with "$" just remember to replace them back.


$SC_start=Get-Date
$CSV_fileName="C:\dnsanalytics\DNSANALYTICALS-"+(Get-Date).ToString("yyyy-MM-dd")+".csv"
$test=get-winevent -Oldest -Path C:\Windows\System32\winevt\Logs\Microsoft-Windows-DNSServer%4Analytical.etl|Where-Object{$.ID -eq "257" -or $.ID -eq "256" -and $.TimeCreated -ge $SC_start.AddMinutes(-5)}|Select-Object @{Label = "Date";Expression = {$.TimeCreated}},
@{Label = "ID";Expression = {$.id}},
@{Label = "Message";Expression = {$.Message -replace ('((?<=;\s)(?=\bPacketData\b))[^"]+')}}|Export-Csv -append -Path $CSV_fileName -NoType


[hughkelley]

I haven't tested fully but I think this code will avoid the "requested operation cannot be performed over an enabled direct channel" issue. I would replace the code in get_dns_analytics.ps1 with this.

Adapted from a Technet post

$logName = 'Microsoft-Windows-DNSServer/Analytical'

$eventlogSettings = get-winevent -ListLog $logName
$logFileRoot = Split-Path $eventlogSettings.LogFilePath -Parent
$logFileA = [System.Environment]::ExpandEnvironmentVariables($eventlogSettings.LogFilePath)  # need to expand variables for later use by Get-WinEvent
$logFileB = "{0}\Microsoft-Windows-DNSServer%4Analytical.{1:yyyyMMddhhmmss}.etl" -f $logFileRoot,(Get-Date).ToUniversalTime()
$filterXPath = "*[System[EventID!=280] and EventData[Data[@Name='InterfaceIP']!='127.0.0.1']]"

# Rotate the log file
Write-Host "rotating out " $eventlogSettings.LogFilePath

Set-Service DNS -Status Paused
$eventlogSettings.IsEnabled = $false
$eventlogSettings.LogFilePath = $logFileB
$eventlogSettings.SaveChanges()

$eventlogSettings.IsEnabled = $true
$eventlogSettings.SaveChanges()
Set-Service DNS -Status Running

# Read events to STDOUT
Get-WinEvent -Oldest -Path $logFileA -FilterXPath $filterXPath | fl

# TODO delete $logFileA

[gjanders · Accepted solution]

Splunk Stream can do this work, we are using the stream forwarder on the non-windows platform and it's great for DNS traffic.

We are evaluating it's usage for windows as the documentation mentions that winpcap allows any local user to use it (not a Splunk issue but the Splunk Stream uses that library on Windows).

[robert_miller]

We are using Stream, but we noticed the logs are not capturing the originating source which is why we are looking at the PS scripts suggested in this thread. It makes it hard using these logs when you don't know which server is making the DNS request.

[coltwanger]

Stream worked beautifully -- thank you!

[gjanders]

Glad I was able to help! Have a good day.

[gawilliams]

We've tried using Splunk Stream for DNS logging (on DNS servers and domain controllers) but it appears the traffic volume is too great (~10k queries per second), and the universal forwarder appears to bottleneck the Streamfwd process, even with maxKBps=0 in the limits.conf file. Has anyone had such issues with Stream? I realize in cases when DNS is run on Linux, you can deploy the standalone Stream Forward binary so there's no potential limiting from a UF, but they don't support a separate install for Windows as of now. Curious if anyone has any details/thoughts.

[coltwanger]

Have you tried aggregation to see if you can reduce the amount of events being forwarded?

https://docs.splunk.com/Documentation/StreamApp/7.0.1/User/StreamAggregationMethods

[gjanders]

Splunk Stream can do this work, we are using the stream forwarder on the non-windows platform and it's great for DNS traffic.

We are evaluating it's usage for windows as the documentation mentions that winpcap allows any local user to use it (not a Splunk issue but the Splunk Stream uses that library on Windows).

[coltwanger]

I have never even considered this option -- thanks for bringing it up, I'll be looking into it 🙂

[gjanders]

I've moved my comment into the answers section as it is effectively a possible answer...