topic Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs? in All Apps and Add-ons

Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

coltwanger — Thu, 19 Jan 2017 21:56:06 GMT

I've deployed the Windows DNS Analytical and Diagnostic Logs add-on to our DNS servers, but the PowerShell script returns the following error:

ERROR ExecProcessor - message from "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command "& 'C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-windnsanalytical\bin\get_dns_analytics.ps1'"" Get-WinEvent : The requested operation cannot be performed over an enabled direct channel. The channel must first be disabled before performing the requested operation.

I suppose this is normal with ETL logs. The fix appears to be to let the log fill up and clear it manually, or let it roll off into archived files, which will eventually need to be deleted manually. This doesn't help the fact that Splunk still wouldn't be able to monitor the active ETL analytics log while it's being written to.

So I guess my option is to roll the log into archive files, and monitor the archive files (a batch input maybe)? How is everyone else monitoring the Analytics log? We really don't want to enable debug logging on our DNS servers due to the performance hit. It seems like this add-on can only work if the retention on the DNS Analytics log is configured to "Do not automatically overwrite events".

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

gjanders — Fri, 20 Jan 2017 07:50:15 GMT

Splunk Stream can do this work, we are using the stream forwarder on the non-windows platform and it's great for DNS traffic.

We are evaluating it's usage for windows as the documentation mentions that winpcap allows any local user to use it (not a Splunk issue but the Splunk Stream uses that library on Windows).

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

coltwanger — Fri, 20 Jan 2017 17:54:41 GMT

I have never even considered this option -- thanks for bringing it up, I'll be looking into it 🙂

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

gjanders — Sat, 21 Jan 2017 03:58:15 GMT

I've moved my comment into the answers section as it is effectively a possible answer...

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

gjanders — Sat, 21 Jan 2017 03:58:24 GMT

Splunk Stream can do this work, we are using the stream forwarder on the non-windows platform and it's great for DNS traffic.

We are evaluating it's usage for windows as the documentation mentions that winpcap allows any local user to use it (not a Splunk issue but the Splunk Stream uses that library on Windows).

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

coltwanger — Fri, 27 Jan 2017 01:00:45 GMT

Stream worked beautifully -- thank you!

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

gjanders — Fri, 27 Jan 2017 01:10:55 GMT

Glad I was able to help! Have a good day.

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

gawilliams — Fri, 24 Mar 2017 19:00:07 GMT

We've tried using Splunk Stream for DNS logging (on DNS servers and domain controllers) but it appears the traffic volume is too great (~10k queries per second), and the universal forwarder appears to bottleneck the Streamfwd process, even with maxKBps=0 in the limits.conf file. Has anyone had such issues with Stream? I realize in cases when DNS is run on Linux, you can deploy the standalone Stream Forward binary so there's no potential limiting from a UF, but they don't support a separate install for Windows as of now. Curious if anyone has any details/thoughts.

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

coltwanger — Fri, 24 Mar 2017 19:16:21 GMT

Have you tried aggregation to see if you can reduce the amount of events being forwarded?

https://docs.splunk.com/Documentation/StreamApp/7.0.1/User/StreamAggregationMethods

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

hughkelley — Tue, 29 Sep 2020 19:03:57 GMT

I haven't tested fully but I think this code will avoid the "requested operation cannot be performed over an enabled direct channel" issue. I would replace the code in get_dns_analytics.ps1 with this.

Adapted from a Technet post

$logName = 'Microsoft-Windows-DNSServer/Analytical'

$eventlogSettings = get-winevent -ListLog $logName
$logFileRoot = Split-Path $eventlogSettings.LogFilePath -Parent
$logFileA = [System.Environment]::ExpandEnvironmentVariables($eventlogSettings.LogFilePath)  # need to expand variables for later use by Get-WinEvent
$logFileB = "{0}\Microsoft-Windows-DNSServer%4Analytical.{1:yyyyMMddhhmmss}.etl" -f $logFileRoot,(Get-Date).ToUniversalTime()
$filterXPath = "*[System[EventID!=280] and EventData[Data[@Name='InterfaceIP']!='127.0.0.1']]"

# Rotate the log file
Write-Host "rotating out " $eventlogSettings.LogFilePath

Set-Service DNS -Status Paused
$eventlogSettings.IsEnabled = $false
$eventlogSettings.LogFilePath = $logFileB
$eventlogSettings.SaveChanges()

$eventlogSettings.IsEnabled = $true
$eventlogSettings.SaveChanges()
Set-Service DNS -Status Running

# Read events to STDOUT
Get-WinEvent -Oldest -Path $logFileA -FilterXPath $filterXPath | fl

# TODO delete $logFileA

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

mpasha — Tue, 29 Sep 2020 21:22:56 GMT

So here is how i do it:
first of all i have configured the log for rotation, although you can not actively see the logs in the event viewer but it is perfectly fine and it is still collecting logs.
i have a powershell script that goes over the log "scheduled to run every 5 minutes" and dump/append the logs for the past 5 minute to a CSV file. I am only looking for event ID 257 and 256 i also have a regex built into the script that will discard the "packet data" which is part of the event message. this will help dramatically controlling the amount of data fed into splunk.
the rest is easy, create an index and feed the CSV to Splunk!!
here is the powershell script i am using :
i am not sure why but when adding the code here; all "$_" are replaced with "$" just remember to replace them back.

$SC_start=Get-Date $CSV_fileName="C:\dnsanalytics\DNSANALYTICALS-"+(Get-Date).ToString("yyyy-MM-dd")+".csv" $test=get-winevent -Oldest -Path C:\Windows\System32\winevt\Logs\Microsoft-Windows-DNSServer%4Analytical.etl|Where-Object{$.ID -eq "257" -or $.ID -eq "256" -and $.TimeCreated -ge $SC_start.AddMinutes(-5)}|Select-Object @{Label = "Date";Expression = {$.TimeCreated}}, @{Label = "ID";Expression = {$.id}}, @{Label = "Message";Expression = {$.Message -replace ('((?<=;\s)(?=\bPacketData\b))[^"]+')}}|Export-Csv -append -Path $CSV_fileName -NoType

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

hughkelley — Sun, 07 Oct 2018 18:03:42 GMT

Are you able to read the logs via PS without getting the "enabled channel" error?

How/where does the CSV get purged?

I took a similar approach with my mods (my log is set for retention) but I read the log in place and then pause it just long enough to clear it (and write the logs to STDOUT).

Separate issue - performance of the Get-WinEvent cmdlet was terrible for us ( a few thousand records every minute) so I had to resort to reading via .NET classes and formatting my own string.

Is there a github site for this TA so that we could merge back some of these fixes?

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

mpasha — Tue, 09 Oct 2018 18:20:55 GMT

I have no issue reading the Events even when it is configured to overwrite the log.
it does not purge the CSV file, i use the Getdate function to append the date to the file so if the file exists the new data will be appended to the existing file and it will create a new file everyday.
you can use another process to deal with the older files. I personally delete the files older than 5 days.
to avoid the performance issue, i have scheduled it to run every 5 minutes and developed the PS script to read the file starting from 5 minutes ago.
it takes 60-70 seconds for our servers to append the new content for the past 5 minutes.

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

exel_wild — Tue, 30 Oct 2018 13:45:15 GMT

OK, so the difference here is:

Get-WinEvent -LogName 'Microsoft-Windows-DNSServer/Analytical' -Oldest

This will throw error about busy channel.
Get-WinEvent : The requested operation cannot be performed over an enabled direct channel. The channel must first be disabled before performing the requested operation
At line:1 char:2
+ Get-WinEvent -LogName 'Microsoft-Windows-DNSServer/Analytical' -Olde ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-WinEvent], EventLogException
+ FullyQualifiedErrorId : System.Diagnostics.Eventing.Reader.EventLogException,Microsoft.PowerShell.Commands.GetWinEventCommand

And code below would work perfectly fine:

Get-WinEvent -Path C:\Windows\System32\winevt\Logs\Microsoft-Windows-DNSServer%4Analytical.etl -Oldest

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

hughkelley — Tue, 30 Oct 2018 14:31:32 GMT

Good find.

For what it's worth, we ultimately found that the performance of Get-WinEvent was terrible. The time spent in FormatDescription() seems to be the issue. We ultimately rewrote the script a bit to bypass it and do the formatting ourselves. We don't send the raw packet data to Splunk (didn't have time to work out the formatting for it).

$logName = 'Microsoft-Windows-DNSServer/Analytical'
$filterXPath = "*[System[EventID!=280] and EventData[Data[@Name='InterfaceIP']!='127.0.0.1']]"

$eventlogSettings = get-winevent -ListLog $logName
$logFile = [System.Environment]::ExpandEnvironmentVariables($eventlogSettings.LogFilePath)  # expand the variables in the file path


# Extract the DNS "message types" from the event provider into a sparse array (element number == event ID).   This is used later for our lightweight message generation
$prov = Get-WinEvent -ListProvider $eventlogSettings.OwningProviderName 
$messageTypes= new-object string[] 999  # no four-digit event IDs so we should be fine with 999
$prov.Events | %{

    $description = $_.Description -replace ";\s+PacketData=%\d+", ""  # remove packetdata  (for now,  too complicated to parse)
    $description = $description -replace "%(?<token>\d{1,2})", "{`${token}}"   # convert for PS-based tokens
    $messageTypes[$_.Id]  = $description
}

# Do not use Get-WinEvent to avoid performance overhead of FormatDescription()

$events = @()

$query = New-Object System.Diagnostics.Eventing.Reader.EventLogQuery($logFile,[System.Diagnostics.Eventing.Reader.PathType]::FilePath , $filterXPath);
$reader = New-Object System.Diagnostics.Eventing.Reader.EventLogReader($query)
while(($record = $reader.ReadEvent()) -ne $null)
{
    # convert the raw data to the format, without relying on EventLogRecord.FormatDescription () 
    $propVals=@($null)
    foreach($prop in $record.psbase.Properties)
    {
        $propVals += $prop.value
    }

    $record | Add-Member -MemberType NoteProperty -Name Message -Value ($messageTypes[$record.Id] -f $propVals)

    $events += $record
}

# emit for Splunk UF to parse
$events | fl 

# Clear the log
$logSize = $eventlogSettings.Filesize  # before clearing
$swLogPaused = [Diagnostics.Stopwatch]::StartNew()
$eventlogSettings.IsEnabled = $false
$eventlogSettings.SaveChanges()
try
{
    [System.Diagnostics.Eventing.Reader.EventLogSession]::GlobalSession.ClearLog($eventlogSettings.LogName) 
}
catch  [System.Management.Automation.MethodException]
{ # eat this exception.   It says "The process cannot access the file because it is being used by another process" but it lies, the log is cleared    
}
$eventlogSettings.IsEnabled = $true
$eventlogSettings.SaveChanges()
$swLogPaused.Stop()

# calculate the run time
$elapsedTimeSecs = (New-TimeSpan -Start (Get-Process -Id $pid).StartTime  -End (Get-Date)).Seconds  

# Emit some performance stats
[pscustomobject]@{
    LoggingPausedMs=$swLogPaused.Elapsed.Milliseconds;
    LogFileMaxBytes=$eventlogSettings.MaximumSizeInBytes
    LoggedBytes=$logSize
    LoggedRecs=$events.Count
    LoggedTimespanSecs=(New-TimeSpan -Start $events[0].timecreated -End $events[-1].timecreated).Seconds
    ScriptRunSecs=$elapsedTimeSecs
}   | fl

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

robert_miller — Thu, 01 Nov 2018 18:13:23 GMT

Hey @hughkelley ,

Since you are clearing out the log file via this script, how often are you running this? (i.e. every x seconds, minute, five minutes, 10 minutes, etc.)

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

hughkelley — Thu, 01 Nov 2018 18:21:48 GMT

We currently run it every minute - and it sometimes runs as long as 20s. I think we're going to filter further (possibly not logging results that were returned from authoritative zones).

In addition to the CPU load on our DCs we're chewing up quite a bit of license volume.

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

robert_miller — Thu, 01 Nov 2018 19:04:05 GMT

We are using Stream, but we noticed the logs are not capturing the originating source which is why we are looking at the PS scripts suggested in this thread. It makes it hard using these logs when you don't know which server is making the DNS request.

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

robert_miller — Thu, 01 Nov 2018 19:10:44 GMT

When the script runs as long as 20s, how much logs do you think are possibly not being captured? I noticed in your script you have "LoggingPausedMs" which might help answer this question. I am trying to determine if rolling the log file vs pausing/clearing is the right solution.

Re: Windows DNS Analytical and Diagnostic Logs: How are others monitoring DNS Analytical Logs?

hughkelley — Thu, 01 Nov 2018 19:53:46 GMT

I think/fear we're missing that whole window. My next move (in what seems like a never-ending saga of optimization) is to switch to the following method for clearing the log and then read/parse from that backup. This will probably drive up the "paused" counter but should increase my visibility overall.

ClearLog (string logName, string backupPath);

I'm open to other ideas. One other thing I've observed, these logs get really slow to read once they get too large. I'd like to have a "clear" command that runs whenever the UF starts. Otherwise, the script simply can't handle the pile of data that might be sitting there in the log.

The original author of this TA doesn't seem to have it in GitHub/similar so I'm not sure how best to get all of our ideas back into the original work.