Windows DNS debug logs TA not using CIM fields?

[asieira]

I set up a test environment with a Windows 2012 R2 server with the DNS server role, and was able to successfully install TA-DNSServer-NT6 (as per the instructions at http://docs.splunk.com/Documentation/MSApp/1.1.2/MSInfra/DownloadandconfiguretheSplunkAdd-onsforWind... ).

However, I was very surprised to discover two things:

• The DNS packet data (which contains the actual replies, such as the IP addresses a hostname resolved to in a A query) was not being parsed, and in fact generated "incomplete" alerts with each packet line as in this example:

• The summary lines were being correctly parsed, but the corresponding CIM fields (http://docs.splunk.com/Documentation/CIM/4.2.0/User/NetworkResolutionDNS ) were not extracted. For example, a query for tag=dns came up empty even though there are DNS events and the Splunk App for Windows Infrastructure "DNS: Top Requested Queries" report is being populated. Here is an example:

For one, I would have expected to see tags with network, resolution and dns values as per the CIM documentation.

Can anyone else with a working installation of the DNS debug log collection confirm whether this field is populated for them? Did I make some mistake in the setup or is this a limitation of the app?

[Richfez]

For many of these reasons we didn't use this method at all but instead deployed the Splunk App for Stream and used that to sniff DNS packets from the DNS servers. This is also how we're getting DHCP packets in.

It may be an option for you as well. It is a bit of work getting it set up and running, but far less work than you've already put into fixing the DNS logs from MS. Because we're selectively only pulling in DNS and DHCP, I believe it's the same license hit as using the logs would be.

Note: a prerequisite is having winpcap libraries installed because the stream app uses winpcap on the clients to actually sniff the packets, and IIRC the docs aren't as clear as they could be on that. But, that's a small, fast, easy install.

[asieira]

@rich7177 is there any walkthrough or tutorial posted anywhere on how to collect DNS data using Splunk App for Stream? Also, would you mind sharing what the DNS events look like (extracted fields, tags, etc)?

[Richfez]

Ryan wrote up a quick an dirty walkthrough on using Stream for DNS. You can also find a handful of things in the docs and in the splunk blogs to help. Ask if you get stuck - there are some slightly "sticky" places in the setup. BTW, you will need the winpcap libraries installed (Wireshark installs them when you install it, and they're available separately).

All the following is extracted properly - stream grabs the packets off the wire and converts them to json to send in, so all these are fields properly parsed (Splunk likes JSON... 🙂 ). The hosts and other multi-value fields are, when multi-valued, correctly done as multi-value fields. Umm, sorry, that was redundant, eh?

 { [-]
   bytes: 444
   bytes_in: 170
   bytes_out: 274
   dest_ip: 208.67.220.220
   dest_mac: 54:75:D0:BC:7E:C0
   dest_port: 53
   endtime: 2016-08-05T13:53:50.457144Z
   host_type: [ [+]
   ]
   message_type: QUERY
   name: 0108cd802801.t-1470405234.i9b462034.a3dad853f144547d6c20bd57d087602b-631-jpg.a3dad853f144547d6c20bd57d087602b-631-jpg.fp.bl.barracudabrts.com
   query: 0108cd802801.t-1470405234.i9b462034.a3dad853f144547d6c20bd57d087602b-631-jpg.a3dad853f144547d6c20bd57d087602b-631-jpg.fp.bl.barracudabrts.com
   query_type: TXT
   reply_code: NoError
   response_time: 105989
   src_ip: 10.134.0.232
   src_mac: 00:50:56:88:04:E9
   src_port: 62794
   time_taken: 105989
   timestamp: 2016-08-05T13:53:50.351155Z
   transaction_id: 54234
   transport: udp
   ttl: [ [+]
   ]
} 

It really is a superior way. It may be a bit more work to get working, but then has SO much more capability making it worth it.

[Lowell]

@rich7177, where is the walk through? The link just points back to the same comment.

[mikaelbje]

There's an app on Splunkbase which I believe correctly cleans up the DNS debug logs in case you don't want to go down the Stream route:

https://splunkbase.splunk.com/app/3377/

[asieira]

That's awesome, thank you for providing the additional details.

Will the JSON include all RRs in a response packet?

[Richfez]

Yep, the one I STARTED pasting was a bit messier, being something on Amazon with 8 responses. It was a bit messier to look at though, so I picked a simpler response for illustrative purposes. All came through as a multi-valued field and "just works".

[asieira]

Thank you very much for that information, using Stream will probably be the best way of gathering the DNS data we'll need, rather than relying on Windows DNS debug logs which are incredibly hard to parse.

[mreynov_splunk]

we filed a bug and this will be prioritized into the dev queue. thanks for bringing this to our attention!

[asieira]

Any updates on this?

[kevinmanson]

Any updates yet? it's been well over 2 years?

[mikaelbje]

Updated 2019-11-27 to account for _carddavs_.tcp.(null) possibility.
Tested with Splunk Add-on for Windows 7.0.0 and Splunk version 8.0.

I'm answering this thread because I will eventually need this myself.

The current limitations I've identified that I'm attempting to fix:

• CIM compliance
• Normalization of the query name to get rid of the (4)imap(5)gmail(3)com(0) ugliness
• IPv6 addresses are not populated in src

Add these additions to your Splunk_TA_windows directory. You need to deploy this to your Indexer and Search Head.

• local/tags.conf:

[eventtype=nt6-dns-events]
dns = enabled
network = enabled
resolution = enabled

* local/props.conf:

[MSAD:NT6:DNS]
# Index-time Normalization of the query name
# Example 1: (3)www(6)google(3)com(0) becomes www.google.com
# Example 2: (9)_carddavs(4)_tcp(6)(null)(0) becomes _carddavs._tcp(null)
#
SEDCMD-win_dns = s/(\\(\d+\\))/./g s/(\\.)$// s/\s(\\.)/ /
# I don't know if MS DNS is Query or Response, so I set both
EVAL-message_type = mvappend("QUERY","RESPONSE")
FIELDALIAS-message_type = questiontype AS record_type
FIELDALIAS-vendor_query_type = opcode AS vendor_query_type
LOOKUP-dns_query_type_lookup = msad_dns_vendor_query_type_lookup vendor_query_type OUTPUT query_type

* local/transforms.conf:

# Reference: tools .ietf.org/html/rfc2929#section-2.2 
[msad_dns_vendor_query_type_lookup]
filename = dns_vendor_query_type_lookup.csv
# IP version agnostic src
[KV_for_RecvdIP]
REGEX = (Snd|Rcv)\s+((?:(?:\d{1,3}\\.){3}(?:\d{1,3}))|(?:(?:::)?(?:[\dA-Fa-f]{1,4}:{1,2}){1,7}(?:[\d\%A-Fa-z\\.]+)?(?:::)?)|(?:::[\dA-Fa-f\\.]{1,15})|(?:::))
FORMAT = src::$2
#
[KV_for_Domain]
REGEX = \\.([a-zA-Z0-9\\-\\_]+\\.[a-zA-Z0-9\\-\\_]+)$
FORMAT = src_domain::$1

• lookups/dns_vendor_query_type_lookup.csv:

vendor_query_type,query_type
Q,Query
N,Notify
U,Update
?,unknown

[thambisetty]

you can use
EVAL-query = replace(replace(replace(questionname, "(\d+)","."), "^.", ""),".$","") to parse query

[mikaelbje]

Four years later and I am returning to this issue. Our official answer and the best way to get DNS logs in is to use Splunk Stream, but in case you are prevented from doing that, please see my updated answer above which should properly clean up Windows DNS logs, also accounting for the edge case raised by @asieira

[asieira]

Turns out that fixing the "ugly" hostnames as generated in Windows DNS debug logs is unfortunately not as simple as the regex replace you implemented there.

First of all, sometimes they contain pointers in brackets that we also need to get rid of: http://stackoverflow.com/questions/20381717/windows-dns-server-debug-log-hostname-format

Second, parenthesis are valid characters inside domain/host names. So it might be that a valid occurrence of a number inside parenthesis will be incorrectly replaced by a dot. One example of an actual use of parenthesis (though not with numbers inside) that is rather common in Windows networks is the following: (9)_carddavs(4)_tcp(6)(null)(0) which should be converted to _carddavs._tcp.(null).

So the only viable solution that will always work is to actually write code that treats the numbers in parenthesis as what they really are - the length of each dot-separated string that makes up the domain/host name in question. Maybe an external command implementation in Python is in order here.

[fharding]

Mikealbje, I just implemented this workaround in our Splunk instance. Thank you! This just saved me hours of work!

BTW, I had to fix a typo - the stanza in transforms.conf:
[msad_dns_vendor_query_type_lookup]
should be:
[dns_vendor_query_type_lookup]

Another modification I made to your method was copying the default DNSServer-NT6 TA to a custom TA "DNSServer-NT6_cimfix", implementing your fixes, and deleting the default/inputs.conf. I have the original vanilla DNSServer-NT6 TA pushing down to the AD boxes for log collection, and the DNSServer-NT6_cimfix band-aid version of the TA pushing to the search head and indexer for index-time and search-time field extraction.

The extractions look like they're working. Just waiting on Network Sessions (DNS) data model acceleration to rebuild and the DNS ES dashboard should be open for business.

[fharding]

Darn - I spoke too soon. Looks the searches behind the panels in the ES dashboard want more fields. Going to see how much work will be involved in normalizing this data source further. Best option might just be to let the Splunk devs do the work and wait for an official update to the TA.

[mikaelbje]

Btw there's a new community add-on for Windows DNS logs. It uses a new method to get the logs and claims to be CIM compliant. I haven't been able to test it yet. Look it up on Splunkbase and let me know if it works 😉

[fharding]

Are you talking about this one?

https://splunkbase.splunk.com/app/2937/

If so, I'm curious if it works with Server 2008 flavors. Says it's built for 2012r2 and up.

[mikaelbje]

That's the one. Untested by me, but claims to be CIM compliant.