All Apps and Add-ons
Highlighted

How to parse the full Windows DNS Trace logs?

Builder

Any idea how to parse the full Windows DNS Trace Log events?

I have regex that will parse the first line no problem, but everything after that is a PIA.

What I need is this:

From the Answer section, tokenized, each name in one field and each data in another field.

From the example below:

The Question field should have three values:

www.google.com
www3.google.com
www9.google.com

The Answer field should have the following values:

108.177.98.104
108.177.98.147
108.177.98.105
108.177.98.106
108.177.98.99
108.177.98.103

Here is the actual event from the trace log:

4/20/2017 12:37:29 PM 0CD8 PACKET  00000000043177B0 UDP Rcv 1.1.1.1    b685 R Q [8081   DR  NOERROR] A      (3)www(6)google(3)com(0)
UDP response info at 00000000043177B0
  Socket = 4676
  Remote addr 1.1.1.1, port 53
  Time Query=1955170, Queued=0, Expire=0
  Buf length = 0x0fa0 (4000)
  Msg length = 0x008b (139)
  Message:
    XID       0xb685
    Flags     0x8180
      QR        1 (RESPONSE)
      OPCODE    0 (QUERY)
      AA        0
      TC        0
      RD        1
      RA        1
      Z         0
      CD        0
      AD        0
      RCODE     0 (NOERROR)
    QCOUNT    1
    ACOUNT    6
    NSCOUNT   0
    ARCOUNT   1
    QUESTION SECTION:
    Offset = 0x000c, RR count = 0
    Name      "(3)www(6)google(3)com(0)"
      QTYPE   A (1)
      QCLASS  1
    ANSWER SECTION:
    Offset = 0x0020, RR count = 0
    Name      "[C00C](3)www(6)google(3)com(0)"
      TYPE   A  (1)
      CLASS  1
      TTL    95
      DLEN   4
      DATA   108.177.98.104
    Offset = 0x0030, RR count = 1
    Name      "[C00C](3)www(6)google(3)com(0)"
      TYPE   A  (1)
      CLASS  1
      TTL    95
      DLEN   4
      DATA   108.177.98.147
    Offset = 0x0040, RR count = 2
    Name      "[C00C](3)www3(6)google(3)com(0)"
      TYPE   A  (1)
      CLASS  1
      TTL    95
      DLEN   4
      DATA   108.177.98.105
    Offset = 0x0050, RR count = 3
    Name      "[C00C](3)www9(6)google(3)com(0)"
      TYPE   A  (1)
      CLASS  1
      TTL    95
      DLEN   4
      DATA   108.177.98.106
    Offset = 0x0060, RR count = 4
    Name      "[C00C](3)www(6)google(3)com(0)"
      TYPE   A  (1)
      CLASS  1
      TTL    95
      DLEN   4
      DATA   108.177.98.99
    Offset = 0x0070, RR count = 5
    Name      "[C00C](3)www(6)google(3)com(0)"
      TYPE   A  (1)
      CLASS  1
      TTL    95
      DLEN   4
      DATA   108.177.98.103
    AUTHORITY SECTION:
      empty
    ADDITIONAL SECTION:
    Offset = 0x0080, RR count = 0
    Name      "(0)"
      TYPE   OPT  (41)
      CLASS  4000
      TTL    32768
      DLEN   0
      DATA  
        Buffer Size  = 4000
        Rcode Ext    = 0
        Rcode Full   = 0
        Version      = 0
        Flags        = 80 DO

Testing regex using a site like http://regexr.com/ gets me the following regex syntax that catches all instances of the fields that I want.

DATA\s+?(.+)\n <--- This captures all groups for the answers field above

Name\s+(?.+)\W.+\W.+\W.+\W.+\W.+DATA <--- This captures all groups for the queries above (note: while the following also works [ Name\s+(?.+)\n ], if I use that, I capture what is under the Question, Authority and Additional sections as well. I would like to only capture what is under the Answer section.

So here is my props.conf

[testWinDNS]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
EXTRACT-Domain = (?i) .*? \.(?P<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-src = (?i) [Rcv|Snd] (?P<source_address>\d+\.\d+\.\d+\.\d+)
EXTRACT-Threat_ID,Context,Int_packet_ID,proto,mode,Xid,type,Opcode,Flags_Hex,char_code,ResponseCode,question_type = .+?[AM|PM]\s+(?<Threat_ID>\w+)\s+(?<Context>\w+)\s+(?<Int_packet_ID>\w+)\s+(?<proto>\w+)\s+(?<mode>\w+)\s+\d+\.\d+\.\d+\.\d+\s+(?<Xid>\w+)\s(?<type>(?:R)?)\s+(?<Opcode>\w+)\s+\[(?<Flags_Hex>\w+)\s(?<char_codes>.+?)(?<ResponseCode>[A-Z]+)\]\s+(?<question_type>\w+)\s
REPORT-fields1 = name
REPORT-fields2 = answer

Here is my transforms.conf

[name]
REGEX = (?m) Name\s+(?<Domains>.+)\W.+\W.+\W.+\W.+\W.+DATA
FORMAT = Domains::$1

[answer]
REGEX = (?m) DATA\s+?(?<Answers>.+)\n
FORMAT = Answers::$1

and now I'm trying to build my fields.conf so I'm testing what my regex should be using the following search. I'm looking at this page:

http://docs.splunk.com/Documentation/Splunk/6.5.3/Knowledge/ConfigureSplunktoparsemulti-valuefields

these forum posts:

https://answers.splunk.com/answers/45546/regex-tokenizer-when-the-multivalue-field-has-values-within...
https://answers.splunk.com/answers/5206/multivalue-fields-and-fields-conf.html

So in my search, I tested the following:

index=test_dns | makemv delim="/n/r" , Answers

which didn't work. So I tried

index=test_dns | makemv delim="/r" , Answers

and

index=test_dns | makemv delim="/n" , Answers

neither worked.

Using the suggested configuration at the bottom of:

https://answers.splunk.com/answers/132364/dns-debug-log-dns-log-format-review.html

got me a little further, but I still can't isolate the domains or the IPs that are part of the ANSWER section so they create a multi-value field.

So I tried to extract from one of the fields created in props.conf. Using the following as reference:

https://answers.splunk.com/answers/83884/multi-line-field-extraction-in-props-conf.html
https://answers.splunk.com/answers/47982/extracting-field-from-a-field-other-than-raw-in-props-conf....

Currently added this to the above props.conf

REPORT-extractdoms = extractdoms

and this to the above transforms.conf

[extractdoms]
SOURCE_KEY = query_domain
REGEX =  Name "(?<NewDomain>[a-zA-Z0-9\[\]\-\.\_]+?)\."
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true

But that still doesn't work.

Then I found the Splunk Add-on for Microsoft Windows DNS (https://splunkbase.splunk.com/app/3208/), but after loading that, it only parses line by line, not the whole event and does not account for the fact the events are multiline.

While I continue to try different things and keep searching, does anyone have any suggestions, corrections, links, or other help?

Thanks

Craig

0 Karma
Highlighted

Re: How to parse the full Windows DNS Trace logs?

Builder

I should add that using sysmon and/or capturing network traffic (i.e. stream app or using bro or some other passive DNS monitor) is not currently an option.

0 Karma
Highlighted

Re: How to parse the full Windows DNS Trace logs?

Builder

Note: A friend pointed out some improvements to one of my REGEX

I changed

 [extractdoms]
 SOURCE_KEY = query_domain
 REGEX =  Name "(?<NewDomain>[a-zA-Z0-9\[\]\-\.\_]+?)\."
 FORMAT = strings::$1
 MV_ADD = true
 REPEAT_MATCH = true

to the following:

 [extractdoms]
 SOURCE_KEY = query_domain
 REGEX =  Name\s+\"(?<NewDomain>[a-zA-Z0-9\[\]\(\)\-\.\_]+\"\n)
 FORMAT = strings::$1
 MV_ADD = true
 REPEAT_MATCH = true

In case the parenthesis were not being parse out. But, that isn't working either.

0 Karma
Highlighted

Re: How to parse the full Windows DNS Trace logs?

SplunkTrust
SplunkTrust

hello,
check out this answer: https://answers.splunk.com/answers/35259/best-method-for-pulling-microsoft-dns-logs-with-splunk.html
specially the last answer by @woodcock

0 Karma
Highlighted

Re: How to parse the full Windows DNS Trace logs?

Esteemed Legend

I will reiterate here: There are many old answers but nowadays almost nobody gets DNS events from a Windows server from the logs; the smart way is to pull them off the wire with stream. Trust me: you will regret trying to do any CIM-normalization and correlations with the app logs but it will all be a BREEZE with stream:

http://www.rfaircloth.com/2015/11/06/get-started-with-splunk-app-stream-6-4-dns/

0 Karma
Highlighted

Re: How to parse the full Windows DNS Trace logs?

Builder

You are absolutely right.... except when your boss explicitly tells you he doesn't want to pull it off the wire and makes you turn off your bro_dns feed.

0 Karma
Highlighted

Re: How to parse the full Windows DNS Trace logs?

Esteemed Legend

Time to get a new boss! 😆

0 Karma
Highlighted

Re: How to parse the full Windows DNS Trace logs?

Builder

OK, I got everything working. Here's the final config:

First, on your DNS servers, enable the following buttons

DNS properties trace log config

Then, I ended up basically modifying an app initially built by a Splunk SME, @yshabano. The Folder structure is below:

On the HF/UF where you are monitoring the DNS Trace log, use this structure:

apps
    WINDNS
               local
                       app.conf
                       props.conf
                       inputs.conf
              metadata
                       local.meta

app.conf

[install]
state = enabled

[ui]
is_visible = false
is_manageable = false

inputs.conf

[monitor://\c:\Logs\DNS\dns.log]
disabled = 0
sourcetype = DNS

Note: you can point this to the index of your choice.

props.conf

[DNS]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
EXTRACT-Domain = (?i) .*? \.(?P<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-src = (?i) [Rcv|Snd] (?P<source_address>\d+\.\d+\.\d+\.\d+)
EXTRACT-Threat_ID,Context,Int_packet_ID,proto,mode,Xid,type,Opcode,Flags_Hex,char_code,ResponseCode,question_type = .+?[AM|PM]\s+(?<Threat_ID>\w+)\s+(?<Context>\w+)\s+(?<Int_packet_ID>\w+)\s+(?<proto>\w+)\s+(?<mode>\w+)\s+\d+\.\d+\.\d+\.\d+\s+(?<Xid>\w+)\s(?<type>(?:R)?)\s+(?<Opcode>\w+)\s+\[(?<Flags_Hex>\w+)\s(?<char_codes>.+?)(?<ResponseCode>[A-Z]+)\]\s+(?<question_type>\w+)\s
EXTRACT-Authoritative_Answer,TrunCation,Recursion_Desired,Recursion_Available = (?m) .+?Message:\W.+\W.+\W.+\W.+\W.+AA\s+(?<Authoritative_Answer>\d)\W.+TC\s+(?<TrunCation>\d)\W.+RD\s+(?<Recursion_Desired>\d)\W.+RA\s+(?<Recursion_Available>\d)
SEDCMD-win_dns = s/\(\d+\)/./g

local.meta

[]
access = read : [ * ], write : [ admin ]
export = system

Then, on the SH, the app is slightly different:

Folder structure

apps
    WINDNS
               local
                       app.conf
                       props.conf
                       inputs.conf
                       transforms.conf
              metadata
                       local.meta

app.conf and local.meta are the same. I left inputs.conf, but it's empty.

props.conf

[DNS]
EXTRACT-Domain = (?i) .*? \.(?<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-windows_dns_000001 = (?<thread_id>[0-9A-Fa-f]{4}) (?<Context>[^\s]+)\s+(?<internal_packet_id>[0-9A-Fa-f]+) (?<protocol>UDP|TCP) (?<direction_flag>Snd|Rcv) (?<client_ip>[0-9\.]+)\s+(?<xid>[0-9A-Fa-f]+) (?<type>[R\s]{1}) (?<opcode>[A-Z\?]{1}) \[(?<flags>[0-9A-Fa-f]+) (?<flagAuthoritativeAnswer>[A\s]{1})(?<flagTrucatedResponse>[T\s]{1})(?<flagRecursionDesire>[D\s]{1})(?<flagRecursionAvailable>[R\s]{1})\s+(?<response_code>[^\]]+)\]\s+(?<query_type>[^\s]+)\s+(?<query_name>[^/]+)

EXTRACT-windows_dns_000010 = ([a-zA-Z0-9\-\_]+)\([0-9]+\)(?<tld>[a-zA-Z0-9\-\_]+)\(0\)$
#EXTRACT-windows_dns_000020 = \([0-9]+\)(?<domain>[a-zA-Z0-9\-\_]+\([0-9]+\)[a-zA-Z0-9\-\_]+)\(0\)$
EXTRACT-windows_dns_000030 = \s\([0-9]+\)(?<hostname>[a-zA-Z0-9\-\_]+)\(0\)$
#EVAL-domain = replace(domain, "([\(0-9\)]+)", ".")
EVAL-query_domain = ltrim(replace(query_name, "(\([\d]+\))", "."),".")
EVAL-type_msg = case(type="R", "Response", isnull(type), "Query")
EVAL-opcode_msg = case(opcode="Q", "Standard Query", opcode="N", "Notify", opcode="U", "Update", opcode="?", "Unknown")
EVAL-direction = case(direction_flag="Snd", "Send", direction_flag="Rcv", "Received")
EVAL-decID = tonumber(xid, 16)

REPORT-win_dns = dns_string_lengths, dns_strings
REPORT-extractdoms = extractdoms
REPORT-extractips = extractips

transforms.conf

[dns_string_lengths]
REGEX = \((\d+)\)
FORMAT = strings_len::$1
MV_ADD = true
REPEAT_MATCH = true

[dns_strings]
REGEX = \([0-9]+\)([a-zA-Z0-9\-\_]+)\([0-9]+\)
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true

[extractdoms]
REGEX = Name\s+\"\[[\w]+\]\.(?<Questions>[a-zA-Z0-9\[\]\(\)\-\.\_]+)\.\"\n
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true

[extractips]
REGEX = DATA\s+(?<Answers>[0-9\.]+\n)
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true

Restarted every thing and this is what you see in your search results:

Splunk search results multi values

CAVEATS:

This currently does not really parse out some things. For example, in the following event:

ANSWER SECTION: 
Offset = 0x0026, RR count = 0 
Name "[CC].dmeserve.newsinc.com." 
TYPE CNAME . 
CLASS 1 
TTL 3374 
DLEN 34 
DATA .wildcard.newsinc.com.edgekey.net. 
Offset = 0x0054, RR count = 1 
Name "[C2].wildcard.newsinc.com.edgekey.net." 
TYPE CNAME . 
CLASS 1 
TTL 16820 
DLEN 21 
DATA .e54.g.akamaiedge[C04F].net. 
Offset = 0x0075, RR count = 2 
Name "[C060].e54.g.akamaiedge[CF].net." 
TYPE A . 
CLASS 1 
TTL 7 
DLEN 4 
DATA 2.56.22.17

where 'DATA' is NOT an IP address, this does not get parsed out, so the only Answer value will be the single IP of 2.56.22.17

Also, where the ANSWER section is empty:

QUESTION SECTION: 
Offset = 0x000c, RR count = 0 
Name ".elb0328-7098004.us-west-1.elb.amazonaws.com." 
QTYPE AAAA . 
QCLASS 1 
ANSWER SECTION: 
empty 
AUTHORITY SECTION: 
Offset = 0x0041, RR count = 0 
Name "[C0].us-west-1.elb.amazonaws.com." 
TYPE SOA . 
CLASS 1 
TTL 60 
DLEN 70 
DATA PrimaryServer: .ns-1009.awsdns-11.org. 
Administrator: .awsdns-hostmaster.amazon[C8].com. 
SerialNo = 1 
Refresh = 7200 
Retry = 900 
Expire = 1209600 
MinimumTTL = 60 
ADDITIONAL SECTION: 
Offset = 0x0093, RR count = 0 
Name "." 
TYPE OPT . 
CLASS 4096 
TTL 32768 
DLEN 0 
DATA 
Buffer Size = 4096 
Rcode Ext = 0 
Rcode Full = 0 
Version = 0 
Flags = 80 DO

You probably will not have anything in the Questions or Answers fields.

Finally, I'm not currently parsing anything out of the AUTHORITY SECTION or the ADDITIONAL SECTION. I may add those later.

I think that's it. Happy Splunking!

View solution in original post