OK, I got everything working. Here's the final config:
First, on your DNS servers, enable the following buttons
Then, I ended up basically modifying an app initially built by a Splunk SME, @yshabano. The Folder structure is below:
On the HF/UF where you are monitoring the DNS Trace log, use this structure:
apps
WINDNS
local
app.conf
props.conf
inputs.conf
metadata
local.meta
app.conf
[install]
state = enabled
[ui]
is_visible = false
is_manageable = false
inputs.conf
[monitor://\c:\Logs\DNS\dns.log]
disabled = 0
sourcetype = DNS
Note: you can point this to the index of your choice.
props.conf
[DNS]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
EXTRACT-Domain = (?i) .*? \.(?P<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-src = (?i) [Rcv|Snd] (?P<source_address>\d+\.\d+\.\d+\.\d+)
EXTRACT-Threat_ID,Context,Int_packet_ID,proto,mode,Xid,type,Opcode,Flags_Hex,char_code,ResponseCode,question_type = .+?[AM|PM]\s+(?<Threat_ID>\w+)\s+(?<Context>\w+)\s+(?<Int_packet_ID>\w+)\s+(?<proto>\w+)\s+(?<mode>\w+)\s+\d+\.\d+\.\d+\.\d+\s+(?<Xid>\w+)\s(?<type>(?:R)?)\s+(?<Opcode>\w+)\s+\[(?<Flags_Hex>\w+)\s(?<char_codes>.+?)(?<ResponseCode>[A-Z]+)\]\s+(?<question_type>\w+)\s
EXTRACT-Authoritative_Answer,TrunCation,Recursion_Desired,Recursion_Available = (?m) .+?Message:\W.+\W.+\W.+\W.+\W.+AA\s+(?<Authoritative_Answer>\d)\W.+TC\s+(?<TrunCation>\d)\W.+RD\s+(?<Recursion_Desired>\d)\W.+RA\s+(?<Recursion_Available>\d)
SEDCMD-win_dns = s/\(\d+\)/./g
local.meta
[]
access = read : [ * ], write : [ admin ]
export = system
Then, on the SH, the app is slightly different:
Folder structure
apps
WINDNS
local
app.conf
props.conf
inputs.conf
transforms.conf
metadata
local.meta
app.conf and local.meta are the same. I left inputs.conf, but it's empty.
props.conf
[DNS]
EXTRACT-Domain = (?i) .*? \.(?<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-windows_dns_000001 = (?<thread_id>[0-9A-Fa-f]{4}) (?<Context>[^\s]+)\s+(?<internal_packet_id>[0-9A-Fa-f]+) (?<protocol>UDP|TCP) (?<direction_flag>Snd|Rcv) (?<client_ip>[0-9\.]+)\s+(?<xid>[0-9A-Fa-f]+) (?<type>[R\s]{1}) (?<opcode>[A-Z\?]{1}) \[(?<flags>[0-9A-Fa-f]+) (?<flagAuthoritativeAnswer>[A\s]{1})(?<flagTrucatedResponse>[T\s]{1})(?<flagRecursionDesire>[D\s]{1})(?<flagRecursionAvailable>[R\s]{1})\s+(?<response_code>[^\]]+)\]\s+(?<query_type>[^\s]+)\s+(?<query_name>[^/]+)
EXTRACT-windows_dns_000010 = ([a-zA-Z0-9\-\_]+)\([0-9]+\)(?<tld>[a-zA-Z0-9\-\_]+)\(0\)$
#EXTRACT-windows_dns_000020 = \([0-9]+\)(?<domain>[a-zA-Z0-9\-\_]+\([0-9]+\)[a-zA-Z0-9\-\_]+)\(0\)$
EXTRACT-windows_dns_000030 = \s\([0-9]+\)(?<hostname>[a-zA-Z0-9\-\_]+)\(0\)$
#EVAL-domain = replace(domain, "([\(0-9\)]+)", ".")
EVAL-query_domain = ltrim(replace(query_name, "(\([\d]+\))", "."),".")
EVAL-type_msg = case(type="R", "Response", isnull(type), "Query")
EVAL-opcode_msg = case(opcode="Q", "Standard Query", opcode="N", "Notify", opcode="U", "Update", opcode="?", "Unknown")
EVAL-direction = case(direction_flag="Snd", "Send", direction_flag="Rcv", "Received")
EVAL-decID = tonumber(xid, 16)
REPORT-win_dns = dns_string_lengths, dns_strings
REPORT-extractdoms = extractdoms
REPORT-extractips = extractips
transforms.conf
[dns_string_lengths]
REGEX = \((\d+)\)
FORMAT = strings_len::$1
MV_ADD = true
REPEAT_MATCH = true
[dns_strings]
REGEX = \([0-9]+\)([a-zA-Z0-9\-\_]+)\([0-9]+\)
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true
[extractdoms]
REGEX = Name\s+\"\[[\w]+\]\.(?<Questions>[a-zA-Z0-9\[\]\(\)\-\.\_]+)\.\"\n
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true
[extractips]
REGEX = DATA\s+(?<Answers>[0-9\.]+\n)
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true
Restarted every thing and this is what you see in your search results:
CAVEATS:
This currently does not really parse out some things. For example, in the following event:
ANSWER SECTION:
Offset = 0x0026, RR count = 0
Name "[CC].dmeserve.newsinc.com."
TYPE CNAME .
CLASS 1
TTL 3374
DLEN 34
DATA .wildcard.newsinc.com.edgekey.net.
Offset = 0x0054, RR count = 1
Name "[C2].wildcard.newsinc.com.edgekey.net."
TYPE CNAME .
CLASS 1
TTL 16820
DLEN 21
DATA .e54.g.akamaiedge[C04F].net.
Offset = 0x0075, RR count = 2
Name "[C060].e54.g.akamaiedge[CF].net."
TYPE A .
CLASS 1
TTL 7
DLEN 4
DATA 2.56.22.17
where 'DATA' is NOT an IP address, this does not get parsed out, so the only Answer value will be the single IP of 2.56.22.17
Also, where the ANSWER section is empty:
QUESTION SECTION:
Offset = 0x000c, RR count = 0
Name ".elb0328-7098004.us-west-1.elb.amazonaws.com."
QTYPE AAAA .
QCLASS 1
ANSWER SECTION:
empty
AUTHORITY SECTION:
Offset = 0x0041, RR count = 0
Name "[C0].us-west-1.elb.amazonaws.com."
TYPE SOA .
CLASS 1
TTL 60
DLEN 70
DATA PrimaryServer: .ns-1009.awsdns-11.org.
Administrator: .awsdns-hostmaster.amazon[C8].com.
SerialNo = 1
Refresh = 7200
Retry = 900
Expire = 1209600
MinimumTTL = 60
ADDITIONAL SECTION:
Offset = 0x0093, RR count = 0
Name "."
TYPE OPT .
CLASS 4096
TTL 32768
DLEN 0
DATA
Buffer Size = 4096
Rcode Ext = 0
Rcode Full = 0
Version = 0
Flags = 80 DO
You probably will not have anything in the Questions or Answers fields.
Finally, I'm not currently parsing anything out of the AUTHORITY SECTION or the ADDITIONAL SECTION. I may add those later.
I think that's it. Happy Splunking!
OK, I got everything working. Here's the final config:
First, on your DNS servers, enable the following buttons
Then, I ended up basically modifying an app initially built by a Splunk SME, @yshabano. The Folder structure is below:
On the HF/UF where you are monitoring the DNS Trace log, use this structure:
apps
WINDNS
local
app.conf
props.conf
inputs.conf
metadata
local.meta
app.conf
[install]
state = enabled
[ui]
is_visible = false
is_manageable = false
inputs.conf
[monitor://\c:\Logs\DNS\dns.log]
disabled = 0
sourcetype = DNS
Note: you can point this to the index of your choice.
props.conf
[DNS]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
EXTRACT-Domain = (?i) .*? \.(?P<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-src = (?i) [Rcv|Snd] (?P<source_address>\d+\.\d+\.\d+\.\d+)
EXTRACT-Threat_ID,Context,Int_packet_ID,proto,mode,Xid,type,Opcode,Flags_Hex,char_code,ResponseCode,question_type = .+?[AM|PM]\s+(?<Threat_ID>\w+)\s+(?<Context>\w+)\s+(?<Int_packet_ID>\w+)\s+(?<proto>\w+)\s+(?<mode>\w+)\s+\d+\.\d+\.\d+\.\d+\s+(?<Xid>\w+)\s(?<type>(?:R)?)\s+(?<Opcode>\w+)\s+\[(?<Flags_Hex>\w+)\s(?<char_codes>.+?)(?<ResponseCode>[A-Z]+)\]\s+(?<question_type>\w+)\s
EXTRACT-Authoritative_Answer,TrunCation,Recursion_Desired,Recursion_Available = (?m) .+?Message:\W.+\W.+\W.+\W.+\W.+AA\s+(?<Authoritative_Answer>\d)\W.+TC\s+(?<TrunCation>\d)\W.+RD\s+(?<Recursion_Desired>\d)\W.+RA\s+(?<Recursion_Available>\d)
SEDCMD-win_dns = s/\(\d+\)/./g
local.meta
[]
access = read : [ * ], write : [ admin ]
export = system
Then, on the SH, the app is slightly different:
Folder structure
apps
WINDNS
local
app.conf
props.conf
inputs.conf
transforms.conf
metadata
local.meta
app.conf and local.meta are the same. I left inputs.conf, but it's empty.
props.conf
[DNS]
EXTRACT-Domain = (?i) .*? \.(?<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-windows_dns_000001 = (?<thread_id>[0-9A-Fa-f]{4}) (?<Context>[^\s]+)\s+(?<internal_packet_id>[0-9A-Fa-f]+) (?<protocol>UDP|TCP) (?<direction_flag>Snd|Rcv) (?<client_ip>[0-9\.]+)\s+(?<xid>[0-9A-Fa-f]+) (?<type>[R\s]{1}) (?<opcode>[A-Z\?]{1}) \[(?<flags>[0-9A-Fa-f]+) (?<flagAuthoritativeAnswer>[A\s]{1})(?<flagTrucatedResponse>[T\s]{1})(?<flagRecursionDesire>[D\s]{1})(?<flagRecursionAvailable>[R\s]{1})\s+(?<response_code>[^\]]+)\]\s+(?<query_type>[^\s]+)\s+(?<query_name>[^/]+)
EXTRACT-windows_dns_000010 = ([a-zA-Z0-9\-\_]+)\([0-9]+\)(?<tld>[a-zA-Z0-9\-\_]+)\(0\)$
#EXTRACT-windows_dns_000020 = \([0-9]+\)(?<domain>[a-zA-Z0-9\-\_]+\([0-9]+\)[a-zA-Z0-9\-\_]+)\(0\)$
EXTRACT-windows_dns_000030 = \s\([0-9]+\)(?<hostname>[a-zA-Z0-9\-\_]+)\(0\)$
#EVAL-domain = replace(domain, "([\(0-9\)]+)", ".")
EVAL-query_domain = ltrim(replace(query_name, "(\([\d]+\))", "."),".")
EVAL-type_msg = case(type="R", "Response", isnull(type), "Query")
EVAL-opcode_msg = case(opcode="Q", "Standard Query", opcode="N", "Notify", opcode="U", "Update", opcode="?", "Unknown")
EVAL-direction = case(direction_flag="Snd", "Send", direction_flag="Rcv", "Received")
EVAL-decID = tonumber(xid, 16)
REPORT-win_dns = dns_string_lengths, dns_strings
REPORT-extractdoms = extractdoms
REPORT-extractips = extractips
transforms.conf
[dns_string_lengths]
REGEX = \((\d+)\)
FORMAT = strings_len::$1
MV_ADD = true
REPEAT_MATCH = true
[dns_strings]
REGEX = \([0-9]+\)([a-zA-Z0-9\-\_]+)\([0-9]+\)
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true
[extractdoms]
REGEX = Name\s+\"\[[\w]+\]\.(?<Questions>[a-zA-Z0-9\[\]\(\)\-\.\_]+)\.\"\n
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true
[extractips]
REGEX = DATA\s+(?<Answers>[0-9\.]+\n)
FORMAT = strings::$1
MV_ADD = true
REPEAT_MATCH = true
Restarted every thing and this is what you see in your search results:
CAVEATS:
This currently does not really parse out some things. For example, in the following event:
ANSWER SECTION:
Offset = 0x0026, RR count = 0
Name "[CC].dmeserve.newsinc.com."
TYPE CNAME .
CLASS 1
TTL 3374
DLEN 34
DATA .wildcard.newsinc.com.edgekey.net.
Offset = 0x0054, RR count = 1
Name "[C2].wildcard.newsinc.com.edgekey.net."
TYPE CNAME .
CLASS 1
TTL 16820
DLEN 21
DATA .e54.g.akamaiedge[C04F].net.
Offset = 0x0075, RR count = 2
Name "[C060].e54.g.akamaiedge[CF].net."
TYPE A .
CLASS 1
TTL 7
DLEN 4
DATA 2.56.22.17
where 'DATA' is NOT an IP address, this does not get parsed out, so the only Answer value will be the single IP of 2.56.22.17
Also, where the ANSWER section is empty:
QUESTION SECTION:
Offset = 0x000c, RR count = 0
Name ".elb0328-7098004.us-west-1.elb.amazonaws.com."
QTYPE AAAA .
QCLASS 1
ANSWER SECTION:
empty
AUTHORITY SECTION:
Offset = 0x0041, RR count = 0
Name "[C0].us-west-1.elb.amazonaws.com."
TYPE SOA .
CLASS 1
TTL 60
DLEN 70
DATA PrimaryServer: .ns-1009.awsdns-11.org.
Administrator: .awsdns-hostmaster.amazon[C8].com.
SerialNo = 1
Refresh = 7200
Retry = 900
Expire = 1209600
MinimumTTL = 60
ADDITIONAL SECTION:
Offset = 0x0093, RR count = 0
Name "."
TYPE OPT .
CLASS 4096
TTL 32768
DLEN 0
DATA
Buffer Size = 4096
Rcode Ext = 0
Rcode Full = 0
Version = 0
Flags = 80 DO
You probably will not have anything in the Questions or Answers fields.
Finally, I'm not currently parsing anything out of the AUTHORITY SECTION or the ADDITIONAL SECTION. I may add those later.
I think that's it. Happy Splunking!
