topic Re: Regex help for Bind DNS logs in Splunk Search

Regex help for Bind DNS logs

jwalzerpitt — Tue, 30 Jul 2019 15:45:09 GMT

At some point in the past month, the existing extract in transforms.conf quit working and the DNS logs (ingesting from BlueCat syslog), specifically queries, are no longer being parsed correctly. The extract in transforms.conf I have is:

EXTRACT-process,pid,src,src_port,query = \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\S+\s\S+\s(?P<process>\S+)\[(?<pid>\d+)\]\:\s+\S+\s+(?P<src>\S+)\#(?P<src_port>\S+)(\/\s|\s)\((?P<query>\S+)\)

Nothing has changed in the custom TA and as far as I can see the format of the DNS logs has not changed before/after parsing stopped working

With that, I started to create a regex to replace the extract in transforms.conf. The regex is as follows:

client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\s(?<message_type>query):\s(?<query>\S+)\s(?<dns_request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*)\s\((?<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})\)

I tested it against generic query events with success:

2019-07-30 08:59:20 8.8.8.8 BLUECAT-LOG-MSG named[311]: client @0x7fc2c5f35e30 10.1.1.1#57195 (e1875.dscg.akamaiedge.net): view default: query: e1875.dscg.akamaiedge.net IN AAAA + (8.8.8.8)

While reviewing DNS query logs, I see a different query related event for cached queries:

2019-07-30 09:50:01 8.8.8.8 BLUECAT-LOG-MSG named[1054]: client @0x7f9f9bff1230 10.1.1.1#35281 (acp-ss-ue1.adobe.io): view default: query (cache) 'acp-ss-ue1.adobe.io/A/IN' denied

How can I modify my regex to include cached query events, or am I better off creating a separate query for them?

Thx

Re: Regex help for Bind DNS logs

richgalloway — Tue, 30 Jul 2019 17:25:19 GMT

Try this regex:

client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<dns_request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*)\s\((?<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})\)| \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<dns_request_class_name>[^']+))

Re: Regex help for Bind DNS logs

jwalzerpitt — Tue, 30 Jul 2019 17:39:31 GMT

Rich,

Thx for the reply.

Plugging in the regex you listed:

 (?J)client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<dns_request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*)\s\((?<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})\)| \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<dns_request_class_name>[^']+))

Returns the following error from the point of '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<dns_request_class_name>[^']+))

(? A subpattern name must be unique
) A subpattern name must be unique
(? A subpattern name must be unique
) A subpattern name must be unique
(? A subpattern name must be unique
) A subpattern name must be unique

Thx

Re: Regex help for Bind DNS logs

richgalloway — Tue, 30 Jul 2019 17:50:41 GMT

Sorry about that. I left out the (?J) flag. Corrected.

Re: Regex help for Bind DNS logs

jwalzerpitt — Tue, 30 Jul 2019 19:24:27 GMT

NP at all - added the (?J) as a global modifier at https://regex101.com/ and was able to match on both events - regular query and cache query.

I modified the regex to remove 'host' as I am already pulling that via transforms.conf.

Updated regex is:

client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*) | \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<request_class_name>[^']+))

Via props.conf and transforms.conf I am pulling the host as follows:
[props.conf]

TRANSFORMS-host = bluecat_dns-host
FIELDALIAS-dns = host AS dns

[transforms.conf]

[bluecat_dns-host]
DEST_KEY = MetaData:Host
REGEX = \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s(\S+)\s 
FORMAT = host::$1

With all of that, can I replace the extract in props.conf:

EXTRACT-process,pid,src,src_port,query = \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\S+\s\S+\s(?P<process>\S+)\[(?<pid>\d+)\]\:\s+\S+\s+(?P<src>\S+)\#(?P<src_port>\S+)(\/\s|\s)\((?P<query>\S+)\)

With the new extract:

EXTRACT-src,src_port,message_type,query,request_class_name,record_type,flag,host = (?J) client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*) | \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<request_class_name>[^']+))

Thx

Re: Regex help for Bind DNS logs

richgalloway — Wed, 31 Jul 2019 12:09:52 GMT

That should work.

Re: Regex help for Bind DNS logs

jwalzerpitt — Wed, 31 Jul 2019 12:35:54 GMT

TYVM Rich!