Regex help for Bind DNS logs

[jwalzerpitt]

At some point in the past month, the existing extract in transforms.conf quit working and the DNS logs (ingesting from BlueCat syslog), specifically queries, are no longer being parsed correctly. The extract in transforms.conf I have is:

EXTRACT-process,pid,src,src_port,query = \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\S+\s\S+\s(?P<process>\S+)\[(?<pid>\d+)\]\:\s+\S+\s+(?P<src>\S+)\#(?P<src_port>\S+)(\/\s|\s)\((?P<query>\S+)\)

Nothing has changed in the custom TA and as far as I can see the format of the DNS logs has not changed before/after parsing stopped working

With that, I started to create a regex to replace the extract in transforms.conf. The regex is as follows:

client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\s(?<message_type>query):\s(?<query>\S+)\s(?<dns_request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*)\s\((?<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})\)

I tested it against generic query events with success:

2019-07-30 08:59:20 8.8.8.8 BLUECAT-LOG-MSG named[311]: client @0x7fc2c5f35e30 10.1.1.1#57195 (e1875.dscg.akamaiedge.net): view default: query: e1875.dscg.akamaiedge.net IN AAAA + (8.8.8.8)

While reviewing DNS query logs, I see a different query related event for cached queries:

2019-07-30 09:50:01 8.8.8.8 BLUECAT-LOG-MSG named[1054]: client @0x7f9f9bff1230 10.1.1.1#35281 (acp-ss-ue1.adobe.io): view default: query (cache) 'acp-ss-ue1.adobe.io/A/IN' denied

How can I modify my regex to include cached query events, or am I better off creating a separate query for them?

Thx

[richgalloway · Accepted solution]

Try this regex:

client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<dns_request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*)\s\((?<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})\)| \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<dns_request_class_name>[^']+))

[jwalzerpitt]

Rich,

Thx for the reply.

Plugging in the regex you listed:

 (?J)client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<dns_request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*)\s\((?<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})\)| \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<dns_request_class_name>[^']+))

Returns the following error from the point of '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<dns_request_class_name>[^']+))

(? A subpattern name must be unique
) A subpattern name must be unique
(? A subpattern name must be unique
) A subpattern name must be unique
(? A subpattern name must be unique
) A subpattern name must be unique

Thx

[richgalloway]

Sorry about that. I left out the (?J) flag. Corrected.

[jwalzerpitt]

NP at all - added the (?J) as a global modifier at https://regex101.com/ and was able to match on both events - regular query and cache query.

I modified the regex to remove 'host' as I am already pulling that via transforms.conf.

Updated regex is:

client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*) | \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<request_class_name>[^']+))

Via props.conf and transforms.conf I am pulling the host as follows:
[props.conf]

TRANSFORMS-host = bluecat_dns-host
FIELDALIAS-dns = host AS dns

[transforms.conf]

[bluecat_dns-host]
DEST_KEY = MetaData:Host
REGEX = \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s(\S+)\s 
FORMAT = host::$1

With all of that, can I replace the extract in props.conf:

EXTRACT-process,pid,src,src_port,query = \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\S+\s\S+\s(?P<process>\S+)\[(?<pid>\d+)\]\:\s+\S+\s+(?P<src>\S+)\#(?P<src_port>\S+)(\/\s|\s)\((?P<query>\S+)\)

With the new extract:

EXTRACT-src,src_port,message_type,query,request_class_name,record_type,flag,host = (?J) client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\sdefault:\s(?<message_type>query)(: (?<query>\S+) (?<request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*) | \(cache\) '(?<query>[^\/]+)\/(?<record_type>[^\/]+)\/(?<request_class_name>[^']+))

Thx

[richgalloway]

That should work.

[jwalzerpitt]

TYVM Rich!