A method to extend the Ubiquiti add-on to support ingestion of DNSMASQ logs

[edlarsen]

I have customized my local installation of the TA-ubiqiti add-on to support the ingestion of DNSMASQ log messages (produced via the "log-queries=extra" configuration option) and thought that others might be interested in doing the same. I plan to post the specific configuration steps that I followed to get my Unifi Security Gateway 3P to generate and forward the desired DNSMASQ log messages to the Ubiquiti Networks Community Forums, so I will not cover those steps here.

In preparation for setting this up, I followed the installation and configuration steps for the TA-Ubiquiti add-on as outlined in "Method 1" up until the UDP Input is added to Splunk. For my installation, I have all Unifi devices configured to forward syslog output to a Debian Linux based syslog server with a Universal Forwarder installed. Here is an example of the rsyslog configuration I am using:

In /etc/rsyslog.d/ubqt.conf:

if $fromhost-ip == '1.1.1.1' then /var/log/splunk/ubqt/controller.log
if $fromhost-ip == '1.1.1.2' then /var/log/splunk/ubqt/gateway.log
if $fromhost-ip == '1.1.1.3' then /var/log/splunk/ubqt/switch_1.log
if $fromhost-ip == '1.1.1.4' then /var/log/splunk/ubqt/switch_2.log
if $fromhost-ip == '1.1.1.5' then /var/log/splunk/ubqt/ap_1.log
if $fromhost-ip == '1.1.1.6' then /var/log/splunk/ubqt/ap_2.log
@ ~

To ingest the files created above via rsyslog, I am using the following local .conf file configurations:

In ~/etc/apps/TA-ubiquiti/inputs.conf:

[monitor:///var/log/splunk/ubqt]
host_regex = /var/log/splunk/ubqt/(\w+)
sourcetype = ubqt
index=ubqt
disabled = 0

In ~/etc/apps/TA-ubiquiti/props.conf:

[default]
TRANSFORMS-changeindex = index-mcad, index-mcad, index-dhcp, index-fw, index-threat, index-dns

[ubqt]
TRANSFORMS-dns = sourcetype-dns

[ubqt:dns]
EXTRACT-dnsmaq_dns_request = dnsmasq[(?P\d+)]:\s+(?P\d+)\s+(?P[^/]+)/(?P\d+)\s+(?P\w+)[(?P\w+)]\s+(?P[^ ]+)
EXTRACT-dnsmasq_dns_response = dnsmasq[(?P\d+)]:\s+(?P\d+)\s+(?P[^/]+)/(?P\d+)\s+(?P[a-z/]+)\s+(?P[^ ]+) is (?P.+)
EXTRACT-dnsmasq_dest_ip = dnsmasq[(?P\d+)]:\s+(?P\d+)\s+(?P[^/]+)/(?P\d+)\s+(?P\w+)[^ \n]* (?P[^ ]+) to (?P.+)
MAX_TIMESTAMP_LOOKAHEAD = 15
SHOULD_LINEMERGE = 0
TIME_FORMAT = %b %d %H:%M:%S

In ~/etc/apps/TA-ubiquiti/transforms.conf, I added the following two stanzas:

[index-dns]
REGEX = dnsmasq[
FORMAT = netdns
DEST_KEY = _MetaData:Index

[sourcetype-dns]
REGEX = dnsmasq[
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ubqt:dns

Once all of the above configuration work is in place, the DNSMASQ events will be added to the netdns index that should have been created when the ubqt_indexes.tar archive was installed as mentioned within the "Method 1" installation instructions.

One last note: It may be noticed that I am not attempting to differentiate the received ubiquiti syslog messages within rsyslog (as mentioned within the "Method 2" installation summary. So the approach I outline here is something of a hybrid between the two installation methods.

I hope that somebody finds this of some value.

Cheers.

Ed

[edlarsen]

I noticed after posting this customization to the TA-ubiquiti add-on that my references to the inputs.conf, props.conf, and transforms.conf files were missing the "local" sub-directory. 🙄 I have not uncovered a way to update my original post, so here is what those file references should actually look like:

In ~/etc/apps/TA-ubiquiti/local/inputs.conf:

In ~/etc/apps/TA-ubiquiti/local/props.conf:

In ~/etc/apps/TA-ubiquiti/local/transforms.conf, I added the following two stanzas:

This documentation error should not have caused an issue for anyone that has used Splunk for any length of time. But hopefully did not cause trouble for anyone new to Splunk that attempted to follow my procedure. 😟

Cheers.

Ed

[edlarsen]

Here is a link to the associated write-up on how to configure the Ubiquiti Networks UniFi Security Gateway 3P to forward the DNSMASQ log messages to a remote syslog server.

https://community.ubnt.com/t5/UniFi-Routing-Switching/Configuring-a-UniFi-Security-Gateway-3P-to-inc...

Cheers.

Ed