Hi,
I'm working with some DNS query logs (actually timestamped tcpdump output) and trying to match them to firewall logs. In the firewall log, I have the destination IP. With the DNS query logs, I have a hostname (request) matching to one or more response values (not always an IP but usually). I'm trying to join the firewall entry with the corresponding DNS query. But I'm unsure how (if possible) to put them together in a transaction.
source="/var/log/dnsqueries.log"
| rex field=_raw "IP (?<src_ip>\d+\.\d+\.\d+\.\d+).(?<src_port>\d+) +\> +(?<dst_ip>\d+\.\d+\.\d+\.\d+).(?<dst_port>\d+): +(?<reqid>\d+)[^\d] *(\[[^\]]+\] )?((?<q_message>(?<q_rectype>[A-Z]+)\? (?<q_value>[^ ]+))|\d+/\d+/\d+ +(?<r_message>(?<r_rectype>[^ ]+) +(?<r_value>.+)) \(\d+\))?"
| eval req_ip=if(src_port=53, dst_ip, src_ip)
| eval req_port=if(src_port=53, dst_port, src_port)
| eval svr_ip=if(src_port=53, src_ip, dst_ip)
| rex field=r_message max_match=25 "(?<r_t>[^ ]+) +(?<r_v>[^ ,]+)(, )?"
| eval r_tv=mvzip(r_t,r_v,"#")
| append [
search (source="/var/log/firewall.log" SRC=123.123.123.123)
]
| eval extip=if(isnotnull(DST), DST, r_v)
| eval reqid=if(isnull(reqid), SPT, reqid)
| eval req_port=if(isnull(req_port), SPT, req_port)
| transaction reqid req_port maxspan=2s
| transaction extip maxspan=5s
I am getting the first transaction matching without issue (pairing request and response of DNS queries). But I can't figure out how to match the multi-value list I created. I want it to join if the single value of the FW entry matches any of the MV list of the DNS response.
Here's an example of data this is trying to join:
Dec 29 12:56:37 ec2 ec2-dns-requests: 12:56:37.990311 IP 123.123.123.123.53332 > 123.123.123.2.53: 54671+ A? 5-8-5-app.agent.datadoghq.com. (47)
Dec 29 12:56:37 ec2 ec2-dns-requests: 12:56:37.991417 IP 123.123.123.2.53 > 123.123.123.123.53332: 54671 9/0/0 CNAME dualstack.agent-520-209329848.us-east-1.elb.amazonaws.com., A 54.225.245.134, A 54.243.126.149, A 54.243.194.58, A 54.225.209.18, A 54.225.213.216, A 54.225.214.228, A 54.225.216.202, A 54.225.223.2 (243)
Dec 29 12:56:38 ec2 kernel: iptables-connections: IN= OUT=eth0 SRC=123.123.123.123 DST=54.225.245.134 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=37699 DF PROTO=TCP SPT=38976 DPT=443 WINDOW=26883 RES=0x00 SYN URGP=0
Essentially, I'm trying to match the connection to the DNS request so I can readily see what the query was for outbound connections. And I'm trying to do this with the log data I have made available to me. In this example, I know a connection was made to 54.225.245.134, and I'd like to be able to have the information that its related to the request for "5-8-5-app.agent.datadoghq.com" tied to it. That way, when dealing with a bulk of data, I can include the requested hostname next to various connections. It's like a lookup, except it's driven by the capture of DNS queries in near-realtime.
Any insight or suggestions as to how to join/match this data would be greatly appreciated.
Thanks,
-Alex
... View more