Firepower log random field order for DNS

[sethion]

Hello,

we have connected FMC with 12 Security Gateways to Splunk using estreamer addon installed on HF. Log ingestion works fine, but we have issues with filtering.

During log analysis, it turned out that the order of the fields in DNS logs is not the same in each message, but they can have ~6 versions which cause great pain for the filtering. (we need to filter out internal DNS requests and leave requests for external resources)

We were able to create 5 filters but unfortunately, since they are rather heavy Splunk throws errors when we implement 6.

Example of such logs

rec_type=71 monitor_rule_7=N/A fw_rule_action=Fastpath src_tos=0 dns_resp_id=0 event
rec_type=71 netbios_domain="" file_count=0 referenced_host="" monitor_rule_7=N/A monitor_rule_6=N/A fw_rule_action=Fastpath
rec_type=71 fw_rule_action=Fastpath dns_rec_id=0 client_app=Unknown event_subtype=1

I would like to ask if there is a way to tackle this problem.

Regards,
Dawid

[PavelP]

Hello @sethion,

the log is in key=value format and is being automatically parsed by splunk without need to specifiy a format of the log. The order of the fields should not be a problem.

All you need is to filter internal domain in the referenced_host field, right?

If not, then provide more info please

[sethion]

Hello @PavelP

We wanted to filter out logs that arrive from internal hosts to internal DNS servers. What we are doing atm is filtering using src_ip dns_record_name dest_ip to get requests done by internal hosts to internal servers. To do that we need a few regexes since there is multiple order, in which those fields appear in logs.

it's true that the logs are parsed by Splunk, but when we want to parse the logs using regex they should be in the same order or we would have to implement multiple conditions that cause an error in Splunk.

I have checked and unfortunately, the referenced_host filed is often empty.

The question is:
a) why the log order is different for some of the messages? I have not seen this issue in the case of other log sources.
b) what would be the correct approach to tackle this?

Thank you for your help.

[PavelP]

Hello @sethion

according https://www.cisco.com/c/en/us/td/docs/security/firepower/Syslogs/b_fptd_syslog_guide.pdf the reference_host field is present for HTTP/HTTPS requests only.

can you post your search please?

[sethion]

Hello @PavelP

1. We created a regex that filters out DNS based on the order below (I'm talking about raw events):

dns_record_name src_ip dest_ip dest_port

rec_type=71\sssl_server_name=\S+\ssec_zone_ingress=\S+\sevent_subtype=1\sdns_record_name=\S+\s.+src_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3})\s[\s\S]?dest_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3})\s[\s\S]?(dest_port=53).+

1. After that the remaining DNS logs started to appear in order below, which we also covered with regex: src_ip dest_port dns_record_name dest_port dest_ip

.src_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3})\s[\s\S]?(dns_record_name=\S+)\s[\s\S]?(dest_port=53)\s[\s\S]?dest_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3}).*

1. Now the DNS rules appear in order below, which requires yet another regex ... dest_ip src_ip dest_port dns_record_name

Each time we filter out traffic, new order logs appear for DNS...

Regards,
Dawid

[PavelP]

Hello @sethion,

Dawid, you are on the wrong track. Let splunk do the parsing for you and use SPL to filter and process.

Just post a couple of raw log lines here and describe what final output you need.

[sethion]

Hello @PavelP,

thank you for your support for my issue.

It seems that I have not mentioned that we need to filter out those logs (drop to null queue) to save the license. We are doing filtering in props / transforms on HF where the streamer is installed.

Below are the logs that appeared after each applied filter.

rec_type=71 src_port=40519 src_bytes=80 dns_rec_id=1 ssl_ticket_id=0000000000000000000000000000000000000000 ips_count=0 fw_policy= event_usec=0 ssl_cert_fingerprint=0000000000000000000000000000000000000000 fw_rule_reason=N/A ip_proto=UDP sec_zone_ingress=  dns_query= dest_ip= src_ip= ssl_server_name="" file_count=0 url_reputation="Risk unknown" event_desc="Flow Statistics" dest_bytes=0 connection_id=20829 ssl_policy_id=00000000000000000000000000000000 iface_egress= netbios_domain="" security_context=00000000000000000000000000000000 iface_ingress= src_tos=0 ssl_url_category=0 ssl_expected_action=Unknown dns_resp_id=0 dest_port=53 first_pkt_sec=1590133282 http_response=0 sensor= user_agent="" dns_ttl=0 event_type=1003 event_subtype=1 netflow_src=00000000-0000-0000-0000-000000000000 rec_type_desc="Connection Statistics" event_sec=1590133283 client_app="DNS client" src_pkts=1 last_pkt_sec=0 http_referrer="" has_ipv6=1 web_app=Unknown sec_zone_egress= url_category=Unknown ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 ip_layer=0 dns_record_name=A ssl_flow_flags=0 rec_type_simple=RNA referenced_host="" dest_pkts=0 src_ip_country=unknown monitor_rule_7=N/A sec_intel_event=No app_proto=DNS ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL client_version="" instance_id=8 dns_record_desc="a host address" url="" fw_rule= dest_ip_country=unknown fw_rule_action=Allow sinkhole_uuid=00000000-0000-0000-0000-000000000000

rec_type=71 monitor_rule_7=N/A fw_rule_action=Allow src_tos=0 dns_resp_id=0 event_sec=1589960731 sec_intel_event=No first_pkt_sec=1589960731 ip_layer=0 sec_zone_ingress= ssl_url_category=0 ssl_policy_id=00000000000000000000000000000000 instance_id=7 fw_policy= connection_id=26787 src_bytes=152 ssl_ticket_id=0000000000000000000000000000000000000000 ssl_expected_action=Unknown iface_ingress= file_count=0 user_agent="" dest_bytes=0 client_version="" client_app="DNS client" dns_rec_id=1 url_category=Unknown ssl_cert_fingerprint=0000000000000000000000000000000000000000 dns_query= src_ip_country="united states" src_ip= app_proto=DNS ssl_server_name="" dns_ttl=0 ssl_flow_flags=0 src_port=50792 url_reputation="Risk unknown" dns_record_desc="a host address" event_usec=0 netflow_src=00000000-0000-0000-0000-000000000000 event_subtype=1 rec_type_simple=RNA http_response=0 sensor= web_app=Unknown ip_proto=UDP fw_rule== event_desc="Flow Statistics" security_context=00000000000000000000000000000000 ips_count=0 dest_ip_country="united states" src_pkts=1 rec_type_desc="Connection Statistics" dns_record_name=A dest_pkts=0 fw_rule_reason=N/A sec_zone_egress= netbios_domain="" last_pkt_sec=0 dest_port=53 dest_ip=209.244.0.3 event_type=1003 has_ipv6=1 url="" http_referrer="" referenced_host="" ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL iface_egress=

rec_type=71 ssl_server_name="" sec_zone_ingress= event_subtype=1 dns_record_name=A src_ip_country= dest_ip_country= dns_record_desc="a host address" src_tos=0 dns_ttl=0 netflow_src=00000000-0000-0000-0000-000000000000 ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 iface_egress= ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL dest_bytes=0 src_bytes=152 dns_resp_id=0 url="" monitor_rule_7=N/A src_port=50467 src_ip= sec_intel_event=No web_app=Unknown netbios_domain="" rec_type_desc="Connection Statistics" dest_ip= ssl_ticket_id=0000000000000000000000000000000000000000 ssl_url_category=0 fw_rule_reason=N/A event_sec=1590047131 url_category=Unknown has_ipv6=1 connection_id=35377 fw_policy= ssl_policy_id=00000000000000000000000000000000 referenced_host="" ssl_expected_action=Unknown sensor= event_type=1003 client_version="" first_pkt_sec=1590047130 user_agent="" last_pkt_sec=0 security_context=00000000000000000000000000000000 ips_count=0 ip_layer=0 dest_port=53 sec_zone_egress=S iface_ingress= fw_rule= file_count=0 dest_pkts=0 event_desc="Flow Statistics" app_proto=DNS event_usec=0 http_referrer="" fw_rule_action=Allow http_response=0 rec_type_simple=RNA src_pkts=1 ssl_cert_fingerprint=0000000000000000000000000000000000000000 instance_id=4 url_reputation="Risk unknown" ip_proto=UDP dns_query= ssl_flow_flags=0 client_app="DNS client" dns_rec_id=1

Thank you.

[PavelP]

Hello @sethion,

thank you for the clarification, now I understand.

It is better to pre-filter on the estreamer side, check TA-eStreamer/bin/encore/estreamer/pipeline.py and bin/encore/estreamer/adapters/splunk.py and maybe others.