Cisco Firepower eStreamer eNcore Add-on - URL field extraction issue?

[tromero3]

Our URLs are not being extracted from our firepower logs. The url field always shows "unknown" even when there is a URL in the logs.
Does anyone else have this issue? When I try to manually extract the URL using the field extractor it never seems to work, since the URL is sometimes in different locations in the logs, and I am not very good at regex so I can't seem to get it to work myself.

If the URL extraction is working for you, can you please share what you have configured for that?
Thank you!

4 sample events below---

rec_type=71 dns_resp_id=0 ips_count=0 ssl_cipher_suite=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 ssl_version=TLSv1.2 ssl_rule_id=45 app_proto=HTTPS src_mask=0 ssl_server_cert_status="Invalid Issuer" sec_intel_event=No ssl_cert_fingerprint=8e9892de2bacb2060b1eb6d2ae732e489f86138a src_pkts=6 rec_type_simple=RNA ssl_expected_action="Do Not Decrypt" has_ipv6=1 num_ioc=0 sec_zone_ingress=inside ssl_flow_flags=75555521 referenced_host="" dest_autonomous_system=0 src_ip_country=unknown event_desc="Flow Statistics" mac_address=00:00:00:00:00:00 ssl_server_name="" ssl_flow_error=0 ssl_flow_messages=16408 fw_rule_reason=N/A iface_ingress=inside connection_id=64687 last_pkt_sec=1588603267 fw_policy=Default-Policy client_version="" url_reputation=Trusted dest_port=443 url_category="Business and Industry" event_type=1003 dns_ttl=0 sec_zone_egress=centurylink-outside ssl_policy_id=3f8afb02550111ecd74b9e4f4488bf9f dest_mask=0 ssl_actual_action="Do Not Decrypt" sensor=fwpr432432 http_response=0 first_pkt_sec=1588603241 dns_rec_id=0 dest_ip=170.146.102.193 ssl_session_id=0493d63c78766ac12c9b68f720ff528817429df265ae12419699be4b700f2219 dest_ip_country="united states" event_usec=0 dest_pkts=8 ssl_flow_status=Success file_count=0 legacy_ip_address=0.0.0.0 ip_proto=TCP user=bsmith ip_layer=0 monitor_rule_3=N/A src_autonomous_system=0 monitor_rule_1=N/A monitor_rule_7=N/A monitor_rule_6=N/A monitor_rule_5=N/A monitor_rule_4=N/A monitor_rule_2=N/A tcp_flags=0 http_referrer="" vlan_id=0 sec_intel_ip=N/A ssl_url_category=0 fw_rule_action=Allow url=https://workforcenow.adp.com netbios_domain="" src_ip=172.1.5.6 netflow_src=00000000-0000-0000-0000-000000000000 instance_id=2 fw_rule=allowed_traffic user_agent="" monitor_rule_8=0 snmp_in=0 dns_query="" iface_egress=outside event_subtype=1 event_sec=1588603268 dest_tos=0 security_context=00000000000000000000000000000000 src_port=57759 src_bytes=668 web_app="ADP Workforce Now" client_app="SSL client" src_tos=0 snmp_out=0 rec_type_desc="Connection Statistics" dest_bytes=627 sinkhole_uuid=00000000-0000-0000-0000-000000000000 ssl_ticket_id=0000000000000000000000000000000000000000

rec_type=71 iface_ingress=INTERNAL event_desc="Flow Statistics" sinkhole_uuid=00000000-0000-0000-0000-000000000000 url_reputation=Unknown src_pkts=5405 sensor=fp003 ssl_ticket_id=0000000000000000000000000000000000000000 event_subtype=1 dest_ip=23.204.249.25 dest_ip_country="united states" dns_resp_id=0 user="No Authentication Required" ssl_flow_status=Unknown fw_policy=00000000-0000-0000-0000-00005eabbd8c fw_rule_action=Allow instance_id=5 user_agent="" client_version="" snmp_out=0 sec_intel_ip=N/A http_referrer="" fw_rule_reason=N/A netbios_domain="" iface_egress=OUTSIDE src_ip_country=unknown url_category=Unknown sec_zone_ingress=INSIDE num_ioc=0 ssl_server_cert_status="Not Checked" ssl_flow_messages=0 ssl_flow_flags=0 last_pkt_sec=1588533371 rec_type_simple=RNA event_type=1003 ssl_rule_id=0 src_bytes=577 dest_bytes=14301799 dest_mask=0 legacy_ip_address=0.0.0.0 referenced_host="" event_usec=0 first_pkt_sec=1588533357 ssl_cert_fingerprint=0000000000000000000000000000000000000000 ssl_policy_id=00000000000000000000000000000000 ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 ips_count=0 snmp_in=0 dns_rec_id=0 ssl_url_category=0 file_count=0 app_proto=HTTPS web_app=Microsoft url=https://definitionupdates.microsoft.com sec_intel_event=No event_sec=1588533357 dest_pkts=10416 rec_type_desc="Connection Statistics" ssl_version=Unknown ssl_actual_action=Unknown src_autonomous_system=0 src_mask=0 tcp_flags=0 dest_tos=0 dest_autonomous_system=0 http_response=0 has_ipv6=1 ip_proto=TCP dns_ttl=0 security_context=00000000000000000000000000000000 netflow_src=00000000-0000-0000-0000-000000000000 src_tos=0 ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL vlan_id=0 dest_port=443 ssl_expected_action=Unknown src_ip=172.1.2.3 monitor_rule_3=N/A src_port=54120 mac_address=00:00:00:00:00:00 sec_zone_egress=OUTSIDE client_app="SSL client" ip_layer=0 fw_rule=allowed_traffic ssl_flow_error=0 connection_id=38534 monitor_rule_8=0 dns_query="" monitor_rule_2=N/A ssl_server_name="" monitor_rule_1=N/A monitor_rule_6=N/A monitor_rule_7=N/A monitor_rule_4=N/A monitor_rule_5=N/A

rec_type=71 sec_zone_ingress=internal_3 ssl_actual_action=Unknown rec_type_simple=RNA src_tos=0 src_pkts=64 security_context=00000000000000000000000000000000 sinkhole_uuid=00000000-0000-0000-0000-000000000000 mac_address=00:00:00:00:00:00 sec_intel_event=No src_ip_country=unknown sensor=fp03 fw_rule_reason=N/A client_version="" fw_policy=00000000-0000-0000-0000-00005eabbd17 sec_intel_ip=N/A src_ip=10.10.5.4 client_app="Web browser" file_count=0 ip_proto=TCP url_category=1048622 dest_mask=0 has_ipv6=1 ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 event_subtype=1 event_sec=1588485658 ssl_version=Unknown monitor_rule_8=0 ssl_policy_id=00000000000000000000000000000000 ssl_flow_flags=0 event_desc="Flow Statistics" ssl_url_category=0 src_mask=0 ssl_cert_fingerprint=0000000000000000000000000000000000000000 ips_count=0 instance_id=4 web_app=Microsoft rec_type_desc="Connection Statistics" fw_rule_action=Allow dns_ttl=0 dns_resp_id=0 dns_query="" iface_ingress=fp03 ssl_ticket_id=0000000000000000000000000000000000000000 user=Unknown tcp_flags=0 dest_bytes=222427 netflow_src=00000000-0000-0000-0000-000000000000 event_type=1003 first_pkt_sec=1588485658 dest_tos=0 src_autonomous_system=0 last_pkt_sec=1588485689 dest_autonomous_system=0 vlan_id=0 ssl_server_cert_status="Not Checked" sec_zone_egress=OUTSIDE iface_egress=OUTSIDE user_agent=Microsoft-Delivery-Optimization/10.0 dest_pkts=159 ssl_flow_status=Unknown url_reputation=Unknown dns_rec_id=0 legacy_ip_address=0.0.0.0 dest_ip_country="united states" snmp_in=0 snmp_out=0 src_port=55403 http_response=0 monitor_rule_6=N/A monitor_rule_7=N/A monitor_rule_4=N/A monitor_rule_5=N/A monitor_rule_2=N/A monitor_rule_3=N/A monitor_rule_1=N/A ssl_rule_id=0 ssl_server_name="" dest_port=80 ip_layer=0 ssl_flow_error=0 netbios_domain="" connection_id=53253 referenced_host=11.tlu.dl.delivery.mp.microsoft.com app_proto=HTTP src_bytes=4565 ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL http_referrer="" num_ioc=0 ssl_expected_action=Unknown dest_ip=72.21.81.240 ssl_flow_messages=0 url="http://11.tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/" event_usec=0 fw_rule=Allowed_traffic

rec_type=71 dns_resp_id=0 ips_count=0 ssl_cipher_suite=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ssl_version=TLSv1.2 ssl_rule_id=999999 app_proto=HTTPS src_mask=0 ssl_server_cert_status=Valid sec_intel_event=No ssl_cert_fingerprint=33b3b7e9da25f5a004e96435d6fb5477dbed27eb src_pkts=13 rec_type_simple=RNA ssl_expected_action="Do Not Decrypt" has_ipv6=1 num_ioc=0 sec_zone_ingress=inside ssl_flow_flags=75557313 referenced_host="" dest_autonomous_system=0 src_ip_country=unknown event_desc="Flow Statistics" mac_address=00:00:00:00:00:00 ssl_server_name="" ssl_flow_error=0 ssl_flow_messages=56 fw_rule_reason=N/A iface_ingress=inside connection_id=14354 last_pkt_sec=1588604153 fw_policy=default_policy client_version="" url_reputation=Trusted dest_port=443 url_category="Computers and Internet" event_type=1003 dns_ttl=0 sec_zone_egress=centurylink-outside ssl_policy_id=3f8afb02550111eab51b943gd488bf2d dest_mask=0 ssl_actual_action="Do Not Decrypt" sensor=fp03 http_response=0 first_pkt_sec=1588604043 dns_rec_id=0 dest_ip=52.114.132.73 ssl_session_id=ed370000c1e5509b2cdbcca12e3e2ebba8fb43fd26e10773c6fds7fdf2342b96 dest_ip_country="united states" event_usec=0 dest_pkts=20 ssl_flow_status=Success file_count=0 legacy_ip_address=0.0.0.0 ip_proto=TCP user=csmith ip_layer=0 monitor_rule_3=N/A src_autonomous_system=0 monitor_rule_1=N/A monitor_rule_7=N/A monitor_rule_6=N/A monitor_rule_5=N/A monitor_rule_4=N/A monitor_rule_2=N/A tcp_flags=0 http_referrer="" vlan_id=0 sec_intel_ip=N/A ssl_url_category=0 fw_rule_action=Allow url=https://self.events.data.microsoft.com netbios_domain="" src_ip=172.1.4.9 netflow_src=00000000-0000-0000-0000-000000000000 instance_id=4 fw_rule=Inside-access-out user_agent="" monitor_rule_8=0 snmp_in=0 dns_query="" iface_egress=outside event_subtype=1 event_sec=1588604154 dest_tos=0 security_context=00000000000000000000000000000000 src_port=63433 src_bytes=2833 web_app=Microsoft client_app="SSL client" src_tos=0 snmp_out=0 rec_type_desc="Connection Statistics" dest_bytes=6466 sinkhole_uuid=00000000-0000-0000-0000-000000000000 ssl_ticket_id=0000000000000000000000000000000000000000

[to4kawa]

|kv
kv command is useful. please try it.

[shivanshu1593]

Can you please share some sample data, and we can help you get it working via regex.

[tromero3]

Okay I uploaded 4 sample events above in my question. Thank you!

[shivanshu1593 · Accepted solution]

Here you go. This regex will extract all the URLs, which will fall under the url field.

url\=(?<url>[^ ]*)

You can go to this link, which I have saved, and try out the regex with multiple sample data to test it out.

https://regex101.com/r/PC5WU0/1/

Although, I'm a bit surprised that despite being the delimiter "=" in your data, Splunk was unable to extract your URLs automatically. You can also do it by navigating to your sourcetype on your search head and implementKV_MODE=auto for it. It should also do the trick.

Else, you can also install url toolbox on your search head, which can take care of the URL extraction permanently.

https://splunkbase.splunk.com/app/2734/

Let me know if the solution works for you.

Cheers,