Splunk Enterprise Security

Cisco Firepower eStreamer eNcore Add-on - URL field extraction issue?

tromero3
Path Finder

Our URLs are not being extracted from our firepower logs. The url field always shows "unknown" even when there is a URL in the logs.
Does anyone else have this issue? When I try to manually extract the URL using the field extractor it never seems to work, since the URL is sometimes in different locations in the logs, and I am not very good at regex so I can't seem to get it to work myself.

If the URL extraction is working for you, can you please share what you have configured for that?
Thank you!

4 sample events below---

rec_type=71 dns_resp_id=0 ips_count=0 ssl_cipher_suite=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 ssl_version=TLSv1.2 ssl_rule_id=45 app_proto=HTTPS src_mask=0 ssl_server_cert_status="Invalid Issuer" sec_intel_event=No ssl_cert_fingerprint=8e9892de2bacb2060b1eb6d2ae732e489f86138a src_pkts=6 rec_type_simple=RNA ssl_expected_action="Do Not Decrypt" has_ipv6=1 num_ioc=0 sec_zone_ingress=inside ssl_flow_flags=75555521 referenced_host="" dest_autonomous_system=0 src_ip_country=unknown event_desc="Flow Statistics" mac_address=00:00:00:00:00:00 ssl_server_name="" ssl_flow_error=0 ssl_flow_messages=16408 fw_rule_reason=N/A iface_ingress=inside connection_id=64687 last_pkt_sec=1588603267 fw_policy=Default-Policy client_version="" url_reputation=Trusted dest_port=443 url_category="Business and Industry" event_type=1003 dns_ttl=0 sec_zone_egress=centurylink-outside ssl_policy_id=3f8afb02550111ecd74b9e4f4488bf9f dest_mask=0 ssl_actual_action="Do Not Decrypt" sensor=fwpr432432 http_response=0 first_pkt_sec=1588603241 dns_rec_id=0 dest_ip=170.146.102.193 ssl_session_id=0493d63c78766ac12c9b68f720ff528817429df265ae12419699be4b700f2219 dest_ip_country="united states" event_usec=0 dest_pkts=8 ssl_flow_status=Success file_count=0 legacy_ip_address=0.0.0.0 ip_proto=TCP user=bsmith ip_layer=0 monitor_rule_3=N/A src_autonomous_system=0 monitor_rule_1=N/A monitor_rule_7=N/A monitor_rule_6=N/A monitor_rule_5=N/A monitor_rule_4=N/A monitor_rule_2=N/A tcp_flags=0 http_referrer="" vlan_id=0 sec_intel_ip=N/A ssl_url_category=0 fw_rule_action=Allow url=https://workforcenow.adp.com netbios_domain="" src_ip=172.1.5.6 netflow_src=00000000-0000-0000-0000-000000000000 instance_id=2 fw_rule=allowed_traffic user_agent="" monitor_rule_8=0 snmp_in=0 dns_query="" iface_egress=outside event_subtype=1 event_sec=1588603268 dest_tos=0 security_context=00000000000000000000000000000000 src_port=57759 src_bytes=668 web_app="ADP Workforce Now" client_app="SSL client" src_tos=0 snmp_out=0 rec_type_desc="Connection Statistics" dest_bytes=627 sinkhole_uuid=00000000-0000-0000-0000-000000000000 ssl_ticket_id=0000000000000000000000000000000000000000

rec_type=71 iface_ingress=INTERNAL event_desc="Flow Statistics" sinkhole_uuid=00000000-0000-0000-0000-000000000000 url_reputation=Unknown src_pkts=5405 sensor=fp003 ssl_ticket_id=0000000000000000000000000000000000000000 event_subtype=1 dest_ip=23.204.249.25 dest_ip_country="united states" dns_resp_id=0 user="No Authentication Required" ssl_flow_status=Unknown fw_policy=00000000-0000-0000-0000-00005eabbd8c fw_rule_action=Allow instance_id=5 user_agent="" client_version="" snmp_out=0 sec_intel_ip=N/A http_referrer="" fw_rule_reason=N/A netbios_domain="" iface_egress=OUTSIDE src_ip_country=unknown url_category=Unknown sec_zone_ingress=INSIDE num_ioc=0 ssl_server_cert_status="Not Checked" ssl_flow_messages=0 ssl_flow_flags=0 last_pkt_sec=1588533371 rec_type_simple=RNA event_type=1003 ssl_rule_id=0 src_bytes=577 dest_bytes=14301799 dest_mask=0 legacy_ip_address=0.0.0.0 referenced_host="" event_usec=0 first_pkt_sec=1588533357 ssl_cert_fingerprint=0000000000000000000000000000000000000000 ssl_policy_id=00000000000000000000000000000000 ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 ips_count=0 snmp_in=0 dns_rec_id=0 ssl_url_category=0 file_count=0 app_proto=HTTPS web_app=Microsoft url=https://definitionupdates.microsoft.com sec_intel_event=No event_sec=1588533357 dest_pkts=10416 rec_type_desc="Connection Statistics" ssl_version=Unknown ssl_actual_action=Unknown src_autonomous_system=0 src_mask=0 tcp_flags=0 dest_tos=0 dest_autonomous_system=0 http_response=0 has_ipv6=1 ip_proto=TCP dns_ttl=0 security_context=00000000000000000000000000000000 netflow_src=00000000-0000-0000-0000-000000000000 src_tos=0 ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL vlan_id=0 dest_port=443 ssl_expected_action=Unknown src_ip=172.1.2.3 monitor_rule_3=N/A src_port=54120 mac_address=00:00:00:00:00:00 sec_zone_egress=OUTSIDE client_app="SSL client" ip_layer=0 fw_rule=allowed_traffic ssl_flow_error=0 connection_id=38534 monitor_rule_8=0 dns_query="" monitor_rule_2=N/A ssl_server_name="" monitor_rule_1=N/A monitor_rule_6=N/A monitor_rule_7=N/A monitor_rule_4=N/A monitor_rule_5=N/A

rec_type=71 sec_zone_ingress=internal_3 ssl_actual_action=Unknown rec_type_simple=RNA src_tos=0 src_pkts=64 security_context=00000000000000000000000000000000 sinkhole_uuid=00000000-0000-0000-0000-000000000000 mac_address=00:00:00:00:00:00 sec_intel_event=No src_ip_country=unknown sensor=fp03 fw_rule_reason=N/A client_version="" fw_policy=00000000-0000-0000-0000-00005eabbd17 sec_intel_ip=N/A src_ip=10.10.5.4 client_app="Web browser" file_count=0 ip_proto=TCP url_category=1048622 dest_mask=0 has_ipv6=1 ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 event_subtype=1 event_sec=1588485658 ssl_version=Unknown monitor_rule_8=0 ssl_policy_id=00000000000000000000000000000000 ssl_flow_flags=0 event_desc="Flow Statistics" ssl_url_category=0 src_mask=0 ssl_cert_fingerprint=0000000000000000000000000000000000000000 ips_count=0 instance_id=4 web_app=Microsoft rec_type_desc="Connection Statistics" fw_rule_action=Allow dns_ttl=0 dns_resp_id=0 dns_query="" iface_ingress=fp03 ssl_ticket_id=0000000000000000000000000000000000000000 user=Unknown tcp_flags=0 dest_bytes=222427 netflow_src=00000000-0000-0000-0000-000000000000 event_type=1003 first_pkt_sec=1588485658 dest_tos=0 src_autonomous_system=0 last_pkt_sec=1588485689 dest_autonomous_system=0 vlan_id=0 ssl_server_cert_status="Not Checked" sec_zone_egress=OUTSIDE iface_egress=OUTSIDE user_agent=Microsoft-Delivery-Optimization/10.0 dest_pkts=159 ssl_flow_status=Unknown url_reputation=Unknown dns_rec_id=0 legacy_ip_address=0.0.0.0 dest_ip_country="united states" snmp_in=0 snmp_out=0 src_port=55403 http_response=0 monitor_rule_6=N/A monitor_rule_7=N/A monitor_rule_4=N/A monitor_rule_5=N/A monitor_rule_2=N/A monitor_rule_3=N/A monitor_rule_1=N/A ssl_rule_id=0 ssl_server_name="" dest_port=80 ip_layer=0 ssl_flow_error=0 netbios_domain="" connection_id=53253 referenced_host=11.tlu.dl.delivery.mp.microsoft.com app_proto=HTTP src_bytes=4565 ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL http_referrer="" num_ioc=0 ssl_expected_action=Unknown dest_ip=72.21.81.240 ssl_flow_messages=0 url="http://11.tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/" event_usec=0 fw_rule=Allowed_traffic

rec_type=71 dns_resp_id=0 ips_count=0 ssl_cipher_suite=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ssl_version=TLSv1.2 ssl_rule_id=999999 app_proto=HTTPS src_mask=0 ssl_server_cert_status=Valid sec_intel_event=No ssl_cert_fingerprint=33b3b7e9da25f5a004e96435d6fb5477dbed27eb src_pkts=13 rec_type_simple=RNA ssl_expected_action="Do Not Decrypt" has_ipv6=1 num_ioc=0 sec_zone_ingress=inside ssl_flow_flags=75557313 referenced_host="" dest_autonomous_system=0 src_ip_country=unknown event_desc="Flow Statistics" mac_address=00:00:00:00:00:00 ssl_server_name="" ssl_flow_error=0 ssl_flow_messages=56 fw_rule_reason=N/A iface_ingress=inside connection_id=14354 last_pkt_sec=1588604153 fw_policy=default_policy client_version="" url_reputation=Trusted dest_port=443 url_category="Computers and Internet" event_type=1003 dns_ttl=0 sec_zone_egress=centurylink-outside ssl_policy_id=3f8afb02550111eab51b943gd488bf2d dest_mask=0 ssl_actual_action="Do Not Decrypt" sensor=fp03 http_response=0 first_pkt_sec=1588604043 dns_rec_id=0 dest_ip=52.114.132.73 ssl_session_id=ed370000c1e5509b2cdbcca12e3e2ebba8fb43fd26e10773c6fds7fdf2342b96 dest_ip_country="united states" event_usec=0 dest_pkts=20 ssl_flow_status=Success file_count=0 legacy_ip_address=0.0.0.0 ip_proto=TCP user=csmith ip_layer=0 monitor_rule_3=N/A src_autonomous_system=0 monitor_rule_1=N/A monitor_rule_7=N/A monitor_rule_6=N/A monitor_rule_5=N/A monitor_rule_4=N/A monitor_rule_2=N/A tcp_flags=0 http_referrer="" vlan_id=0 sec_intel_ip=N/A ssl_url_category=0 fw_rule_action=Allow url=https://self.events.data.microsoft.com netbios_domain="" src_ip=172.1.4.9 netflow_src=00000000-0000-0000-0000-000000000000 instance_id=4 fw_rule=Inside-access-out user_agent="" monitor_rule_8=0 snmp_in=0 dns_query="" iface_egress=outside event_subtype=1 event_sec=1588604154 dest_tos=0 security_context=00000000000000000000000000000000 src_port=63433 src_bytes=2833 web_app=Microsoft client_app="SSL client" src_tos=0 snmp_out=0 rec_type_desc="Connection Statistics" dest_bytes=6466 sinkhole_uuid=00000000-0000-0000-0000-000000000000 ssl_ticket_id=0000000000000000000000000000000000000000

Labels (2)
0 Karma
1 Solution

shivanshu1593
Builder

Here you go. This regex will extract all the URLs, which will fall under the url field.

url\=(?<url>[^ ]*)

You can go to this link, which I have saved, and try out the regex with multiple sample data to test it out.

https://regex101.com/r/PC5WU0/1/

Although, I'm a bit surprised that despite being the delimiter "=" in your data, Splunk was unable to extract your URLs automatically. You can also do it by navigating to your sourcetype on your search head and implementKV_MODE=auto for it. It should also do the trick.

Else, you can also install url toolbox on your search head, which can take care of the URL extraction permanently.

https://splunkbase.splunk.com/app/2734/

Let me know if the solution works for you.

Cheers,

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###

View solution in original post

to4kawa
Ultra Champion

|kv
kv command is useful. please try it.

0 Karma

shivanshu1593
Builder

Can you please share some sample data, and we can help you get it working via regex.

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma

tromero3
Path Finder

Okay I uploaded 4 sample events above in my question. Thank you!

0 Karma

shivanshu1593
Builder

Here you go. This regex will extract all the URLs, which will fall under the url field.

url\=(?<url>[^ ]*)

You can go to this link, which I have saved, and try out the regex with multiple sample data to test it out.

https://regex101.com/r/PC5WU0/1/

Although, I'm a bit surprised that despite being the delimiter "=" in your data, Splunk was unable to extract your URLs automatically. You can also do it by navigating to your sourcetype on your search head and implementKV_MODE=auto for it. It should also do the trick.

Else, you can also install url toolbox on your search head, which can take care of the URL extraction permanently.

https://splunkbase.splunk.com/app/2734/

Let me know if the solution works for you.

Cheers,

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...