Getting Data In

Tip: Sample pfSense Logs Parsed Here

arizvi801
Explorer

Hi,

I have parsed some pfSense logs. For anyone making an app, please go ahead and use this info.

Cheers and use in good health.

pfsense_dhcp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+(.*|\s)\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+(.*|\[\d+\]))\:\s(?P<dhcp_type>\w+)\s\w+\s(?P<request_address>\d+\.\d+\.\d+\.\d+)\s\w+\s(?P<request_mac>\w+\:\w+\:\w+\:\w+\:\w+\:\w+)\s|\((?P<network>(.*))\)\s\w+\s(?P<interface>\w+)

pfsense_ipv4_icmp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<tos>\d+x\d+)\,\,(?P<ttl>\d+)\,(?P<id>\d+)\,(?P<offset>\d+)\,(?P<flags>\w+)\,(?P<protocol_id>\d+)\,(?P<protocol_text>\w+)\,(?P<length>\d+)\,(?P<source_address>\d+\.\d+\.\d+\.\d+)\,(?P<destination_address>\d+\.\d+\.\d+\.\d+)\,(?P<icmp_type>\w+)\,(?P<icmp_id>\d+)\,(?P<icmp_sequence>\d+)

pfsense_ipv4_tcp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<tos>\d+x\d+)\,\,(?P<ttl>\d+)\,(?P<id>\d+)\,(?P<offset>\d+)\,(?P<flags>\w+)\,(?P<protocol_id>\d+)\,(?P<protocol_text>\w+)\,(?P<length>\d+)\,(?P<source_address>\d+\.\d+\.\d+\.\d+)\,(?P<destination_address>\d+\.\d+\.\d+\.\d+)\,(?P<source_port>\d+)\,(?P<destination_port>\d+)\,(?P<data_length>\d+)\,(?P<tcp_flag>\w+)\,(?P<sequence_number>\d+)\,(?P<ack_number>.*|\d+)\,(?P<tcp_window>\d+)\,\,(?P<tcp_options>.*|\S+)

pfsense_ipv4_udp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<tos>\d+x\d+)\,\,(?P<ttl>\d+)\,(?P<id>\d+)\,(?P<offset>\d+)\,(?P<flags>\w+)\,(?P<protocol_id>\d+)\,(?P<protocol_text>\w+)\,(?P<length>\d+)\,(?P<source_address>\d+\.\d+\.\d+\.\d+)\,(?P<destination_address>\d+\.\d+\.\d+\.\d+)\,(?P<source_port>\d+)\,(?P<destination_port>\d+)\,(?P<data_length>\d+)

pfsense_ipv6_icmpv6

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<class>\d+x\d+)\,(?P<flow_label>\w+)\,(?P<hop_limit>\d+)\,(?P<protocol_text>\w+)\,(?P<protocol_id>\d+)\,(?P<length>\d+)\,(?P<source_address>(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))\,(?P<destination_address>(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))\,

pfsense_nginx

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+(.*|\s)\s\d+\s\d+\:\d+\:\d+)\s(?P<fqdn>\w+\.\w+\.\w+)\s(?P<logtype>\w+)\:\s(?P<remote_address>\d+\.\d+\.\d+\.\d+)\s\-\s\-\s\[\S+\s\-\d+\]\s\"(?P<request>.*)\"\s(?P<status>\d+)\s(?P<body_bytes_sent>\d+)\s\"(?P<http_referrer>\S+)\"\s\"(?P<http_user_agent>.*)\"

pfsense_snort

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+(.*|\s)\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)(.*)\:\s\[(?P<GID>\d+)\:(?P<SID>\d+)\:(?P<rev>\d+)\]\s(?P<description>.*)\s\[Classification:\s(?P<classification>.*)\]\s\[Priority:\s(?P<priority>\d+)\]\s\{(?P<protocol>\w+)\}\s(?P<source_address>\d+\.\d+\.\d+\.\d+)\:(?P<source_port>\d+)\s\-\>\s(?P<destination_address>\d+\.\d+\.\d+\.\d+)\:(?P<destination_port>\d+)
0 Karma
1 Solution

arizviherjavec
Explorer

Resolved in Question.

View solution in original post

arizviherjavec
Explorer

Resolved in Question.

aaraneta_splunk
Splunk Employee
Splunk Employee

Hi @arizvi801 - Thank you so much for sharing your pfsense parsing. It would be great if you could put your code in an Answer below that can be accepted. That way other users will know this post is resolved and it can be easily found as a reference 🙂 Thanks!

I've slighted edited your post to make it easier to read all the code. You can click the gear icon to the right of the title and click "Edit" where you can copy and paste all the code into an Answer below.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...