Getting Data In

Tip: Sample pfSense Logs Parsed Here

New Member

Hi,

I have parsed some pfSense logs. For anyone making an app, please go ahead and use this info.

Cheers and use in good health.

pfsense_dhcp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+(.*|\s)\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+(.*|\[\d+\]))\:\s(?P<dhcp_type>\w+)\s\w+\s(?P<request_address>\d+\.\d+\.\d+\.\d+)\s\w+\s(?P<request_mac>\w+\:\w+\:\w+\:\w+\:\w+\:\w+)\s|\((?P<network>(.*))\)\s\w+\s(?P<interface>\w+)

pfsense_ipv4_icmp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<tos>\d+x\d+)\,\,(?P<ttl>\d+)\,(?P<id>\d+)\,(?P<offset>\d+)\,(?P<flags>\w+)\,(?P<protocol_id>\d+)\,(?P<protocol_text>\w+)\,(?P<length>\d+)\,(?P<source_address>\d+\.\d+\.\d+\.\d+)\,(?P<destination_address>\d+\.\d+\.\d+\.\d+)\,(?P<icmp_type>\w+)\,(?P<icmp_id>\d+)\,(?P<icmp_sequence>\d+)

pfsense_ipv4_tcp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<tos>\d+x\d+)\,\,(?P<ttl>\d+)\,(?P<id>\d+)\,(?P<offset>\d+)\,(?P<flags>\w+)\,(?P<protocol_id>\d+)\,(?P<protocol_text>\w+)\,(?P<length>\d+)\,(?P<source_address>\d+\.\d+\.\d+\.\d+)\,(?P<destination_address>\d+\.\d+\.\d+\.\d+)\,(?P<source_port>\d+)\,(?P<destination_port>\d+)\,(?P<data_length>\d+)\,(?P<tcp_flag>\w+)\,(?P<sequence_number>\d+)\,(?P<ack_number>.*|\d+)\,(?P<tcp_window>\d+)\,\,(?P<tcp_options>.*|\S+)

pfsense_ipv4_udp

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<tos>\d+x\d+)\,\,(?P<ttl>\d+)\,(?P<id>\d+)\,(?P<offset>\d+)\,(?P<flags>\w+)\,(?P<protocol_id>\d+)\,(?P<protocol_text>\w+)\,(?P<length>\d+)\,(?P<source_address>\d+\.\d+\.\d+\.\d+)\,(?P<destination_address>\d+\.\d+\.\d+\.\d+)\,(?P<source_port>\d+)\,(?P<destination_port>\d+)\,(?P<data_length>\d+)

pfsense_ipv6_icmpv6

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+.*|\s\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)\:\s(?P<rule_number>.*|\d+)\,(?P<sub_rule_number>.*|\d+)\,(?P<anchor>.*|\w+)\,(?P<tracker>\d+)\,(?P<real_interface>\w+)\,(?P<reason>\w+)\,(?P<action>\w+)\,(?P<direction>\w+)\,(?P<ip_version>\d)\,(?P<class>\d+x\d+)\,(?P<flow_label>\w+)\,(?P<hop_limit>\d+)\,(?P<protocol_text>\w+)\,(?P<protocol_id>\d+)\,(?P<length>\d+)\,(?P<source_address>(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))\,(?P<destination_address>(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])))\,

pfsense_nginx

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+(.*|\s)\s\d+\s\d+\:\d+\:\d+)\s(?P<fqdn>\w+\.\w+\.\w+)\s(?P<logtype>\w+)\:\s(?P<remote_address>\d+\.\d+\.\d+\.\d+)\s\-\s\-\s\[\S+\s\-\d+\]\s\"(?P<request>.*)\"\s(?P<status>\d+)\s(?P<body_bytes_sent>\d+)\s\"(?P<http_referrer>\S+)\"\s\"(?P<http_user_agent>.*)\"

pfsense_snort

(?P<host>\d+\.\d+\.\d+\.\d+)\s(?P<timestamp>\S+(.*|\s)\s\d+\s\d+\:\d+\:\d+)\s(?P<logtype>\w+)(.*)\:\s\[(?P<GID>\d+)\:(?P<SID>\d+)\:(?P<rev>\d+)\]\s(?P<description>.*)\s\[Classification:\s(?P<classification>.*)\]\s\[Priority:\s(?P<priority>\d+)\]\s\{(?P<protocol>\w+)\}\s(?P<source_address>\d+\.\d+\.\d+\.\d+)\:(?P<source_port>\d+)\s\-\>\s(?P<destination_address>\d+\.\d+\.\d+\.\d+)\:(?P<destination_port>\d+)
0 Karma
1 Solution

Explorer

Resolved in Question.

View solution in original post

Explorer

Resolved in Question.

View solution in original post

Splunk Employee
Splunk Employee

Hi @arizvi801 - Thank you so much for sharing your pfsense parsing. It would be great if you could put your code in an Answer below that can be accepted. That way other users will know this post is resolved and it can be easily found as a reference 🙂 Thanks!

I've slighted edited your post to make it easier to read all the code. You can click the gear icon to the right of the title and click "Edit" where you can copy and paste all the code into an Answer below.

0 Karma