Why don't you make the props.conf not for json? The eve json has a lot more data   This ELK dashboard is amazing and you need the additional fields to drive it https://github.com/robcowart/synesis_lite_suricata To be honest this seems to really give Splunk a run for the money https://www.youtube.com/watch?v=YA2tGrBQ4v0 Years ago I would have argued that Splunk has more support and documentation but I'm finding more and more tutorials geared towards ELK. ... View more

Sorry to4kawa, I'm not advanced enough to understand your answer. I changed the props to have your inputs, but not sure on what is and how to apply source="udp:514" | rex "(?<json>{.*})" | spath input=json   Example of the TA-pfsense props.conf: [pfsense] # timestamp fix to prevent default extraction issues when adding more apps TIME_PREFIX = ^ TIME_FORMAT = %b %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 15 # rest is original code TRANSFORMS-pfsense_sourcetyper = pfsense_sourcetyper SHOULD_LINEMERGE = false # suspect the below 3 lines could be causing the time stamps being missing on some log types test 7.18.2020 # SEDCMD-event_cleaner = s/^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+\S+\.\S+\s+/\1/g # SEDCMD-event_cleaner2 = s/^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+/\1/g # SEDCMD-event_cleaner3 = s/^\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s\S+\s(\S+\s)/\1/g [pfsense:filterlog] KV_MODE = none EXTRACT-ipv4_tcp = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*), (?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*) ,(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>tcp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<d est_port>[^,]*),(?<payload_bytes>[^,]*),(?<vendor_tcp_flags>[^,]*),(?<sequence_number>[^,]*),(?<ack>[^,]*),(?<window>[^,]*),(?<urg>[^,]*),( ?<options>[^$]*)$ EXTRACT-ipv4_udp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*), (?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*) ,(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>udp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<d est_port>[^,]*),(?<payload_bytes>[^,]*) EXTRACT-ipv4_icmp_request = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reaso n>[^,]*),(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offs et>[^,]*),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>icmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<icmp_code> request),(?<echo_id>\d+),(?<echo_sequence>\d+) EXTRACT-ipv4_icmp_unreachport = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<r eason>[^,]*),(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?< offset>[^,]*),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>icmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<icmp_c ode>unreachport),(?<icmp_dest_ip>[^,]*),(?<unreachable_protocol_id>[^,]*),(?<unreachable_port_data>[^$]+)$ EXTRACT-ipv4_redirect_icmp_unreach_timeexceed = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest _int>[^,]*),(?<reason>[^,]*),(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*), (?<id>[^,]*),(?<offset>[^,]*),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>icmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip> [^,]*),(?<icmp_code>redirect|unreach|timexceed),(?<icmp_text>[^$]+)$ EXTRACT-ipv4_carp = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*) ,(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]* ),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>carp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<carp_type>[^,]*),( ?<carp_ttl>[^,]*),(?<vhid>[^,]*),(?<version>[^,]*),(?<advbase>[^,]*),(?<advskew>[^,]*) EXTRACT-ipv4_igmp = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*) ,(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]* ),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>igmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),datalength=(?<data_le ngth>\d+) EXTRACT-ipv4_pim = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*), (?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*) ,(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>pim),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),datalength=(?<data_leng th>\d+) EXTRACT-ipv6_icmp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*) ,(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),(?<vendor_tra nsport>ICMPv6),(?<transport_id>[^,]*),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*), EXTRACT-ipv6_tcp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*), (?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),(?<vendor_tran sport>TCP),(?<transport_id>[^,]*),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<dest_port>[^,]*),(?<payload_byte s>[^,]*),(?<vendor_tcp_flags>[^,]*),(?<sequence_number>[^,]*),(?<ack>[^,]*),(?<window>[^,]*),(?<urg>[^,]*),(?<options>[^$]*)$ EXTRACT-ipv6_udp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*), (?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),(?<vendor_tran sport>UDP),(?<transport_id>[^,]*),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<dest_port>[^,]*),(?<payload_byte s>[^,]*) EXTRACT-ipv6-hbh = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*), (?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),Options,\d+,\d +,(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<extension_header>\w+), REPORT-vendor_tcp_flag = pfsense_vendor_tcp_flag EVAL-app = "pfsense:filterlog" EVAL-bytes_in = if(vendor_direction == "in", bytes,null) EVAL-bytes_out = if(vendor_direction == "out", bytes,null) EVAL-direction = case(vendor_direction == "in", "inbound", vendor_direction == "out", "outbound") FIELDALIAS-pfsense_filterlog_src=src_ip AS src FIELDALIAS-pfsense_filterlog_dest = dest_ip AS dest LOOKUP-filterlog_tcpflags = pfsense_filter_tcpflags vendor_tcp_flag OUTPUTNEW tcp_flag LOOKUP-filterlog_transport = pfsense_filter_transport vendor_transport OUTPUTNEW transport LOOKUP-filterlog_action = pfsense_filter_action vendor_action OUTPUTNEW action [pfsense:dhcpd] EXTRACT-ipv4_dhcp = (?<vendor_action>DHCPACK|DHCPREQUEST) (?:on|for) (?<dest_ip>\S+) (?:from|to) (?<src_mac>\S+) \(.*\) via (?<src_interfac e>\S+) EXTRACT-ipv6_dhcp_rebind = dhcpd: Rebind message from (?<src_mac>\S+) port (?<src_port>\d+), transaction ID (?<transaction_id>\S+) EXTRACT-ipv6_dhcp_reply = dhcpd: Sending Reply to (?<src_mac>\S+) port (?<src_port>\S+) EVAL-app = "pfsense:dhcpd" [pfsense:openvpn] EXTRACT-process_id = \w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s+\w+(\[(?<process_id>\d+)\])?: EXTRACT-authentication = openvpn: user '(?<user>[^']+)' (?<vendor_action>could not authenticate\.|authenticated) EXTRACT-user,src_ip,src_port = openvpn\[\d+\]: (?<user>\S+)\/(?<src_ip>\S+):(?<src_port>\d+) EXTRACT-src_ip,src_port = openvpn\[\d+\]:\s+(?<src_ip>\d+\.\d+\.\d+\.\d+):(?<src_port>\d+) EXTRACT-src_ip,src_port,user,reason = openvpn\[\S+\]: (?P<src_ip>[^:]+):(?P<src_port>\d+).*\[(?<user>\S+)\]\s+(?<reason>Peer Connection Ini tiated|Inactivity timeout) EVAL-app = "pfsense:openvpn" LOOKUP-openvpn_action = pfsense_openvpn_action vendor_action OUTPUTNEW action [pfsense:nginx] EXTRACT-pf_nginx = nginx: (?P<dest>[^ ]+) \-(?P<user>[^\s"]*) \- \[(?P<request_time>[^\]]+)[^"\n]*"(?P<http_method>\w+)\s+(?P<url>[^\s"]+)[ ^ \n]* (?P<http_proto>[^"]+)[^ \n]* (?P<status>[^ ]+)\s+(?P<bytes_out>\d+)[^"\n]*"(?P<http_referrer>[^"]+)(?:[^"\n]*"){2}(?P<http_user_agen t>[^"]+) EVAL-app = "pfsense:nginx" [pfsense:unbound] EXTRACT-queries = info: resolving (?P<query>\S{2,})\s(?P<query_type>\S+) EXTRACT-response = info: response for (?P<answer>\S+)\s(?P<record_type>\S+) EXTRACT-reply = info: reply from\s(\S+)\s(?P<src>(\d|.)+)#(?P<src_port>(\d+)) EVAL-app = "pfsense:unbound" # added due to Barnyard2 support dropped by Snort and Suricata 8.7.20 #[pfsense:suricata] #EXTRACT-classification = ^(?:[^:\n]*:){6}\s+(?P<classification>[^\]]+) #EXTRACT-generator_id = ^[^\[\n]*\[(?P<generator_id>\d+) #EXTRACT-signature_id = ^\[\d+:(?P<signature_id>\d+) #EXTRACT-sigrev_id = ^\[\d+:\d+:(?P<sigrev_id>\d+) #EXTRACT-description = ^[^\]\n]*\](?P<description>[^\[]+) #EXTRACT-priority = ^(?:[^:\n]*:){4}\s+(?P<priority>\d+) #EXTRACT-interface = ^[^<\n]*<(?P<interface>\w+) #EXTRACT-protocol = ^[^\{\n]*\{(?P<protocol>\w+) #EXTRACT-source_ip = ^[^\}\n]*\}\s+(?P<source_ip>[^:]+) #EXTRACT-source_port = ^(?:[^:\n]*:){3}(?P<source_port>\d+) #EXTRACT-destination_ip = ^[^\-\n]*\->\s+(?P<destination_ip>[^:]+) #EXTRACT-destination_port = ^[^\-\n]*\->\s+\d+\.\d+\.\d+\.\d+:(?P<destination_port>.+) # converted suricata to eve json for testing [pfsense:suricata] INDEXED_EXTRACTIONS = json KV_MODE = none ... View more

Another complexity, it looks like the suricata plugin for pfsense is sending both unified2 logs (for the alerts) and then eve json for the addition data to the same data input UDP.  I guess some transforms would be need to sourcetype the two differently to apply extractions to? In order to have eve json sent over syslog, send to system logs must be enabled:   So I get duplicates in two different formats:   Due to pfsense, I don't think there is a way around this (maybe in a config somewhere vs the GUI?)   So the challenge, tell Splunk to shed the unified2 format, keep the eve json format, pull fiends from the json format from without a props that has stanzas for the other pfsense sourcetypes such as filterlog, openvpn etc. Also, Splunk is cutting off the source, the line break is too soon I guess- Splunk vs. pfsense GUI.   ... View more

I'm fairly certain with Splunk 7.3.1 following the BindIP instructions brakes the webUI. When I clear out the stanza to bind the IP I can log back into splunk again from within the VM (as Splunk is choosing the virbr0 interface for some reason). ... View more