TA-pfsense + Suricata + Barnyard2 gone + eve json = oh fun

[token2]

TL:DR- How do I specify in the props.conf that for "pfsense:suricata" to then use Splunk's json extraction?

Situation explained below:

Hello All, so I've done some tweaking to make the TA-pfsense add more sourcetypes effectively.  The app's author has a really neat transforms that looks into the syslog event and assign a sourcetype so that the appropriate props.conf stanza is then used to field extract.  This way you get good fields from an openvpn event, good fields from a firewall (filterlog) event etc.

For Snort and then Suricata being hosted on pfsense, I was using the barnyard2 support and sending those logs over a different port, making the apps and sourcetyping easy.

Now both Snort and Suricata have deprecated Barnyard2 support on pfsense.  Snort still supports Unified2 output, Suricata supporting eve json- over the same UDP data input that the TA-pfsense uses.

Thanks to the TA-pfsense transforms I mentioned earlier, the data coming into that UDP feed gets sourcetyped as "pfsense:suricata" and I have a props.conf stanza for it with some rough regex to get fields like Classification, src_ip etc.

For the question:  I selected the eve json format option for fun- you get lots more data like protocols used, data flows etc.  Really neat stuff.  ELK already has an amazing dashboard for this stuff.  How do I specify in the props.conf that for "pfsense:suricata" to then use Splunk's json extraction?

[token2]

Another complexity, it looks like the suricata plugin for pfsense is sending both unified2 logs (for the alerts) and then eve json for the addition data to the same data input UDP.  I guess some transforms would be need to sourcetype the two differently to apply extractions to?

In order to have eve json sent over syslog, send to system logs must be enabled:

So I get duplicates in two different formats:

Due to pfsense, I don't think there is a way around this (maybe in a config somewhere vs the GUI?)

So the challenge, tell Splunk to shed the unified2 format, keep the eve json format, pull fiends from the json format from without a props that has stanzas for the other pfsense sourcetypes such as filterlog, openvpn etc.

Also, Splunk is cutting off the source, the line break is too soon I guess- Splunk vs. pfsense GUI.

[to4kawa]

source="udp:514"
| rex "(?<json>{.*})"
| spath input=json

json can be extracted like above.
Why don't you make the props.conf not for json?

https://github.com/Graylog2/graylog-guide-snort
this is good regex sample.
 

[token2]

Why don't you make the props.conf not for json?

The eve json has a lot more data

This ELK dashboard is amazing and you need the additional fields to drive it

https://github.com/robcowart/synesis_lite_suricata

To be honest this seems to really give Splunk a run for the money

https://www.youtube.com/watch?v=YA2tGrBQ4v0

Years ago I would have argued that Splunk has more support and documentation but I'm finding more and more tutorials geared towards ELK.

[to4kawa]

# converted suricata to eve json for testing
[pfsense:suricata]
SEDCMD-json = s/.*?({.*})/\1/
KV_MODE = json

your json log in not valid json format.so it can't use INDEXED_EXTRACTIONS

https://community.splunk.com/t5/Getting-Data-In/Make-extractions-in-props-conf-from-search-query/m-p...
please check my old Q&A

[token2]

Sorry to4kawa, I'm not advanced enough to understand your answer.

I changed the props to have your inputs, but not sure on what is and how to apply

source="udp:514"
| rex "(?<json>{.*})"
| spath input=json

Example of the TA-pfsense props.conf:

[pfsense]
# timestamp fix to prevent default extraction issues when adding more apps
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15

# rest is original code
TRANSFORMS-pfsense_sourcetyper = pfsense_sourcetyper
SHOULD_LINEMERGE = false
# suspect the below 3 lines could be causing the time stamps being missing on some log types test 7.18.2020
# SEDCMD-event_cleaner = s/^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+\S+\.\S+\s+/\1/g
# SEDCMD-event_cleaner2 = s/^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+/\1/g
# SEDCMD-event_cleaner3 = s/^\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s\S+\s(\S+\s)/\1/g

[pfsense:filterlog]
KV_MODE = none
EXTRACT-ipv4_tcp = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*),
(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*)
,(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>tcp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<d
est_port>[^,]*),(?<payload_bytes>[^,]*),(?<vendor_tcp_flags>[^,]*),(?<sequence_number>[^,]*),(?<ack>[^,]*),(?<window>[^,]*),(?<urg>[^,]*),(
?<options>[^$]*)$
EXTRACT-ipv4_udp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*),
(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*)
,(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>udp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<d
est_port>[^,]*),(?<payload_bytes>[^,]*)
EXTRACT-ipv4_icmp_request = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reaso
n>[^,]*),(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offs
et>[^,]*),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>icmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<icmp_code>
request),(?<echo_id>\d+),(?<echo_sequence>\d+)
EXTRACT-ipv4_icmp_unreachport = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<r
eason>[^,]*),(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<
offset>[^,]*),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>icmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<icmp_c
ode>unreachport),(?<icmp_dest_ip>[^,]*),(?<unreachable_protocol_id>[^,]*),(?<unreachable_port_data>[^$]+)$
EXTRACT-ipv4_redirect_icmp_unreach_timeexceed = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest
_int>[^,]*),(?<reason>[^,]*),(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),
(?<id>[^,]*),(?<offset>[^,]*),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>icmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>
[^,]*),(?<icmp_code>redirect|unreach|timexceed),(?<icmp_text>[^$]+)$
EXTRACT-ipv4_carp = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*)
,(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*
),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>carp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<carp_type>[^,]*),(
?<carp_ttl>[^,]*),(?<vhid>[^,]*),(?<version>[^,]*),(?<advbase>[^,]*),(?<advskew>[^,]*)
EXTRACT-ipv4_igmp = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*)
,(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*
),(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>igmp),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),datalength=(?<data_le
ngth>\d+)
EXTRACT-ipv4_pim = filterlog:\s(?<rule>[^,])*,(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*),
(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>4),(?<tos>[^,]*),(?<ecn>[^,]*),(?<ttl>[^,]*),(?<id>[^,]*),(?<offset>[^,]*)
,(?<flags>[^,]*),(?<transport_id>[^,]*),(?<vendor_transport>pim),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),datalength=(?<data_leng
th>\d+)
EXTRACT-ipv6_icmp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*)
,(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),(?<vendor_tra
nsport>ICMPv6),(?<transport_id>[^,]*),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),
EXTRACT-ipv6_tcp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*),
(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),(?<vendor_tran
sport>TCP),(?<transport_id>[^,]*),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<dest_port>[^,]*),(?<payload_byte
s>[^,]*),(?<vendor_tcp_flags>[^,]*),(?<sequence_number>[^,]*),(?<ack>[^,]*),(?<window>[^,]*),(?<urg>[^,]*),(?<options>[^$]*)$
EXTRACT-ipv6_udp = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*),
(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),(?<vendor_tran
sport>UDP),(?<transport_id>[^,]*),(?<bytes>[^,]*),(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<src_port>[^,]*),(?<dest_port>[^,]*),(?<payload_byte
s>[^,]*)
EXTRACT-ipv6-hbh = filterlog:\s(?<rule>[^,]*),(?<sub_rule>[^,]*),(?<anchor>[^,]*),(?<tracker_id>[^,]*),(?<dest_int>[^,]*),(?<reason>[^,]*),
(?<vendor_action>[^,]*),(?<vendor_direction>[^,]*),(?<ip_version>6),(?<class>[^,]*),(?<flow_label>[^,]*),(?<hop_limit>[^,]*),Options,\d+,\d
+,(?<src_ip>[^,]*),(?<dest_ip>[^,]*),(?<extension_header>\w+),
REPORT-vendor_tcp_flag = pfsense_vendor_tcp_flag
EVAL-app = "pfsense:filterlog"
EVAL-bytes_in = if(vendor_direction == "in", bytes,null)
EVAL-bytes_out = if(vendor_direction == "out", bytes,null)
EVAL-direction = case(vendor_direction == "in", "inbound", vendor_direction == "out", "outbound")
FIELDALIAS-pfsense_filterlog_src=src_ip AS src
FIELDALIAS-pfsense_filterlog_dest = dest_ip AS dest
LOOKUP-filterlog_tcpflags = pfsense_filter_tcpflags vendor_tcp_flag OUTPUTNEW tcp_flag
LOOKUP-filterlog_transport = pfsense_filter_transport vendor_transport OUTPUTNEW transport
LOOKUP-filterlog_action = pfsense_filter_action vendor_action OUTPUTNEW action

[pfsense:dhcpd]
EXTRACT-ipv4_dhcp = (?<vendor_action>DHCPACK|DHCPREQUEST) (?:on|for) (?<dest_ip>\S+) (?:from|to) (?<src_mac>\S+) \(.*\) via (?<src_interfac
e>\S+)
EXTRACT-ipv6_dhcp_rebind = dhcpd: Rebind message from (?<src_mac>\S+) port (?<src_port>\d+), transaction ID (?<transaction_id>\S+)
EXTRACT-ipv6_dhcp_reply = dhcpd: Sending Reply to (?<src_mac>\S+) port (?<src_port>\S+)
EVAL-app = "pfsense:dhcpd"

[pfsense:openvpn]
EXTRACT-process_id = \w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s+\w+(\[(?<process_id>\d+)\])?:
EXTRACT-authentication = openvpn: user '(?<user>[^']+)' (?<vendor_action>could not authenticate\.|authenticated)
EXTRACT-user,src_ip,src_port = openvpn\[\d+\]: (?<user>\S+)\/(?<src_ip>\S+):(?<src_port>\d+)
EXTRACT-src_ip,src_port = openvpn\[\d+\]:\s+(?<src_ip>\d+\.\d+\.\d+\.\d+):(?<src_port>\d+)
EXTRACT-src_ip,src_port,user,reason = openvpn\[\S+\]: (?P<src_ip>[^:]+):(?P<src_port>\d+).*\[(?<user>\S+)\]\s+(?<reason>Peer Connection Ini
tiated|Inactivity timeout)
EVAL-app = "pfsense:openvpn"
LOOKUP-openvpn_action = pfsense_openvpn_action vendor_action OUTPUTNEW action

[pfsense:nginx]
EXTRACT-pf_nginx = nginx: (?P<dest>[^ ]+) \-(?P<user>[^\s"]*) \- \[(?P<request_time>[^\]]+)[^"\n]*"(?P<http_method>\w+)\s+(?P<url>[^\s"]+)[
^ \n]* (?P<http_proto>[^"]+)[^ \n]* (?P<status>[^ ]+)\s+(?P<bytes_out>\d+)[^"\n]*"(?P<http_referrer>[^"]+)(?:[^"\n]*"){2}(?P<http_user_agen
t>[^"]+)
EVAL-app = "pfsense:nginx"

[pfsense:unbound]
EXTRACT-queries = info: resolving (?P<query>\S{2,})\s(?P<query_type>\S+)
EXTRACT-response = info: response for (?P<answer>\S+)\s(?P<record_type>\S+)
EXTRACT-reply = info: reply from\s(\S+)\s(?P<src>(\d|.)+)#(?P<src_port>(\d+))
EVAL-app = "pfsense:unbound"

# added due to Barnyard2 support dropped by Snort and Suricata 8.7.20
#[pfsense:suricata]
#EXTRACT-classification = ^(?:[^:\n]*:){6}\s+(?P<classification>[^\]]+)
#EXTRACT-generator_id = ^[^\[\n]*\[(?P<generator_id>\d+)
#EXTRACT-signature_id = ^\[\d+:(?P<signature_id>\d+)
#EXTRACT-sigrev_id = ^\[\d+:\d+:(?P<sigrev_id>\d+)
#EXTRACT-description = ^[^\]\n]*\](?P<description>[^\[]+)
#EXTRACT-priority = ^(?:[^:\n]*:){4}\s+(?P<priority>\d+)
#EXTRACT-interface = ^[^<\n]*<(?P<interface>\w+)
#EXTRACT-protocol = ^[^\{\n]*\{(?P<protocol>\w+)
#EXTRACT-source_ip = ^[^\}\n]*\}\s+(?P<source_ip>[^:]+)
#EXTRACT-source_port = ^(?:[^:\n]*:){3}(?P<source_port>\d+)
#EXTRACT-destination_ip = ^[^\-\n]*\->\s+(?P<destination_ip>[^:]+)
#EXTRACT-destination_port = ^[^\-\n]*\->\s+\d+\.\d+\.\d+\.\d+:(?P<destination_port>.+)

# converted suricata to eve json for testing
[pfsense:suricata]
INDEXED_EXTRACTIONS = json
KV_MODE = none

[to4kawa]

https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/Wheretofindtheconfigurationfiles

props.conf

[pfsense:suricata]
INDEXED_EXTRACTIONS=json
KV_MODE=none

Here's the setup, but there's going to be a conflict.