I am doing some field extractions for Juniper JunOS logs and I created the following field extractions via props/transforms files in the /opt/splunk/etc/apps/Splunk_TA_juniper/local directory:
props.conf
[juniper:junos:firewall]
REPORT-filter_for_junos_fw = filter_for_junos_fw
REPORT-src_interface_for_junos_fw = src_interface_for_junos_fw
REPORT-action_for_junos_fw = action_for_junos_fw
REPORT-transport_for_junos_fw = transport_for_junos_fw
REPORT-src_ip_for_junos_fw = src_ip_for_junos_fw
transforms.conf
[filter_for_junos_fw]
REGEX = \s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW
[src_interface_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s(?<src_interface>[^ ]+)\s
[action_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s(?<action>\w+)\s
[transport_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s+[^ ]\s[^ ]+\s[^ ]+\s*[^ ]+\s[^ ]+\s+(?<transport>tcp|udp|icmp)\s
[src_ip_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s+[^ ]\s[^ ]+\s[^ ]+\s*[^ ]+\s[^ ]+\s+[^ ]+\s+(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s
Every field extraction is working except for src
. I validated on https://regex101.com/, ran the regex via rex on the search line, and even ran the regex through "Extract Fields" and values for src
were returned. I did restart Splunk after modifying the conf files.
Any ideas?
Thx
okay try this regex which covers four extractions : filter
, action
, src_interface
, transport
, src
remove all 5 regexes and put all in one.
https://regex101.com/r/rYVwTS/2
\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s
Or if you want to do it separately then try:
\sPFE_FW_SYSLOG_ETH_IP:\sFW:\s.*(tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s
okay try this regex which covers four extractions : filter
, action
, src_interface
, transport
, src
remove all 5 regexes and put all in one.
https://regex101.com/r/rYVwTS/2
\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s
Or if you want to do it separately then try:
\sPFE_FW_SYSLOG_ETH_IP:\sFW:\s.*(tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s
Thx for the regex - I added the src and dest ports capture groups:
\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s.*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s(?<dst>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s(?<src_port>[^ ]+)\s+(?<dest_port>[^ ]+)\s+
and the issue I have is that last event is not extracting fields
https://regex101.com/r/rYVwTS/3
How does one account for K/V pairs that are sometimes there and sometimes not as the 3rd event does not have a src or dest port listed in the event?
Thx
try the updated answer and it should work:
\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s
Modified the regex to as follows - https://regex101.com/r/rYVwTS/6
- and I think it's extracting fields for all events.
Weirdness with the icmp event as I think the 3
listed twice is the source and destination bytes. Can't find any clarifying documentation on Juniper's site other than this - https://forums.juniper.net/t5/Junos/What-is-the-log-format-for-deny-hosts-ACL/m-p/318906 - which talks about not icmp denied traffic, but I can live with it.
Thx a million for the help!
This regex - https://regex101.com/r/rYVwTS/4
- doesn't pick up the dst_port for the 1st event and picks up the value on the 3rd event and assigns it to dst_port even though it's not.
Thx
Are all these extractions for \sPFE_FW_SYSLOG_ETH_IP:\sFW: for one event?
I mean, are you expecting that any event that matches the \sPFE_FW_SYSLOG_ETH_IP:\sFW: would do the last 4 extractions?
If yes, I would just do one big regex with all the extractions.
Also, make sure you don't have another knowledge object rewriting the src field, like a lookup table
The problem I have is that the Juniper logs have different event types that every time I try and do on regex I miss some relevant events so I figured that breaking the regex down to simple field extractions and then building on that would ensure I extract K/V pairs for all relevant events.
I wish I could write one regex that covers all events, but I'm not a regex guru
I do not have a lookup table rewriting the src field. Would running btool show all knowledge objects (using grep for relevant fields)?
Thx
hi
Can you give us a sample event? and which one you are exactly talking about src_interface_for_junos_fw
or src_ip_for_junos_fw
?
Sample data:
Aug 2 09:16:37 10.10.10.10 Aug 2 09:16:37 externals-cl fpc1 PFE_FW_SYSLOG_ETH_IP: FW: xe-1/0/0.0 D 0800 34:62:5a:74:8f:c3 -> 64:b2:9a:7e:1b:4a tcp 184.154.189.91 10.1.1.1 41860 465 (1 packets)
Aug 2 09:23:13 10.10.10.10 Aug 2 09:23:13 externals-fq fpc0 PFE_FW_SYSLOG_ETH_IP: FW: xe-0/0/0.447 D 03af:0700 a6:e7:f2:2e:13:c7 -> 72:a3:9c:3a:22:00 icmp 185.176.27.46 10.1.1.1 44927 53389 (1 packets)
Aug 2 10:00:42 10.10.10.10 Aug 2 10:00:42 externals-fq fpc2 PFE_FW_SYSLOG_ETH_IP: FW: et-2/1/0.716 D 02cc:0800 52:a2:2f:7a:1d:5a -> 84:c9:2b:9e:24:e6 icmp 10.1.1.2 10.1.1.1 3 3 (1 packets)
src_interface_for_junos_fw
and src_ip_for_junos_fw
and stanza header names I assigned as I looked in the /opt/splunk/etc/apps/Splunk_TA_juniper/default directory to for some guidance