Field extraction via props/transforms not working

[jwalzerpitt]

I am doing some field extractions for Juniper JunOS logs and I created the following field extractions via props/transforms files in the /opt/splunk/etc/apps/Splunk_TA_juniper/local directory:

props.conf
[juniper:junos:firewall]
REPORT-filter_for_junos_fw = filter_for_junos_fw
REPORT-src_interface_for_junos_fw = src_interface_for_junos_fw
REPORT-action_for_junos_fw = action_for_junos_fw
REPORT-transport_for_junos_fw = transport_for_junos_fw
REPORT-src_ip_for_junos_fw = src_ip_for_junos_fw

transforms.conf
[filter_for_junos_fw]
REGEX = \s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW

[src_interface_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s(?<src_interface>[^ ]+)\s

[action_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s(?<action>\w+)\s

[transport_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s+[^ ]\s[^ ]+\s[^ ]+\s*[^ ]+\s[^ ]+\s+(?<transport>tcp|udp|icmp)\s

[src_ip_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s+[^ ]\s[^ ]+\s[^ ]+\s*[^ ]+\s[^ ]+\s+[^ ]+\s+(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

Every field extraction is working except for src. I validated on https://regex101.com/, ran the regex via rex on the search line, and even ran the regex through "Extract Fields" and values for src were returned. I did restart Splunk after modifying the conf files.

Any ideas?
Thx

[mayurr98 · Accepted solution]

okay try this regex which covers four extractions : filter , action, src_interface , transport , src

remove all 5 regexes and put all in one.

https://regex101.com/r/rYVwTS/2

\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

Or if you want to do it separately then try:

\sPFE_FW_SYSLOG_ETH_IP:\sFW:\s.*(tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

[jwalzerpitt]

Thx for the regex - I added the src and dest ports capture groups:

\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s.*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s(?<dst>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s(?<src_port>[^ ]+)\s+(?<dest_port>[^ ]+)\s+

and the issue I have is that last event is not extracting fields

https://regex101.com/r/rYVwTS/3

How does one account for K/V pairs that are sometimes there and sometimes not as the 3rd event does not have a src or dest port listed in the event?

Thx

[mayurr98]

try the updated answer and it should work:

\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

[jwalzerpitt]

Modified the regex to as follows - https://regex101.com/r/rYVwTS/6 - and I think it's extracting fields for all events.

Weirdness with the icmp event as I think the 3 listed twice is the source and destination bytes. Can't find any clarifying documentation on Juniper's site other than this - https://forums.juniper.net/t5/Junos/What-is-the-log-format-for-deny-hosts-ACL/m-p/318906 - which talks about not icmp denied traffic, but I can live with it.

Thx a million for the help!

[jwalzerpitt]

This regex - https://regex101.com/r/rYVwTS/4 - doesn't pick up the dst_port for the 1st event and picks up the value on the 3rd event and assigns it to dst_port even though it's not.

Thx

[guarisma]

Are all these extractions for \sPFE_FW_SYSLOG_ETH_IP:\sFW: for one event?
I mean, are you expecting that any event that matches the \sPFE_FW_SYSLOG_ETH_IP:\sFW: would do the last 4 extractions?

If yes, I would just do one big regex with all the extractions.

Also, make sure you don't have another knowledge object rewriting the src field, like a lookup table

[jwalzerpitt]

The problem I have is that the Juniper logs have different event types that every time I try and do on regex I miss some relevant events so I figured that breaking the regex down to simple field extractions and then building on that would ensure I extract K/V pairs for all relevant events.

I wish I could write one regex that covers all events, but I'm not a regex guru

I do not have a lookup table rewriting the src field. Would running btool show all knowledge objects (using grep for relevant fields)?

Thx

[mayurr98]

hi

Can you give us a sample event? and which one you are exactly talking about src_interface_for_junos_fw or src_ip_for_junos_fw?

[jwalzerpitt]

Sample data:

Aug  2 09:16:37 10.10.10.10 Aug  2 09:16:37 externals-cl fpc1 PFE_FW_SYSLOG_ETH_IP: FW: xe-1/0/0.0   D 0800 34:62:5a:74:8f:c3 -> 64:b2:9a:7e:1b:4a  tcp 184.154.189.91 10.1.1.1 41860   465 (1 packets)

Aug  2 09:23:13 10.10.10.10 Aug  2 09:23:13 externals-fq fpc0 PFE_FW_SYSLOG_ETH_IP: FW: xe-0/0/0.447 D 03af:0700 a6:e7:f2:2e:13:c7 -> 72:a3:9c:3a:22:00  icmp 185.176.27.46 10.1.1.1 44927 53389 (1 packets)

Aug  2 10:00:42 10.10.10.10 Aug  2 10:00:42 externals-fq fpc2 PFE_FW_SYSLOG_ETH_IP: FW: et-2/1/0.716 D 02cc:0800 52:a2:2f:7a:1d:5a -> 84:c9:2b:9e:24:e6 icmp 10.1.1.2 10.1.1.1     3     3 (1 packets)

src_interface_for_junos_fw and src_ip_for_junos_fw and stanza header names I assigned as I looked in the /opt/splunk/etc/apps/Splunk_TA_juniper/default directory to for some guidance