Splunk Search

Field extraction via props/transforms not working

jwalzerpitt
Influencer

I am doing some field extractions for Juniper JunOS logs and I created the following field extractions via props/transforms files in the /opt/splunk/etc/apps/Splunk_TA_juniper/local directory:

props.conf
[juniper:junos:firewall]
REPORT-filter_for_junos_fw = filter_for_junos_fw
REPORT-src_interface_for_junos_fw = src_interface_for_junos_fw
REPORT-action_for_junos_fw = action_for_junos_fw
REPORT-transport_for_junos_fw = transport_for_junos_fw
REPORT-src_ip_for_junos_fw = src_ip_for_junos_fw

transforms.conf
[filter_for_junos_fw]
REGEX = \s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW

[src_interface_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s(?<src_interface>[^ ]+)\s

[action_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s(?<action>\w+)\s

[transport_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s+[^ ]\s[^ ]+\s[^ ]+\s*[^ ]+\s[^ ]+\s+(?<transport>tcp|udp|icmp)\s

[src_ip_for_junos_fw]
REGEX = \sPFE_FW_SYSLOG_ETH_IP:\sFW:\s[^ ]+\s+[^ ]\s[^ ]+\s[^ ]+\s*[^ ]+\s[^ ]+\s+[^ ]+\s+(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

Every field extraction is working except for src. I validated on https://regex101.com/, ran the regex via rex on the search line, and even ran the regex through "Extract Fields" and values for src were returned. I did restart Splunk after modifying the conf files.

Any ideas?
Thx

0 Karma
1 Solution

mayurr98
Super Champion

okay try this regex which covers four extractions : filter , action, src_interface , transport , src

remove all 5 regexes and put all in one.

https://regex101.com/r/rYVwTS/2

\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

Or if you want to do it separately then try:

\sPFE_FW_SYSLOG_ETH_IP:\sFW:\s.*(tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

View solution in original post

mayurr98
Super Champion

okay try this regex which covers four extractions : filter , action, src_interface , transport , src

remove all 5 regexes and put all in one.

https://regex101.com/r/rYVwTS/2

\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

Or if you want to do it separately then try:

\sPFE_FW_SYSLOG_ETH_IP:\sFW:\s.*(tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

jwalzerpitt
Influencer

Thx for the regex - I added the src and dest ports capture groups:

\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s.*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s(?<dst>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s(?<src_port>[^ ]+)\s+(?<dest_port>[^ ]+)\s+

and the issue I have is that last event is not extracting fields

https://regex101.com/r/rYVwTS/3

How does one account for K/V pairs that are sometimes there and sometimes not as the 3rd event does not have a src or dest port listed in the event?

Thx

0 Karma

mayurr98
Super Champion

try the updated answer and it should work:

\s(?<filter>PFE_FW_SYSLOG_ETH.*:)\sFW:\s(?<src_interface>[^ ]+)\s+(?<action>\w+).*(?<transport>tcp|udp|icmp)\s(?<src>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$)\s

jwalzerpitt
Influencer

Modified the regex to as follows - https://regex101.com/r/rYVwTS/6 - and I think it's extracting fields for all events.

Weirdness with the icmp event as I think the 3 listed twice is the source and destination bytes. Can't find any clarifying documentation on Juniper's site other than this - https://forums.juniper.net/t5/Junos/What-is-the-log-format-for-deny-hosts-ACL/m-p/318906 - which talks about not icmp denied traffic, but I can live with it.

Thx a million for the help!

0 Karma

jwalzerpitt
Influencer

This regex - https://regex101.com/r/rYVwTS/4 - doesn't pick up the dst_port for the 1st event and picks up the value on the 3rd event and assigns it to dst_port even though it's not.

Thx

0 Karma

guarisma
Contributor

Are all these extractions for \sPFE_FW_SYSLOG_ETH_IP:\sFW: for one event?
I mean, are you expecting that any event that matches the \sPFE_FW_SYSLOG_ETH_IP:\sFW: would do the last 4 extractions?

If yes, I would just do one big regex with all the extractions.

Also, make sure you don't have another knowledge object rewriting the src field, like a lookup table

0 Karma

jwalzerpitt
Influencer

The problem I have is that the Juniper logs have different event types that every time I try and do on regex I miss some relevant events so I figured that breaking the regex down to simple field extractions and then building on that would ensure I extract K/V pairs for all relevant events.

I wish I could write one regex that covers all events, but I'm not a regex guru

I do not have a lookup table rewriting the src field. Would running btool show all knowledge objects (using grep for relevant fields)?

Thx

0 Karma

mayurr98
Super Champion

hi

Can you give us a sample event? and which one you are exactly talking about src_interface_for_junos_fw or src_ip_for_junos_fw?

0 Karma

jwalzerpitt
Influencer

Sample data:

Aug  2 09:16:37 10.10.10.10 Aug  2 09:16:37 externals-cl fpc1 PFE_FW_SYSLOG_ETH_IP: FW: xe-1/0/0.0   D 0800 34:62:5a:74:8f:c3 -> 64:b2:9a:7e:1b:4a  tcp 184.154.189.91 10.1.1.1 41860   465 (1 packets)

Aug  2 09:23:13 10.10.10.10 Aug  2 09:23:13 externals-fq fpc0 PFE_FW_SYSLOG_ETH_IP: FW: xe-0/0/0.447 D 03af:0700 a6:e7:f2:2e:13:c7 -> 72:a3:9c:3a:22:00  icmp 185.176.27.46 10.1.1.1 44927 53389 (1 packets)

Aug  2 10:00:42 10.10.10.10 Aug  2 10:00:42 externals-fq fpc2 PFE_FW_SYSLOG_ETH_IP: FW: et-2/1/0.716 D 02cc:0800 52:a2:2f:7a:1d:5a -> 84:c9:2b:9e:24:e6 icmp 10.1.1.2 10.1.1.1     3     3 (1 packets)

src_interface_for_junos_fw and src_ip_for_junos_fw and stanza header names I assigned as I looked in the /opt/splunk/etc/apps/Splunk_TA_juniper/default directory to for some guidance

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...