All Apps and Add-ons

Splunk add-on for opsec (4.0): Handling fields containg escaped pipes (\|)

hcpr
Path Finder

Hi, I'm trying to find a good way to handle a situation with logs from CheckPoint URL filter and Application Control.
They contain the relevant URL's in the resource field. The field separater is by default the pipe character '|'.
However some of the logentries contain URLs tha themselves contain pipes that are escaped.
This is an example from the raw log from lea-loggrabber:

http://fonts.googleapis.com/css?family=Roboto Condensed:400,300,700\|Open Sans:400,300,600,700\|Roboto:400,100,300,500,700\|Cuprum:400,700

The problem is that Splunk parses the pipe as a field separator. ANy suggestion son how to prevent this?

I've tried to just replace the text with SEDCMD, but I haven't been able to get that to work so far.
So any suggestions would be appreaciated 🙂

Thanks

0 Karma
1 Solution

jamesarmitage
Path Finder

The regex in the auto_kv_for_opsec stanza in transforms.conf breaks when parsing the | in the google fonts URLs.

Create a local folder in the TA and make copies of the default props.conf and transforms.conf.

I added the following to my local transforms.conf, just beneath the [auto_kv_for_opsec] stanza:

#fixes resource field, specifically for some google fonts URLs which contain \|
[application_control_resource]
REGEX = resource=(.*[^\\|])\|proxy_src_ip
FORMAT = resource::$1

and in the local props.conf, just below the REPORT-auto_kv_for_opsec = auto_kv_for_opsec line I added:

REPORT-application_control_resource = application_control_resource

This will give you a clean resource field at search time.

View solution in original post

jamesarmitage
Path Finder

The regex in the auto_kv_for_opsec stanza in transforms.conf breaks when parsing the | in the google fonts URLs.

Create a local folder in the TA and make copies of the default props.conf and transforms.conf.

I added the following to my local transforms.conf, just beneath the [auto_kv_for_opsec] stanza:

#fixes resource field, specifically for some google fonts URLs which contain \|
[application_control_resource]
REGEX = resource=(.*[^\\|])\|proxy_src_ip
FORMAT = resource::$1

and in the local props.conf, just below the REPORT-auto_kv_for_opsec = auto_kv_for_opsec line I added:

REPORT-application_control_resource = application_control_resource

This will give you a clean resource field at search time.

hcpr
Path Finder

Thank you! This worked fine.
The resource field is cleand up, and the pipes removed. And the URl still works if needed for some reason.

0 Karma

astalv
New Member

You can change separator to any other symbol, you like, in usual lea loggraber it is in fw1-loggrabber.conf for example RECORD_SEPARATOR="|"

May be make a special extraction for this field, which will start at "|resource=" and ends with the beginning of next field?

0 Karma

hcpr
Path Finder

Hi, thanks for thew suggestion.
I'm not sure if I easily can get the app to read the fw1-loggrabber.conf file. It does not by default.

Any suggestions on how I can get Splunk to ignore escaped field separators? eg. "|"

0 Karma

astalv
New Member

It seems that in a 4.0 opsec app it is impossible to change separator, i have tried to find it in config files and couldn't ...

OK, if you make a new field with custom regex will it solve your problem? As i wrote, make a new field which started by "|resource=" and ends with the beginning of next field

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...