Hi, I'm trying to find a good way to handle a situation with logs from CheckPoint URL filter and Application Control.
They contain the relevant URL's in the resource
field. The field separater is by default the pipe character '|'.
However some of the logentries contain URLs tha themselves contain pipes that are escaped.
This is an example from the raw log from lea-loggrabber:
http://fonts.googleapis.com/css?family=Roboto Condensed:400,300,700\|Open Sans:400,300,600,700\|Roboto:400,100,300,500,700\|Cuprum:400,700
The problem is that Splunk parses the pipe as a field separator. ANy suggestion son how to prevent this?
I've tried to just replace the text with SEDCMD, but I haven't been able to get that to work so far.
So any suggestions would be appreaciated 🙂
Thanks
The regex in the auto_kv_for_opsec stanza in transforms.conf
breaks when parsing the | in the google fonts URLs.
Create a local folder in the TA and make copies of the default props.conf and transforms.conf.
I added the following to my local transforms.conf, just beneath the [auto_kv_for_opsec]
stanza:
#fixes resource field, specifically for some google fonts URLs which contain \|
[application_control_resource]
REGEX = resource=(.*[^\\|])\|proxy_src_ip
FORMAT = resource::$1
and in the local props.conf, just below the REPORT-auto_kv_for_opsec = auto_kv_for_opsec
line I added:
REPORT-application_control_resource = application_control_resource
This will give you a clean resource field at search time.
The regex in the auto_kv_for_opsec stanza in transforms.conf
breaks when parsing the | in the google fonts URLs.
Create a local folder in the TA and make copies of the default props.conf and transforms.conf.
I added the following to my local transforms.conf, just beneath the [auto_kv_for_opsec]
stanza:
#fixes resource field, specifically for some google fonts URLs which contain \|
[application_control_resource]
REGEX = resource=(.*[^\\|])\|proxy_src_ip
FORMAT = resource::$1
and in the local props.conf, just below the REPORT-auto_kv_for_opsec = auto_kv_for_opsec
line I added:
REPORT-application_control_resource = application_control_resource
This will give you a clean resource field at search time.
Thank you! This worked fine.
The resource field is cleand up, and the pipes removed. And the URl still works if needed for some reason.
You can change separator to any other symbol, you like, in usual lea loggraber it is in fw1-loggrabber.conf for example RECORD_SEPARATOR="|"
May be make a special extraction for this field, which will start at "|resource=" and ends with the beginning of next field?
Hi, thanks for thew suggestion.
I'm not sure if I easily can get the app to read the fw1-loggrabber.conf file. It does not by default.
Any suggestions on how I can get Splunk to ignore escaped field separators? eg. "|"
It seems that in a 4.0 opsec app it is impossible to change separator, i have tried to find it in config files and couldn't ...
OK, if you make a new field with custom regex will it solve your problem? As i wrote, make a new field which started by "|resource=" and ends with the beginning of next field