topic Re: How to debug transforms.conf regex errors in Splunk Search

How to debug transforms.conf regex errors

huaraz — Mon, 28 Sep 2020 19:46:56 GMT

I have a logfile with different formated lines and I want to extract comon fields . My props.conf looks like:

[mylog]
TRUNCATE=0
SHOULD_LINEMERGE=false
TIME_PREFIX=(
TRANSFORMS-header_null_queue=send_header_to_nullqueue
REPORT-main=type1, type2, type3
TIME_FORMAT=%s.%6N

[send_header_to_nullqueue]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

[type1]
REGEX=...

[type2]
REGEX=..

[type3]
REGEX=..

I checked using pcregextest that each regex works for my 3 types of log lines. But all together seems not to work as I don't get the fields.

Thank you
Markus

Re: How to debug transforms.conf regex errors

woodcock — Sun, 10 May 2015 20:04:23 GMT

I use expresso:
http://www.ultrapico.com/expresso.htm

This tool allows you to paste your log text as-is into one window and paste your REGEX as-is into another window and check the results. If your REGEXs work in expresso then either there is something wrong with your configuration file organization or your (possibly incorrect expectations of) search.

Re: How to debug transforms.conf regex errors

Runals — Mon, 28 Sep 2020 19:56:16 GMT

I use regex101.com. Just to confirm though because it isn't in your examples

1) do your REGEX lines exist in your props.conf or your transforms.conf? Your send_header_to_nullqueue, type1 - type3 stanzas need to live in a transforms.conf
2) do your REGEX lines in type1 - 3 have a capturing group with a field name in them? For example

REGEX= I want (foo) for breakfast

has a capturing group but there is no defined field. You could do one of the approaches below

REGEX = I want (?<breakfast_food>foo) for breakfast
or
REGEX = I want (?<breakfast_food>\S) for breakfast

REGEX = I want (foo) for breakfast
FORMAT = breakfast_food::$1