topic Re: Why does my regular expression work in search, but it does not work in transforms.conf? in Splunk Search

Why does my regular expression work in search, but it does not work in transforms.conf?

murhammr — Thu, 09 Feb 2017 22:07:00 GMT

I'm having trouble converting a search string into a working regular expression in transforms.conf to send events to the nullQueue. here is a sample XML event:

<record version="2" event="stat(2)" modifier="fe" host="hostname.goeshere.com " iso8601="2017-02-04 04:03:52.223 -06:00"> <path>/path/to/oracle/product/version/db/lib/libavl.so.1</path> <subject audit-uid="username" uid="oracle" gid="dba" ruid="oracle" rgid="dba" pid="18395" sid="2390772688" tid="16257 131094 hostname.goeshere.com"/> <return errval="failure: No such file or directory" retval="-1"/> </record>

i want to send an event to nullQueue if all 3 strings are in the event:

event="stat(2)"    
uid="oracle"
retval="-1"

i can craft a regex that finds these entries in search

"event="stat(2)"*uid="oracle"*retval="-1""

but i can't seem to figure out how to get this working properly in transforms.conf. I've tried removing the outer set of quotes, escaping the quotes, escaping the non-alphas, using different regex for the wildcards besides *.

cat transforms.conf

[null_queue_filter]
REGEX = event=\"stat(2)\"*uid=\"oracle\"*retval=\"-1\"
DEST_KEY = queue
FORMAT = nullQueue

cat props.conf

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter

Re: Why does my regular expression work in search, but it does not work in transforms.conf?

ehudb — Tue, 29 Sep 2020 12:49:21 GMT

I didn't see you pointed the props to use the transforms:
TRANSFORMS-audit_xml= audit_xml

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601=\"
BREAK_ONLY_BEFORE = \
TRANSFORMS-audit_xml= audit_xml

The following REGEX worked, tested at regex101.com:

event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"

transforms:

[null_queue_filter] 
REGEX = event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"
DEST_KEY = queue 
FORMAT = nullQueue

Re: Why does my regular expression work in search, but it does not work in transforms.conf?

murhammr — Thu, 09 Feb 2017 23:00:21 GMT

i must have omitted part of my props.conf in the original post:

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \<record
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter

Re: Why does my regular expression work in search, but it does not work in transforms.conf?

alacercogitatus — Sat, 11 Feb 2017 17:47:17 GMT

Escaping quotes is not necessary in the Transforms.conf, and additionally, for the REGEX to match and filter, you must have a capture group. Be careful with the uid matching, as your sample data has ruid which might match and be a false positive. So in the below regex, I made the .* capture non-greedy to capture up to the first instance of uid=, instead of the match of ruid.

 REGEX = (event="stat\(2\)".*?uid="oracle".+retval="-1")

This should filter your events to the null queue.

Re: Why does my regular expression work in search, but it does not work in transforms.conf?

murhammr — Tue, 14 Feb 2017 19:29:05 GMT

Thanks alacercogitatus but this is not working for me either. I tried your REGEX on my heavy forwarder but these events are still getting to my indexer.

Re: Why does my regular expression work in search, but it does not work in transforms.conf?

somesoni2 — Tue, 14 Feb 2017 19:48:21 GMT

Give this a try

 REGEX = event=\"stat\(2\)\".+\suid=\"oracle\".+\sretval=\"-1\"

Re: Why does my regular expression work in search, but it does not work in transforms.conf?

murhammr — Tue, 14 Feb 2017 22:37:49 GMT

Thanks somesoni2. your REGEX works when i test it at regex101.com but not in my transforms.conf. this data is still getting to my indexer.

Re: Why does my regular expression work in search, but it does not work in transforms.conf?

murhammr — Tue, 14 Feb 2017 23:26:40 GMT

I finally got this working. This is my working REGEX in transforms.conf.

REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")

This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html

index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"

I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .* ) didn't work.