Why does my regular expression work in search, but it does not work in transforms.conf?

[murhammr]

I'm having trouble converting a search string into a working regular expression in transforms.conf to send events to the nullQueue. here is a sample XML event:

<record version="2" event="stat(2)" modifier="fe" host="hostname.goeshere.com " iso8601="2017-02-04 04:03:52.223 -06:00"> <path>/path/to/oracle/product/version/db/lib/libavl.so.1</path> <subject audit-uid="username" uid="oracle" gid="dba" ruid="oracle" rgid="dba" pid="18395" sid="2390772688" tid="16257 131094 hostname.goeshere.com"/> <return errval="failure: No such file or directory" retval="-1"/> </record>

i want to send an event to nullQueue if all 3 strings are in the event:

event="stat(2)"    
uid="oracle"
retval="-1"

i can craft a regex that finds these entries in search

"event="stat(2)"*uid="oracle"*retval="-1""

but i can't seem to figure out how to get this working properly in transforms.conf. I've tried removing the outer set of quotes, escaping the quotes, escaping the non-alphas, using different regex for the wildcards besides *.

cat transforms.conf

[null_queue_filter]
REGEX = event=\"stat(2)\"*uid=\"oracle\"*retval=\"-1\"
DEST_KEY = queue
FORMAT = nullQueue

cat props.conf

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter

[murhammr · Accepted solution]

I finally got this working. This is my working REGEX in transforms.conf.

REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")

This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html

index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"

I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .* ) didn't work.

[somesoni2]

Give this a try

 REGEX = event=\"stat\(2\)\".+\suid=\"oracle\".+\sretval=\"-1\"

[murhammr]

Thanks somesoni2. your REGEX works when i test it at regex101.com but not in my transforms.conf. this data is still getting to my indexer.

[alacercogitatus]

Escaping quotes is not necessary in the Transforms.conf, and additionally, for the REGEX to match and filter, you must have a capture group. Be careful with the uid matching, as your sample data has ruid which might match and be a false positive. So in the below regex, I made the .* capture non-greedy to capture up to the first instance of uid=, instead of the match of ruid.

 REGEX = (event="stat\(2\)".*?uid="oracle".+retval="-1")

This should filter your events to the null queue.

[murhammr]

Thanks alacercogitatus but this is not working for me either. I tried your REGEX on my heavy forwarder but these events are still getting to my indexer.

[ehudb]

1. I didn't see you pointed the props to use the transforms:
   TRANSFORMS-audit_xml= audit_xml

   [audit_xml]
   KV_MODE = xml
   TIME_PREFIX = iso8601=\"
   BREAK_ONLY_BEFORE = \
   TRANSFORMS-audit_xml= audit_xml

The following REGEX worked, tested at regex101.com:

event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"

transforms:

[null_queue_filter] 
REGEX = event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"
DEST_KEY = queue 
FORMAT = nullQueue 

[murhammr]

i must have omitted part of my props.conf in the original post:

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \<record
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter