I'm having trouble converting a search string into a working regular expression in transforms.conf to send events to the nullQueue. here is a sample XML event:
<record version="2" event="stat(2)" modifier="fe" host="hostname.goeshere.com " iso8601="2017-02-04 04:03:52.223 -06:00"> <path>/path/to/oracle/product/version/db/lib/libavl.so.1</path> <subject audit-uid="username" uid="oracle" gid="dba" ruid="oracle" rgid="dba" pid="18395" sid="2390772688" tid="16257 131094 hostname.goeshere.com"/> <return errval="failure: No such file or directory" retval="-1"/> </record>
i want to send an event to nullQueue if all 3 strings are in the event:
event="stat(2)"
uid="oracle"
retval="-1"
i can craft a regex that finds these entries in search
"event="stat(2)"*uid="oracle"*retval="-1""
but i can't seem to figure out how to get this working properly in transforms.conf. I've tried removing the outer set of quotes, escaping the quotes, escaping the non-alphas, using different regex for the wildcards besides *.
cat transforms.conf
[null_queue_filter]
REGEX = event=\"stat(2)\"*uid=\"oracle\"*retval=\"-1\"
DEST_KEY = queue
FORMAT = nullQueue
cat props.conf
[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter
I finally got this working. This is my working REGEX in transforms.conf.
REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")
This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html
index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"
I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .*
) didn't work.
I finally got this working. This is my working REGEX in transforms.conf.
REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")
This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html
index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"
I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .*
) didn't work.
Give this a try
REGEX = event=\"stat\(2\)\".+\suid=\"oracle\".+\sretval=\"-1\"
Thanks somesoni2. your REGEX works when i test it at regex101.com but not in my transforms.conf. this data is still getting to my indexer.
Escaping quotes is not necessary in the Transforms.conf, and additionally, for the REGEX
to match and filter, you must have a capture group. Be careful with the uid
matching, as your sample data has ruid
which might match and be a false positive. So in the below regex, I made the .*
capture non-greedy to capture up to the first instance of uid=
, instead of the match of ruid
.
REGEX = (event="stat\(2\)".*?uid="oracle".+retval="-1")
This should filter your events to the null queue.
Thanks alacercogitatus but this is not working for me either. I tried your REGEX on my heavy forwarder but these events are still getting to my indexer.
I didn't see you pointed the props to use the transforms:
TRANSFORMS-audit_xml= audit_xml
[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601=\"
BREAK_ONLY_BEFORE = \
TRANSFORMS-audit_xml= audit_xml
The following REGEX worked, tested at regex101.com:
event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"
transforms:
[null_queue_filter]
REGEX = event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"
DEST_KEY = queue
FORMAT = nullQueue
i must have omitted part of my props.conf in the original post:
[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \<record
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter