Splunk Search

Why does my regular expression work in search, but it does not work in transforms.conf?

Path Finder

I'm having trouble converting a search string into a working regular expression in transforms.conf to send events to the nullQueue. here is a sample XML event:

<record version="2" event="stat(2)" modifier="fe" host="hostname.goeshere.com " iso8601="2017-02-04 04:03:52.223 -06:00"> <path>/path/to/oracle/product/version/db/lib/libavl.so.1</path> <subject audit-uid="username" uid="oracle" gid="dba" ruid="oracle" rgid="dba" pid="18395" sid="2390772688" tid="16257 131094 hostname.goeshere.com"/> <return errval="failure: No such file or directory" retval="-1"/> </record>

i want to send an event to nullQueue if all 3 strings are in the event:

event="stat(2)"    
uid="oracle"
retval="-1"

i can craft a regex that finds these entries in search

"event="stat(2)"*uid="oracle"*retval="-1""

but i can't seem to figure out how to get this working properly in transforms.conf. I've tried removing the outer set of quotes, escaping the quotes, escaping the non-alphas, using different regex for the wildcards besides *.

cat transforms.conf

[null_queue_filter]
REGEX = event=\"stat(2)\"*uid=\"oracle\"*retval=\"-1\"
DEST_KEY = queue
FORMAT = nullQueue

cat props.conf

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter
0 Karma
1 Solution

Path Finder

I finally got this working. This is my working REGEX in transforms.conf.

REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")

This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html

index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"

I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .* ) didn't work.

View solution in original post

0 Karma

Path Finder

I finally got this working. This is my working REGEX in transforms.conf.

REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")

This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html

index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"

I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .* ) didn't work.

View solution in original post

0 Karma

Revered Legend

Give this a try

 REGEX = event=\"stat\(2\)\".+\suid=\"oracle\".+\sretval=\"-1\"
0 Karma

Path Finder

Thanks somesoni2. your REGEX works when i test it at regex101.com but not in my transforms.conf. this data is still getting to my indexer.

0 Karma

SplunkTrust
SplunkTrust

Escaping quotes is not necessary in the Transforms.conf, and additionally, for the REGEX to match and filter, you must have a capture group. Be careful with the uid matching, as your sample data has ruid which might match and be a false positive. So in the below regex, I made the .* capture non-greedy to capture up to the first instance of uid=, instead of the match of ruid.

 REGEX = (event="stat\(2\)".*?uid="oracle".+retval="-1")

This should filter your events to the null queue.

Path Finder

Thanks alacercogitatus but this is not working for me either. I tried your REGEX on my heavy forwarder but these events are still getting to my indexer.

0 Karma

Contributor
  1. I didn't see you pointed the props to use the transforms:
    TRANSFORMS-audit_xml= audit_xml

    [audit_xml]
    KV_MODE = xml
    TIME_PREFIX = iso8601=\"
    BREAK_ONLY_BEFORE = \
    TRANSFORMS-audit_xml= audit_xml

The following REGEX worked, tested at regex101.com:

event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"

transforms:

[null_queue_filter] 
REGEX = event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"
DEST_KEY = queue 
FORMAT = nullQueue 
0 Karma

Path Finder

i must have omitted part of my props.conf in the original post:

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \<record
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!