Splunk Search

Why does my regular expression work in search, but it does not work in transforms.conf?

murhammr
Path Finder

I'm having trouble converting a search string into a working regular expression in transforms.conf to send events to the nullQueue. here is a sample XML event:

<record version="2" event="stat(2)" modifier="fe" host="hostname.goeshere.com " iso8601="2017-02-04 04:03:52.223 -06:00"> <path>/path/to/oracle/product/version/db/lib/libavl.so.1</path> <subject audit-uid="username" uid="oracle" gid="dba" ruid="oracle" rgid="dba" pid="18395" sid="2390772688" tid="16257 131094 hostname.goeshere.com"/> <return errval="failure: No such file or directory" retval="-1"/> </record>

i want to send an event to nullQueue if all 3 strings are in the event:

event="stat(2)"    
uid="oracle"
retval="-1"

i can craft a regex that finds these entries in search

"event="stat(2)"*uid="oracle"*retval="-1""

but i can't seem to figure out how to get this working properly in transforms.conf. I've tried removing the outer set of quotes, escaping the quotes, escaping the non-alphas, using different regex for the wildcards besides *.

cat transforms.conf

[null_queue_filter]
REGEX = event=\"stat(2)\"*uid=\"oracle\"*retval=\"-1\"
DEST_KEY = queue
FORMAT = nullQueue

cat props.conf

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter
0 Karma
1 Solution

murhammr
Path Finder

I finally got this working. This is my working REGEX in transforms.conf.

REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")

This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html

index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"

I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .* ) didn't work.

View solution in original post

0 Karma

murhammr
Path Finder

I finally got this working. This is my working REGEX in transforms.conf.

REGEX = (?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")

This was a great debugging tip from the answer entitled REGEX and NullQueue problem: https://answers.splunk.com/answers/108326/regex-and-nullqueue-problem.html

index=blah| regex _raw="(?:event\=\"stat\(2\)\"(\w|\W)*\suid\=\"oracle\"(\w|\W)*retval\=\"-1\")"

I could successfully find events with any of the 3 string parts event=\"stat(2)\" or \suid=\"oracle\" or retval=\"-1\"
but putting them together was the problem. Not sure exactly why the other wildcard regex ( * or .+ or .* ) didn't work.

0 Karma

somesoni2
Revered Legend

Give this a try

 REGEX = event=\"stat\(2\)\".+\suid=\"oracle\".+\sretval=\"-1\"
0 Karma

murhammr
Path Finder

Thanks somesoni2. your REGEX works when i test it at regex101.com but not in my transforms.conf. this data is still getting to my indexer.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Escaping quotes is not necessary in the Transforms.conf, and additionally, for the REGEX to match and filter, you must have a capture group. Be careful with the uid matching, as your sample data has ruid which might match and be a false positive. So in the below regex, I made the .* capture non-greedy to capture up to the first instance of uid=, instead of the match of ruid.

 REGEX = (event="stat\(2\)".*?uid="oracle".+retval="-1")

This should filter your events to the null queue.

murhammr
Path Finder

Thanks alacercogitatus but this is not working for me either. I tried your REGEX on my heavy forwarder but these events are still getting to my indexer.

0 Karma

ehudb
Contributor
  1. I didn't see you pointed the props to use the transforms:
    TRANSFORMS-audit_xml= audit_xml

    [audit_xml]
    KV_MODE = xml
    TIME_PREFIX = iso8601=\"
    BREAK_ONLY_BEFORE = \
    TRANSFORMS-audit_xml= audit_xml

The following REGEX worked, tested at regex101.com:

event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"

transforms:

[null_queue_filter] 
REGEX = event=\"stat\(2\)\".*uid=\"oracle\".*retval=\"-1\"
DEST_KEY = queue 
FORMAT = nullQueue 
0 Karma

murhammr
Path Finder

i must have omitted part of my props.conf in the original post:

[audit_xml]
KV_MODE = xml
TIME_PREFIX = iso8601\=\"
BREAK_ONLY_BEFORE = \<record
SHOULD_LINEMERGE = true
TRANSFORMS-audit_xml = null_queue_filter
0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...