Using the transforms.conf file to only forward events that match a regex.

[JordanPeterson]

I've got a log file that get's 2 different event formats depending on if debugging is turned on. When debugging is turned on I don't want the debug events forwarded but I do want the normal events forwarded as normal.

I have a regular expression that will only include my normal events that looks like this: [0-9]*:.*[%].*
I know that I can create a transforms.conf file in $SPLUNK_HOME/etc/apps/appName/local to filter events.

In inputs.conf I have the following:

[monitor:///var/log/boot.log]
disabled = false
followTail = 0
index = zod-os
sourcetype = linux_bootlog

I think if I add the following to transforms.conf it will do what I want:
[linux_bootlog]
REGEX = [0-9]*:.*[%].*

What I'm not 100% sure of is if I need to create a props.conf file to point to the transform like I've seen in other answers. I don't want to extract any additional fields other than what Splunk appears to automatically be doing. Also, the debug events are multiline but since they don't match the regex I think they will drop automatically.

Does all of that sound like it will work?

[DalJeanis · Accepted solution]

Yes, props.conf is what tells the system when to run the transforms in transforms.conf, so there needs to be a stanza there for your source (or however you want to identify the stuff that you are treating this way).

What I would tend to do is route everything to the nullqueue and then route the matching events back.

props.conf

 [source::/var/log/foo]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-set = setnull, setparsing

transforms.conf

[setnull]
REGEX=.* 
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]*:.*[%].*
DEST_KEY = queue
FORMAT = nullQueue

The above is barely changed at all (only the second regex) from this answer... https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html

[morethanyell]

setparsing stanza should be FORMAT = indexQueue

[JordanPeterson]

This is exactly the kind of thing I needed. Thank you very much.