Splunk Search

Using the transforms.conf file to only forward events that match a regex.

JordanPeterson
Path Finder

I've got a log file that get's 2 different event formats depending on if debugging is turned on. When debugging is turned on I don't want the debug events forwarded but I do want the normal events forwarded as normal.

I have a regular expression that will only include my normal events that looks like this: [0-9]*:.*[%].*
I know that I can create a transforms.conf file in $SPLUNK_HOME/etc/apps/appName/local to filter events.

In inputs.conf I have the following:

[monitor:///var/log/boot.log]
disabled = false
followTail = 0
index = zod-os
sourcetype = linux_bootlog

I think if I add the following to transforms.conf it will do what I want:
[linux_bootlog]
REGEX = [0-9]*:.*[%].*

What I'm not 100% sure of is if I need to create a props.conf file to point to the transform like I've seen in other answers. I don't want to extract any additional fields other than what Splunk appears to automatically be doing. Also, the debug events are multiline but since they don't match the regex I think they will drop automatically.

Does all of that sound like it will work?

0 Karma
1 Solution

DalJeanis
Legend

Yes, props.conf is what tells the system when to run the transforms in transforms.conf, so there needs to be a stanza there for your source (or however you want to identify the stuff that you are treating this way).

What I would tend to do is route everything to the nullqueue and then route the matching events back.

props.conf

 [source::/var/log/foo]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-set = setnull, setparsing

transforms.conf

[setnull]
REGEX=.* 
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]*:.*[%].*
DEST_KEY = queue
FORMAT = nullQueue

The above is barely changed at all (only the second regex) from this answer... https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html

View solution in original post

DalJeanis
Legend

Yes, props.conf is what tells the system when to run the transforms in transforms.conf, so there needs to be a stanza there for your source (or however you want to identify the stuff that you are treating this way).

What I would tend to do is route everything to the nullqueue and then route the matching events back.

props.conf

 [source::/var/log/foo]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-set = setnull, setparsing

transforms.conf

[setnull]
REGEX=.* 
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]*:.*[%].*
DEST_KEY = queue
FORMAT = nullQueue

The above is barely changed at all (only the second regex) from this answer... https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html

morethanyell
Builder

setparsing stanza should be FORMAT = indexQueue

JordanPeterson
Path Finder

This is exactly the kind of thing I needed. Thank you very much.

Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...