transforms.conf returning events that match REGEX value as value of FORMAT key

[JordanPeterson]

I am trying to build a filter so I only index events that match this regex:

.*[%].* 

I asked a question previously that was answered

I was able to change the data coming into my system using this transform:

[filter-debug]
REGEX=.*[%].*
DEST_KEY = _raw
FORMAT = $1

The problem I am experiencing is that anything matching my REGEX has it's _raw replaced with whatever I set the FORMAT to be. So in the example above all of my events matching my regex would return a literal "$1" as their event data. This was the case no matter what I put into FORMAT and I can't get rid of FORMAT because it just defaults to

[filter-debug]::$

And so then that's what all my events look like. Thoughts?

[JordanPeterson · Accepted solution]

I believe I figured it out. My new transforms.conf looks like this:

# drop everything that doesn't match [passthru]
 [drop-debug]
  REGEX =.*
  DEST_KEY = queue
  FORMAT = nullQueue

# send everything else on to the indexer
 [passthru]
 REGEX=.*[%].*
 DEST_KEY = queue
 FORMAT = indexQueue

I think my previous problem was trying to send to _raw after sending to indexQueue didn't work.

[MuS]

Usually $1 refers to the first capturing group, but I don't see a capturing group in your regex?

[JordanPeterson]

Could you clarify what a capturing group is? If I just wrapped my regex in a couple parans would that make it a capturing group?

[MuS]

A capturing group is basically within ( ) have a read here https://answers.splunk.com/answers/214487/can-i-extract-a-field-with-a-regexed-dynamic-field.html where I used two capturing groups $1 and $2 both were created using this regex ([a-z]+)=([a-z]+).

Also starting a regex with .* is .... let's call suboptimal because it matches everything. I would optimise the nullQueue regex to match the things you don't want and skip the [passthru] completely 😉

cheers, MuS