I am trying to build a filter so I only index events that match this regex:
.*[%].*
I asked a question previously that was answered
I was able to change the data coming into my system using this transform:
[filter-debug]
REGEX=.*[%].*
DEST_KEY = _raw
FORMAT = $1
The problem I am experiencing is that anything matching my REGEX has it's _raw replaced with whatever I set the FORMAT to be. So in the example above all of my events matching my regex would return a literal "$1" as their event data. This was the case no matter what I put into FORMAT and I can't get rid of FORMAT because it just defaults to
[filter-debug]::$
And so then that's what all my events look like. Thoughts?
I believe I figured it out. My new transforms.conf looks like this:
# drop everything that doesn't match [passthru]
[drop-debug]
REGEX =.*
DEST_KEY = queue
FORMAT = nullQueue
# send everything else on to the indexer
[passthru]
REGEX=.*[%].*
DEST_KEY = queue
FORMAT = indexQueue
I think my previous problem was trying to send to _raw after sending to indexQueue didn't work.
I believe I figured it out. My new transforms.conf looks like this:
# drop everything that doesn't match [passthru]
[drop-debug]
REGEX =.*
DEST_KEY = queue
FORMAT = nullQueue
# send everything else on to the indexer
[passthru]
REGEX=.*[%].*
DEST_KEY = queue
FORMAT = indexQueue
I think my previous problem was trying to send to _raw after sending to indexQueue didn't work.
Usually $1
refers to the first capturing group, but I don't see a capturing group in your regex?
Could you clarify what a capturing group is? If I just wrapped my regex in a couple parans would that make it a capturing group?
A capturing group is basically within ( )
have a read here https://answers.splunk.com/answers/214487/can-i-extract-a-field-with-a-regexed-dynamic-field.html where I used two capturing groups $1
and $2
both were created using this regex ([a-z]+)=([a-z]+)
.
Also starting a regex with .*
is .... let's call suboptimal because it matches everything. I would optimise the nullQueue regex to match the things you don't want and skip the [passthru]
completely 😉
cheers, MuS