Getting Data In

transforms.conf returning events that match REGEX value as value of FORMAT key

JordanPeterson
Path Finder

I am trying to build a filter so I only index events that match this regex:

.*[%].* 

I asked a question previously that was answered

I was able to change the data coming into my system using this transform:

[filter-debug]
REGEX=.*[%].*
DEST_KEY = _raw
FORMAT = $1

The problem I am experiencing is that anything matching my REGEX has it's _raw replaced with whatever I set the FORMAT to be. So in the example above all of my events matching my regex would return a literal "$1" as their event data. This was the case no matter what I put into FORMAT and I can't get rid of FORMAT because it just defaults to

[filter-debug]::$

And so then that's what all my events look like. Thoughts?

Tags (2)
0 Karma
1 Solution

JordanPeterson
Path Finder

I believe I figured it out. My new transforms.conf looks like this:

# drop everything that doesn't match [passthru]
 [drop-debug]
  REGEX =.*
  DEST_KEY = queue
  FORMAT = nullQueue

# send everything else on to the indexer
 [passthru]
 REGEX=.*[%].*
 DEST_KEY = queue
 FORMAT = indexQueue

I think my previous problem was trying to send to _raw after sending to indexQueue didn't work.

View solution in original post

0 Karma

JordanPeterson
Path Finder

I believe I figured it out. My new transforms.conf looks like this:

# drop everything that doesn't match [passthru]
 [drop-debug]
  REGEX =.*
  DEST_KEY = queue
  FORMAT = nullQueue

# send everything else on to the indexer
 [passthru]
 REGEX=.*[%].*
 DEST_KEY = queue
 FORMAT = indexQueue

I think my previous problem was trying to send to _raw after sending to indexQueue didn't work.

0 Karma

MuS
Legend

Usually $1 refers to the first capturing group, but I don't see a capturing group in your regex?

JordanPeterson
Path Finder

Could you clarify what a capturing group is? If I just wrapped my regex in a couple parans would that make it a capturing group?

0 Karma

MuS
Legend

A capturing group is basically within ( ) have a read here https://answers.splunk.com/answers/214487/can-i-extract-a-field-with-a-regexed-dynamic-field.html where I used two capturing groups $1 and $2 both were created using this regex ([a-z]+)=([a-z]+).

Also starting a regex with .* is .... let's call suboptimal because it matches everything. I would optimise the nullQueue regex to match the things you don't want and skip the [passthru] completely 😉

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...