Splunk Search

Use of tokenizer option with makemv

agodoy
Communicator

I am trying to break a field based on some regex. Apparently this can be done with the tokenizer option of the makemv command. However, there is no example on how to use it and I keep getting the following error when I try "Error in 'makemv' command: The tokenizer regular expression is invalid"

Basically, I am trying to break on commas(,) that are not followed by a blank space.

End goal: "4,Something" would result in a new value, but "4, Something" would not.

Tags (2)
0 Karma
1 Solution

jonuwz
Influencer

Example :

| gentimes start=-1 
| eval john="1 something,2 something else,3 something, with a comma,4 wibble"
| table john
| makemv tokenizer="(.+?)(?=,\S|$),?" john

What is this? : "(.+?)(?=,\S|$),?"

For the tokenizer to work you need capture groups.

What we're saying here is

(.+?)      grab everything - this is the capture group
(?=,\S|$)  until you get to a comma followed by a non-whitespace, or the end of the line
,?         if there's a comma at the end of the pattern, eat it

result :

alt text

View solution in original post

ckp123
Path Finder

As simple replace would do this job.

| replace "," with ", " in john

PS : As per my understood on the requirement

0 Karma

jonuwz
Influencer

Example :

| gentimes start=-1 
| eval john="1 something,2 something else,3 something, with a comma,4 wibble"
| table john
| makemv tokenizer="(.+?)(?=,\S|$),?" john

What is this? : "(.+?)(?=,\S|$),?"

For the tokenizer to work you need capture groups.

What we're saying here is

(.+?)      grab everything - this is the capture group
(?=,\S|$)  until you get to a comma followed by a non-whitespace, or the end of the line
,?         if there's a comma at the end of the pattern, eat it

result :

alt text

martin_mueller
SplunkTrust
SplunkTrust

This probably works for you:

tokenizer="([^,]*)(,(\s[^,]*,?)*)?"

The tokenizer first captures a value:

([^,]*)`)

and then gobbles up everything that's not a field:

(,(\s[^,]*,?)*)?

PS: As per jonuwz's answer I may have treated ", " badly 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...