topic Re: makemv with tokenizer while keeping non-matching events? in Splunk Search

How to use makemv with tokenizers while keeping non-matching events?

Gunnar — Wed, 01 Jul 2020 18:17:45 GMT

Hi,

I have events similar to this example:

1) date1, id1, misc

2) date2, id2, misc

3) date3, , misc

4) date4, id3 and id4, misc

The ids in 4) should be split into two separate lines. The result should look like this:

1) date1, id1, misc

2) date2, id2, misc

3) date3, , misc

4) date4, id3 , misc

5) date4, id4, misc

But when using makemv with tokenizer lines which do not match, the tokenizers are skipped in the result, e.g.:

... | makemv tokenizer="(id\d)" ID | mvexpand ID | ...

Results in:

1) date1, id1, misc

2) date2, id2, misc

3) date4, id3 , misc

4) date4, id4, misc

How can I keep the non-matching lines? Is there a way to only use makemv where it is necessary?

My workaround at the moment is to append a second search that looks for events with an empty ID and adds those events again after the makemv, e,g,:

first search | ... | makemv tokenizer="(id\d)" ID | mvexpand ID | append [ first search again | ... | search NOT ID="*" | ... ] | ...

But searching twice can't be an optimal solution.

Thank you,

Gunnar

Re: makemv with tokenizer while keeping non-matching events?

bowesmana — Tue, 30 Jun 2020 22:33:01 GMT

Not sure of the exact format of your data, but look at this example using split/mvexpand where fillnull is used to ensure the missing id gets preserved in the mvexpand

It's likely that fillnull will solve your problem in your example makemv case also - set the value to whatever makes sense in your context.

Hope this helps

Re: makemv with tokenizer while keeping non-matching events?

Gunnar — Wed, 01 Jul 2020 07:34:47 GMT

Hi bowesmana,

great idea - this put me on the right track.

What I do now is to use an eval before the makemv that creates an ID value if empty, that matches my tokenizer but cannot occur in the real data.

After the the mvexpand I use another eval to remove this dummy-ID again.

Seems to work fine. And two evals should be better than two searches.

Thank you very much and kind regards,

Gunnar