How to use makemv with tokenizers while keeping non-matching events?

[Gunnar]

Hi,

I have events similar to this example:

1) date1, id1, misc

2) date2, id2, misc

3) date3, , misc

4) date4, id3 and id4, misc

The ids in 4) should be split into two separate lines.  The result should look like this:

1) date1, id1, misc

2) date2, id2, misc

3) date3, , misc

4) date4, id3 , misc

5) date4, id4, misc

But when using makemv with tokenizer lines which do not match, the tokenizers are skipped in the result, e.g.:

... | makemv tokenizer="(id\d)" ID | mvexpand ID | ...

Results in:

1) date1, id1, misc

2) date2, id2, misc

3) date4, id3 , misc

4) date4, id4, misc

How can I keep the non-matching lines? Is there a way to only use makemv where it is necessary?

My workaround at the moment is to append a second search that looks for events with an empty ID and adds those events again after the makemv, e,g,:

first search | ...  |  makemv tokenizer="(id\d)" ID | mvexpand ID | append [ first search again | ... | search NOT ID="*" | ... ] | ...

But searching twice can't be an optimal solution.

Thank you,

Gunnar

[bowesmana · Accepted solution]

Not sure of the exact format of your data, but look at this example using split/mvexpand where fillnull is used to ensure the missing id gets preserved in the mvexpand

| makeresults count=4
| eval t=1 
| accum t
| eval _time=now() - (random() % 86400)
| eval id=case(t=1,"id1",t=2,"id2",t=3,null,t=4,"id3 id4")
| eval misc=random()
| fields - t
| eval id=split(id," ")
| fillnull value="" id
| mvexpand id

It's likely that fillnull will solve your problem in your example makemv case also - set the value to whatever makes sense in your context.

Hope this helps

[Gunnar]

Hi bowesmana,

great idea - this put me on the right track.

What I do now is to use an eval before the makemv that creates an ID value if empty, that matches my tokenizer but cannot occur in the real data.

After the the mvexpand I use another eval to remove this dummy-ID again.

Seems to work fine. And two evals should be better than two searches.

Thank you very much and kind regards,

Gunnar