topic Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index? in Knowledge Management

How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

gesman — Sun, 26 Apr 2015 17:22:40 GMT

I want to write transactions with full list of pages accessed into summary index in this manner:

... | transaction ip maxpause=15m mvlist=page | fields _time, ip, page | fields - _raw | collect index=my_summary

But the resulting summary index contains the 'page' field in flattened format, no longer multivalue.

Is this documented behavior? Can I force my summary index to keep fields in multivalue format, or do I need to do makemv every time I want to search my summary index?

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

stephane_cyrill — Sun, 26 Apr 2015 17:38:33 GMT

Hi, Try using mvlist=t

mvlist =< bool > |
Description: Flag controlling
whether the multivalued fields of
the transaction are (mvlist = t) a
list of the original events ordered
in arrival order or (mvlist = f ) a
set of unique field values ordered
lexigraphically . If a comma / space
delimited list of fields is provided
only those fields are rendered as
lists . Defaults to f .

http://docs.splunk.com/Documentation/Splunk/latest/
SearchReference/Transaction

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

gesman — Sun, 26 Apr 2015 17:53:03 GMT

Thanks for your effort, but my question is about multivalue field losing it's format when transferred into summary index, and not about the way transaction creates these fields.

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

martin_mueller — Sun, 26 Apr 2015 21:43:07 GMT

One way to tackle this could be to un-mv your field before collecting, adding a delimiter between the values. Using that delimiter you could then set up field extractions with MV_ADD to avoid doing the mv dance in the search itself.

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

gesman — Sun, 26 Apr 2015 21:57:02 GMT

Thanks for the tip though - good to be aware of alternatives.

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

martin_mueller — Sun, 26 Apr 2015 22:47:07 GMT

Well, the idea is to do complicated stuff once - when collecting - and do simple stuff many times - when searching.

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

fdi01 — Mon, 27 Apr 2015 20:33:25 GMT

try with table command.

...|table _time   ip page | fields - _raw | collect index=my_summar

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

gesman — Tue, 28 Apr 2015 02:07:27 GMT

Same result. Multivalues flattened to single string.

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

gesman — Tue, 28 Apr 2015 15:21:48 GMT

Hi Martin,
with MV_ADD approach what else do i need to do to make it happen automagically?
I've looked into DELIM param but still not sure if it applies to my case, or whether I need any other params to customize?

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

gesman — Wed, 29 Apr 2015 02:06:50 GMT

One of the working solution is to add this to ./etc/system/local/fields.conf:

[ips]
TOKENIZER = ([^\|]+)

[uas]
TOKENIZER = ([^\|]+)

[usernames]
TOKENIZER = ([^\|]+)

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

gesman — Fri, 01 May 2015 19:37:57 GMT

To note: there is actually benefit of having multivalues flattened and separated by some character.
"Flattened" values (say 'usernames') is searchable via index=logs usernames=*johnsmith* | ... query vs. multivalues are not.

So in above case if I'd need to find only events where one of the username is (or contains) 'johnsmith' - that would work nicely and reduce number of events before pipe.

If usernames would be stored in multivalued format - we'd need to use slower logic to either flatten usernames first or use functions like mvfilter to search everything.

Re: How do I prevent the collect command from flattening multivalue fields when writing into a summary index?

martin_mueller — Fri, 01 May 2015 21:06:16 GMT

Nitpicking point: If you're searching for johnsmith as a username value, searching for username=*johnsmith* will lead to tears if you have a johnsmithy...

Usually, doing field=value in a search will be translated to "value is in field" if field is a multivalue field, so make sure it really doesn't work for you. Sample search:

| stats count | eval foo = "a b c" | makemv foo | search foo="b"

That'll keep the one row, because foo contains a value of b.