Multivalue delimited field extraction

[bfontneau]

I am extracting fields from tabular data containing headers with entries in props.conf like the following:

EXTRACT-categories = (?i)^(?:[^\t]*\t){24}(?P<%FieldName%>[^\t]+)

It is working as desired, though it may not be the best method. If you know of a better way please advise.

One of the extracted fields contains multivalue data delimited with pipes. For example the data after the 24th tab looks like the below:

option1|option2|option3

Currently this is being extracted as categories=option1|option2|option3 but this results in more categories than exist due to all the possible combinations of options. I'd like each event to be categories=option1 categories=option2 so they are listed as option1 and option2 not option1|option2.

I have tried various attempts including the below from splunkbase but I can't seem to get anything to work.

props.conf

[eventtypename]
REPORT-categories = mv-categories

transforms.conf

[mv-categories]
REX = (?i)^(?:[^\t]*\t){24}(?P<%fieldname%>\w+[^\|]|[^\t])
MV_ADD = true

Can anyone point me towards a better solution?

[bfontneau · Accepted solution]

makemv at search time was working using the below but I as looking for auto extraction.

| makemv delim="|" categories

I was finally able to get it to work using parts of each of your suggestions.

props.conf

[eventtypename]
REPORT-mvalue = categoriesmv

transforms.conf

[categoriesmv]
REGEX = (?i)^(?:[^\t]*\t){24}(?P<'categories>[^\t]+)
MV_ADD = true
FORMAT = categories::$1

fields.conf

[categories]
TOKENIZER = ([^\|]+)\|*

Thanks for your help!

[spammenot66]

is it possible possible to setup the fields.conf / TOKENIZER using SPLUNK Web - without directly modifying fields.conf file?

[bfontneau]

makemv is working using | makemv delim="|" categories but I am looking for auto extraction. I looked at the wiki previously but wasn't able to make it all work until just now.
Thanks!

[dart]

Try this:

[mv-categories]
SOURCE_KEY=categories
REX = (?<category>[^\|]+)\|*
MV_ADD = true

[mbenwell]

try the makemv command at search time
or
you could try the props.conf, transforms.conf example here:
http://wiki.splunk.com/Community:Comma-Separated_Multi-Value_Field_Extraction_In_Single-line_Event