Splunk Search

Multivalue delimited field extraction

Explorer

I am extracting fields from tabular data containing headers with entries in props.conf like the following:

EXTRACT-categories = (?i)^(?:[^\t]*\t){24}(?P<%FieldName%>[^\t]+)

It is working as desired, though it may not be the best method. If you know of a better way please advise.

One of the extracted fields contains multivalue data delimited with pipes. For example the data after the 24th tab looks like the below:

option1|option2|option3

Currently this is being extracted as categories=option1|option2|option3 but this results in more categories than exist due to all the possible combinations of options. I'd like each event to be categories=option1 categories=option2 so they are listed as option1 and option2 not option1|option2.

I have tried various attempts including the below from splunkbase but I can't seem to get anything to work.

props.conf

[eventtypename]
REPORT-categories = mv-categories

transforms.conf

[mv-categories]
REX = (?i)^(?:[^\t]*\t){24}(?P<%fieldname%>\w+[^\|]|[^\t])
MV_ADD = true

Can anyone point me towards a better solution?

1 Solution

Explorer

makemv at search time was working using the below but I as looking for auto extraction.

| makemv delim="|" categories

I was finally able to get it to work using parts of each of your suggestions.

props.conf

[eventtypename]
REPORT-mvalue = categoriesmv

transforms.conf

[categoriesmv]
REGEX = (?i)^(?:[^\t]*\t){24}(?P<'categories>[^\t]+)
MV_ADD = true
FORMAT = categories::$1

fields.conf

[categories]
TOKENIZER = ([^\|]+)\|*

Thanks for your help!

View solution in original post

Explorer

makemv at search time was working using the below but I as looking for auto extraction.

| makemv delim="|" categories

I was finally able to get it to work using parts of each of your suggestions.

props.conf

[eventtypename]
REPORT-mvalue = categoriesmv

transforms.conf

[categoriesmv]
REGEX = (?i)^(?:[^\t]*\t){24}(?P<'categories>[^\t]+)
MV_ADD = true
FORMAT = categories::$1

fields.conf

[categories]
TOKENIZER = ([^\|]+)\|*

Thanks for your help!

View solution in original post

Contributor

is it possible possible to setup the fields.conf / TOKENIZER using SPLUNK Web - without directly modifying fields.conf file?

0 Karma

Explorer

makemv is working using | makemv delim="|" categories but I am looking for auto extraction. I looked at the wiki previously but wasn't able to make it all work until just now.
Thanks!

0 Karma

Splunk Employee
Splunk Employee

Try this:

[mv-categories]
SOURCE_KEY=categories
REX = (?<category>[^\|]+)\|*
MV_ADD = true

Communicator

try the makemv command at search time
or
you could try the props.conf, transforms.conf example here:
http://wiki.splunk.com/Community:Comma-Separated_Multi-Value_Field_Extraction_In_Single-line_Event

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!