topic Multi-value Field extraction in Splunk Search

Multi-value Field extraction

tallasky — Mon, 28 Sep 2020 15:21:20 GMT

Hello,

I would like to create a multi-value field for my data, how can i do that?

here's a sample of my data (Starts at QAM)

event1 = QAM 32209 Prog 238 Path PATH_MW

event2 = QAM 23001 Prog 25 Path PATH_MG PATH_VERTE

event3 = QAM 46201 Prog 180 Path PATH_MD_1 PATH_JAUNE

The field i am trying to extract is Path but as you can see in the sample above, path can have a single value in the case of event1(PATH_MW) or multiple values in the case of event2 (PATH _MG PATH _VERTE) and event3 (PATH _MD _1 PATH _JAUNE)

This regular expression finds them but considers multiple values as one (?i) Path (?P[a-z_]+.[a-z_])

Here the values for event 2 and 3 are considered as one even if we have two values, what can i do so that it stores the two values in the path field for the same event but see them as two different values.

Thank you in advance.

Re: Multi-value Field extraction

chris — Mon, 28 Sep 2020 15:21:23 GMT

the rex command has a max_matches option so you could try something like:

| rex "Path (?.*)" | rex field=temporary_field max_match=10 "(?[a-z_]+)"

Update

To do this automatically you will have to use the regular expression in a configs file called props.conf & transforms.conf. You can place those files in $SPLUNK_HOME/etc/system/local to start.
props.conf

[my_sourcetype]
TRANSFORMS-tempfield = tempfield
TRANSFORMS-mv_field = mv_field

transforms.conf

[tempfield]
REGEX=Path (.*)
FORMAT=temporary_field::$1

[mv_field]
SOURCE_KEY=temporary_field
REGEX=([a-z_]+)
FORMAT=mv_field::$1
MV_ADD=true

There is more information here --> documentation. I also suggest that you read about Technology Add ons the ESS and PCI Apps/Suites from Splunk are built on this approach. The idea is to put all the configuration that is necessary to parse & extract fields for a technology into an app that can be used by all the splunk users/apps at your site. Does that make sense?

Re: Multi-value Field extraction

tallasky — Mon, 25 Nov 2013 16:13:10 GMT

Hi Chris, thanks for getting back to me:

Now, what i'm trying to do is use the extract field option so this is done automatically with new data coming in everyday, your solution with rex implies that i have to use the search bar? i don't want that.

Thank you for your help.

Re: Multi-value Field extraction

tallasky — Mon, 25 Nov 2013 16:13:31 GMT

Hi Chris, thanks for getting back to me:

Thank you for your help.

Re: Multi-value Field extraction

chris — Mon, 25 Nov 2013 16:54:02 GMT

Hi, I updated the answer let me know if this works for you

Re: Multi-value Field extraction

tallasky — Mon, 28 Sep 2020 15:21:42 GMT

Hello Chris, thanks again,

It's tricky to acces those files in a professional environement with many levels of security and different teams using the same Splunk, but i found a way around it:
I use this regular expression (?i) Path (?P[a-z_-]+.[a-z_-]) to get my path values wheter it One(event1) or many(event2,event3) and i use the makemv function to separate them before my research by using the space between them as delimiter

Sourcetype = ...| makemv delim = " " path

Thanks you.

Re: Multi-value Field extraction

chris — Tue, 26 Nov 2013 08:45:05 GMT

Hi, I'm glad you found a solution