Multi value field extraction props.conf transforms.conf

[DonaldvdHoogenb]

Hello fellow Splunker,

I have a question about my props.conf and transforms.conf.
I want to extract a multi valued field for the messages in which are displayed in the following .txt file.

#######################################
System: System_02

Message
---------------                                                                                                                 RHSA-2016:0001  Important: security update - 1         12/23/16                                                  RHSA-2016:0002  Important: security update - 2         12/24/16                                                  RHSA-2016:0003  Important: bug fix update - 1          12/25/16
#######################################

I want to extract the messages as a multi-valued field. Therefore I have written the following props.conf and transforms.conf. (by looking at similar questions asked on splunk answers.)

props.conf

[mymessagetest]
MUST_BREAK_AFTER = #####
TIME_FORMAT = %m/%d/%y
category = Custom
disabled = false
pulldown_type = true
REPORT-mv_sec = mv_sec

transforms.conf

 [mv_sec]               
 REGEX = (?<mv_sec>RHSA-[\d\:]+.+)
 MV_ADD = true

However, these .conf files are not extracting the messages as a multi-value field, but as one field.
Does anyone has an idea why this is happening and how I can extract a mv-field?
To be clear, the output is now one field and looks like this:

 RHSA-2016:0001  Important: security update - 1  12/23/16 RHSA-2016:0002  Important: security update - 2  12/24/16  RHSA-2016:0003  Important: bug fix update - 1  12/25/16

Thank you very much for the help!

[arkadyz1 · Accepted solution]

Try adding REPEAT_MATCH=true to your mv_sec stanza, like this:

[mv_sec]
REGEX = RHSA-(?<mv_sec>[^\/]+\/\S+)
MV_ADD = true
REPEAT_MATCH = true

Edit: Just noticed that you are not matching the event as a multi-line. Your regex must start with (?m) to indicate a multi-line event. REGEX = (?m)RHSA-(?<mv_sec>[^\/]+\/\S+)

[DonaldvdHoogenb]

This works like a charm! Thank you very much!!! 🙂
I didnt know you should add that (?m) in the REGEX, but now I do.
Have a wonderful new year 🙂

[somesoni2]

Well in that case you don't need separate props.conf. Lets try with following regex in transforms.conf (restart Splunk after changing)

transforms.conf

[mv_sec]
REGEX = RHSA-(?<mv_sec>[^\/]+\/\S+)
MV_ADD = true

Assuming you want output like this
mv_sec
2016:0001 Important: security update - 1 12/23/16
2016:0002 Important: security update - 2 12/24/16
2016:0003 Important: bug fix update - 1 12/25/16

[somesoni2]

You want all RHSA message in one event and then create a new field mv_sec which is multivalued field with different messages? (Guessing first part is already working)

[DonaldvdHoogenb]

Correct, but I want to do it at index time, and not by using the makemv/mvexpand commands. Do you have any idea how I can achieve this?

[somesoni2]

The field extractions can be done at 3 stages, index times, search time [both are saved in props/transforms) and in-line in search. You don't want the third one, then out of first two, search time field extractions are recommended as they don't slow down indexing and don't take additional disk space.

The REPORT-mv_sec that you're using is for search time field extractions only and should be available on the Search Head servers. So, below part of your sourcetype definition should be in props.conf on Indexer/heavy forwarder. (assuming event breaking and timestamp recognition is working)
props.conf

[mymessagetest]
 MUST_BREAK_AFTER = #####
 TIME_FORMAT = %m/%d/%y
 category = Custom
 disabled = false
 pulldown_type = true

And the multivalued field extraction should be in props.conf/transforms.conf on Search Head

props.conf

[mymessagetest]
 REPORT-mv_sec = mv_sec

transforms.conf

  [mv_sec]               
  REGEX = (?<mv_sec>RHSA-[\d\:]+.+)
  MV_ADD = true

[DonaldvdHoogenb]

Thank you.

I still have a question though.
I tested this on my local Splunk (which is a Search Head and Indexer). Then I do not need the seperate props.conf do I? Why is it not working the way I used it?

Thank you for answering so fast 🙂