How to configure multivalue field extraction?

[ninisimonishvil]

Hello,

I cannot configure multivalue field extraction. I have a following event. the last 4 lines Time Stamp and Message shall be extracted as separate values togather with value following the FROM: section on the first line. I used props.conf. and Transforms.conf (MV_ADD) however no use.

From: "Rnvr" <r@cou.ge>
Subject:  Control Center System Event
Date: Fri, 15 Jun 2018 18:14:07 +0400
Message-ID: <CE4FFC373CC843A4AFE30E855AE5839F@court.gov.ge>
Return-Path: r@cou.ge
Received: from mail.cou.ge (LHLO mail.cou.ge) (192.168.222.10) by
 mail.cou.ge with LMTP; Fri, 15 Jun 2018 18:13:58 +0400 (GET)
Received: from localhost (localhost [])
    by mail.court.ge (Postfix) with ESMTP id 75C1519E007B
    for <asupport@cou.ge>; Fri, 15 Jun 2018 18:13:58 +0400 (+04)
[2018-Jun-15 06:04:42 PM (GET)] Hardware event occurred (The controller write policy has been changed to Write Back.) on server 

[2018-Jun-15 06:04:43 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server 

[2018-Jun-15 06:04:44 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server 

[2018-Jun-15 06:13:16 PM (GET)] Digital input 'Digital Input 1' deactivated.

[ninisimonishvil]

thanks it worked for me.

[FrankVl · Accepted solution]

Looks like the extraction of the first message doesn't stop at the end of the line and just captures the whole last 4 lines as 1 message entry. You may need to adjust your regex slightly to ensure it only captures a single message at a time.

For example: (?<=\[\d{4}\-\w{3}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\w{2}\s\(\w{3}\)\]\s)(?P<message>\w[^\r\n]+)

[Ayn]

Please include relevant sections from props.conf and transforms.conf, and elaborate on how things are not working. Are you getting any relevant fields at all?

[ninisimonishvil]

Hello, yes I'm getting results but it is not extracting all the values in 1 event. here are my conf. file sections.

Transforms:

[date]
REGEX= (?<=\[)(?P<date>\d{4}\-\w{3}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\w{2})(?=\s\(\w{3})
MV_ADD = true


[message]
REGEX=(?<=\[\d{4}\-\w{3}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\w{2}\s\(\w{3}\)\]\s)(?P<message>\w.*\X)
MV_ADD = true


Props:

[mysourcetype]
DATETIME_CONFIG = 
LINE_BREAKER = ()\wrom:\s"
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
REPORT-date = date
REPORT-message= message

[FrankVl]

Well, the regular expressions seem to be OK. Have you deployed this on the Search Head(s)?

The transforms namings are a bit short and generic, you might want to check with btool if those are unique.

[ninisimonishvil]

Frank yes it is a SH. as for the naming yes they are unique, the problem is only with the message extraction.

in message field extracted results shows exactly like this - Hardware event occurred (The controller write policy has been changed to Write Back.) on server RUSTAVI-VIDEO. [2018-Jun-15 06:04:43 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server RUSTAVI-VIDEO. [2018-Jun-15 06:04:44 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server RUSTAVI-VIDEO. [2018-Jun-15 06:13:16 PM (GET)] Digital input 'Digital Input 1' deactivated.

even though my regex says to ignore the beginning (date) it still appears to be in results and moreover, it does not separate messages as a separate values, it shows it as a one big string.