Splunk Search

How to configure multivalue field extraction?

ninisimonishvil
Path Finder

Hello,

I cannot configure multivalue field extraction. I have a following event. the last 4 lines Time Stamp and Message shall be extracted as separate values togather with value following the FROM: section on the first line. I used props.conf. and Transforms.conf (MV_ADD) however no use.

From: "Rnvr" <r@cou.ge>
Subject:  Control Center System Event
Date: Fri, 15 Jun 2018 18:14:07 +0400
Message-ID: <CE4FFC373CC843A4AFE30E855AE5839F@court.gov.ge>
Return-Path: r@cou.ge
Received: from mail.cou.ge (LHLO mail.cou.ge) (192.168.222.10) by
 mail.cou.ge with LMTP; Fri, 15 Jun 2018 18:13:58 +0400 (GET)
Received: from localhost (localhost [])
    by mail.court.ge (Postfix) with ESMTP id 75C1519E007B
    for <asupport@cou.ge>; Fri, 15 Jun 2018 18:13:58 +0400 (+04)
[2018-Jun-15 06:04:42 PM (GET)] Hardware event occurred (The controller write policy has been changed to Write Back.) on server 

[2018-Jun-15 06:04:43 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server 

[2018-Jun-15 06:04:44 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server 

[2018-Jun-15 06:13:16 PM (GET)] Digital input 'Digital Input 1' deactivated.
0 Karma
1 Solution

FrankVl
Ultra Champion

Looks like the extraction of the first message doesn't stop at the end of the line and just captures the whole last 4 lines as 1 message entry. You may need to adjust your regex slightly to ensure it only captures a single message at a time.

For example: (?<=\[\d{4}\-\w{3}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\w{2}\s\(\w{3}\)\]\s)(?P<message>\w[^\r\n]+)

View solution in original post

0 Karma

ninisimonishvil
Path Finder

thanks it worked for me.

0 Karma

FrankVl
Ultra Champion

Looks like the extraction of the first message doesn't stop at the end of the line and just captures the whole last 4 lines as 1 message entry. You may need to adjust your regex slightly to ensure it only captures a single message at a time.

For example: (?<=\[\d{4}\-\w{3}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\w{2}\s\(\w{3}\)\]\s)(?P<message>\w[^\r\n]+)

0 Karma

Ayn
Legend

Please include relevant sections from props.conf and transforms.conf, and elaborate on how things are not working. Are you getting any relevant fields at all?

0 Karma

ninisimonishvil
Path Finder

Hello, yes I'm getting results but it is not extracting all the values in 1 event. here are my conf. file sections.

Transforms:

[date]
REGEX= (?<=\[)(?P<date>\d{4}\-\w{3}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\w{2})(?=\s\(\w{3})
MV_ADD = true


[message]
REGEX=(?<=\[\d{4}\-\w{3}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\w{2}\s\(\w{3}\)\]\s)(?P<message>\w.*\X)
MV_ADD = true


Props:

[mysourcetype]
DATETIME_CONFIG = 
LINE_BREAKER = ()\wrom:\s"
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
REPORT-date = date
REPORT-message= message
0 Karma

FrankVl
Ultra Champion

Well, the regular expressions seem to be OK. Have you deployed this on the Search Head(s)?

The transforms namings are a bit short and generic, you might want to check with btool if those are unique.

0 Karma

ninisimonishvil
Path Finder

Frank yes it is a SH. as for the naming yes they are unique, the problem is only with the message extraction.

in message field extracted results shows exactly like this - Hardware event occurred (The controller write policy has been changed to Write Back.) on server RUSTAVI-VIDEO. [2018-Jun-15 06:04:43 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server RUSTAVI-VIDEO. [2018-Jun-15 06:04:44 PM (GET)] Hardware event occurred (The virtual disk cache policy has changed.) on server RUSTAVI-VIDEO. [2018-Jun-15 06:13:16 PM (GET)] Digital input 'Digital Input 1' deactivated.

even though my regex says to ignore the beginning (date) it still appears to be in results and moreover, it does not separate messages as a separate values, it shows it as a one big string.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...