How to Extract Fields from raw events with regex

[pchintha]

Need to extract fields from the below raw data currently no fields automatically extracted.

Raw Event:
Server: autoparts01, Userid: monika, Alias: autoparts01monika, Return Code: 400, Password Len: 32, Host: ELKSPL3212, Execution ID: autodr1, Directory: C:\windows\system32, Program: C:\windows\Sys64\dllhost.exe, Elapsed Time: 0, Bypass Cache: false, Type: Windows dll - 0, Version: 3.6

Output Sample: need regex and the fields are every separated by (,)

Server: autoparts01 to Server=autoparts01
Userid: monika to Userid=monika

[mayurr98]

For SPL try this:

<your base search> 
| extract kvdelim=":" pairdelim=","

[pchintha]

@mayurr98 So good thanks for the quick help, this is working for me.

am getting the all fields separated from raw events except from Server

and also need STANZA same thing need to fix this automatic in props.conf and transforms.conf in the app

[mayurr98]

Server should be there as well, you can search for that field in all fields.

Transforms.conf 

[myplaintransform]
REGEX=\s([^\:]+)\:\s+([^\,]+)
FORMAT=$1::$2

props.conf

[sourcetype_name]
REPORT-a = myplaintransform

Accept/Like if it works for you.

[youngsuh]

@mayurr98 

Here is my search that I want to put into props & transform.

index=nsips_horizon sourcetype="vmware:uag:esmanager" 
| rex "^(?:[^ \n]* ){3}(?P<UAG_hostname>[a-z]+\d+)\s+(?P<app_name>[^:]+)[^:\n]*:\s+\[(?P<thread_id>\S+?)\](?P<log_level>[^ ]+)\s+(?P<file_name>[^\[]+)\[(?P<function_name>[^:]+):\s+(?P<line_num>[^\]]+)[^\[\n]*\[(?P<client_IP>[^\]]+|.?)\]\[(?P<username>.+|.?)\]\[(?P<session_type>.+|.?)\]\[(?P<session_id>[^\]]+)[^\]\n]*\]\s+\-\s+(?P<message>.+)"

here is sample log:

Sep 9 05:36:55 UAG Name UAG-ESMANAGER: [Curator-QueueBuilder-0]INFO utils.SyslogManager[start: 355][][][][] - Edge Service Manager : started
Sep 9 05:36:54 UAG Name UAG-ESMANAGER: [Curator-QueueBuilder-0]INFO utils.SyslogManager[stop: 1071][][][][] - Edge Service Manager : stopped

based on what I read the props.conf 

[vmware:uag:esmanager]
REPORT-esmanager = esmanager

Transform.conf

[esmanager]
REGEX = ^(?:[^ \n]* ){3}(?P<UAG_hostname>[a-z]+\d+)\s+(?P<app_name>[^:]+)[^:\n]*:\s+\[(?P<thread_id>\S+?)\](?P<log_level>[^ ]+)\s+(?P<file_name>[^\[]+)\[(?P<function_name>[^:]+):\s+(?P<line_num>[^\]]+)[^\[\n]*\[(?P<client_IP>[^\]]+|.?)\]\[(?P<username>.+|.?)\]\[(?P<session_type>.+|.?)\]\[(?P<session_id>[^\]]+)[^\]\n]*\]\s+\-\s+(?P<message>.+)
FORMAT = $1::$2

Right?  What's $1::$2 doing?

[pchintha]

@mayurr98 No i already saw i guess why its not taking up is there is a space before the Server field you can see the sample raw data below

04-07-2022 00:44 Server: bug, Userid: monika, Alias: bugmonika, Return Code: 400, Password Len: 16, Host: lhplc3216, Execution ID: oracle, Directory: /bug/lds/oracle/rdbms/dbh_7347285/dbs Program: bugeco.exe, Elapsed Time: 0, Bypass Cache: false, Type: Unix, Version: 2.0

[mayurr98]

I have updated the transform to accumulate the server field:

The raw event that you gave , it should work now.

https://regex101.com/r/dL6JPE/1

[pchintha]

Hi as you provided this is not working for me, and the regex 101 link which you provided its on working only on that not in Splunk query.

And the transforms.conf and props.conf is also not working.

[mayurr98]

could you please give me below details:

1) please share more sample raw events

2) share the props and transforms that you have wrote

3) where did you write the props and transforms?

4) have you restarted splunk instance after updating props and transforms ?

[pchintha]

1) please share more sample raw events

04-08-2022 06:31 Server: O85XDC7, Userid: PKDPRW8, Alias: o34xda3pkspaw8, Return Code: 400, Password Len: 8, Host: hello1625.winki.com, Execution ID: ccms, Directory: /usr/local/scripts, Program: CoWXatrix.jar, Elapsed Time: 0, Bypass Cache: false, Type: Unix, Version: 2.0

04-08-2022 06:31 Server: O87XDC0, Userid: TOWDHU1, Alias: o73xda3sppklxl, Return Code: 400, Password Len: 25, Host: hello1625.winki.com, Execution ID: ccms, Directory: /usr/local/scripts, Program: CoWXatrix.jar, Elapsed Time: 0, Bypass Cache: false, Type: Unix, Version: 2.0

04-08-2022 06:31 Server: O85XDC7, Userid: PKDPRW8, Alias: o34xda3pkspaw8, Return Code: 400, Password Len: 8, Host: hello1625.winki.com, Execution ID: ccms, Directory: /usr/local/scripts, Program: CoWXatrix.jar, Elapsed Time: 0, Bypass Cache: false, Type: Unix, Version: 2.0

2) share the props and transforms that you have wrote
We tried in all different ways but no luck

props.conf
[ssc_cloakware]

REPORT-extractions = extractions

transforms.conf

[extractions]

DELIMS = ",", ":"

#SOURCE_KEY = _raw

#REGEX = \s([^\:]+)\:\s+([^\,]+)

#FORMAT = $1::$2

#MV_ADD = true

#REPEAT_MATCH = true

#CLEAN_KEYS = false

3) where did you write the props and transforms?

In Search Head Deployer

4) have you restarted splunk instance after updating props and transforms ?

Yes, we restarted if any changes done we do always restart and bundle push as well

Actually this is we are taking sample events from Prod and uploading in UAT there we are trying. And index we created in indexer for this custom app

[mayurr98]

Hi for the search to work you would have to write this :

<your base search>
| extract kvdelim=":" pairdelim="," 
| rex "Server\s*:\s*(?<Server>[^\,]+)"

props.conf and transforms.conf is best to put on the heavy forwarder if you have one or the indexing layer.

The regex that I provided is for transforms only and it works well for all the events that you have given.

https://regex101.com/r/0rdToo/1

use below configuration on HF or Indexers.

props.conf
[ssc_cloakware]
REPORT-extractions = field_extractions
 
transforms.conf
[field_extractions]
REGEX = \s([^\:]+)\:\s+([^\,]+)
FORMAT = $1::$2

Restart the instance after editing the configuration. 

I am gonna test this configuration on my lab instance. meanwhile you do the same.

[pchintha]

Hi Sorry for late responding, as you said i did same its working in the props.conf and transforms.conf but except the Server field in the logs, can you please share the extract field for the Server for transforms.conf

[pchintha]

Hi, please update for this

[pchintha]

Any update by chance

[mayurr98 · Accepted solution]

Put below in props.conf

props.conf
[ssc_cloakware]
REPORT-extractions = field_extractions
EXTRACT-server = Server\s*:\s*(?<Server>[^\,]+)

This is search time field extraction so make sure you write this regex in SH.

OR simply go to search head:

Settings » Fields » Field Extractions » Add new

Destination App: <Choose appropriate app>
Name: Server
Apply to: Sourcetype: <sourcetype_name>
Extraction/Transform:  Server\s*:\s*(?<Server>[^\,]+)

Please upvote/accept to close this question if it works for you.