topic Re: Fields Extraction from raw events with regex in Splunk Enterprise

How to Extract Fields from raw events with regex

pchintha — Thu, 07 Apr 2022 15:29:00 GMT

Need to extract fields from the below raw data currently no fields automatically extracted.

Raw Event:
Server: autoparts01, Userid: monika, Alias: autoparts01monika, Return Code: 400, Password Len: 32, Host: ELKSPL3212, Execution ID: autodr1, Directory: C:\windows\system32, Program: C:\windows\Sys64\dllhost.exe, Elapsed Time: 0, Bypass Cache: false, Type: Windows dll - 0, Version: 3.6

Output Sample: need regex and the fields are every separated by (,)

Server: autoparts01 to Server=autoparts01
Userid: monika to Userid=monika

Re: Fields Extraction from raw events with regex

mayurr98 — Thu, 07 Apr 2022 04:16:56 GMT

For SPL try this:

<your base search> | extract kvdelim=":" pairdelim=","

Re: Fields Extraction from raw events with regex

pchintha — Thu, 07 Apr 2022 04:28:56 GMT

@mayurr98 So good thanks for the quick help, this is working for me.

am getting the all fields separated from raw events except from Server

and also need STANZA same thing need to fix this automatic in props.conf and transforms.conf in the app

Re: Fields Extraction from raw events with regex

mayurr98 — Thu, 07 Apr 2022 05:06:19 GMT

Server should be there as well, you can search for that field in all fields.

Transforms.conf [myplaintransform] REGEX=\s([^\:]+)\:\s+([^\,]+) FORMAT=$1::$2 props.conf [sourcetype_name] REPORT-a = myplaintransform

Accept/Like if it works for you.

Re: Fields Extraction from raw events with regex

pchintha — Thu, 07 Apr 2022 04:47:58 GMT

@mayurr98 No i already saw i guess why its not taking up is there is a space before the Server field you can see the sample raw data below

04-07-2022 00:44 Server: bug, Userid: monika, Alias: bugmonika, Return Code: 400, Password Len: 16, Host: lhplc3216, Execution ID: oracle, Directory: /bug/lds/oracle/rdbms/dbh_7347285/dbs Program: bugeco.exe, Elapsed Time: 0, Bypass Cache: false, Type: Unix, Version: 2.0

Re: Fields Extraction from raw events with regex

mayurr98 — Thu, 07 Apr 2022 05:07:39 GMT

I have updated the transform to accumulate the server field:

The raw event that you gave , it should work now.

https://regex101.com/r/dL6JPE/1

Re: Fields Extraction from raw events with regex

pchintha — Fri, 08 Apr 2022 13:24:22 GMT

Hi as you provided this is not working for me, and the regex 101 link which you provided its on working only on that not in Splunk query.

And the transforms.conf and props.conf is also not working.

Re: Fields Extraction from raw events with regex

mayurr98 — Fri, 08 Apr 2022 13:38:01 GMT

could you please give me below details:

1) please share more sample raw events

2) share the props and transforms that you have wrote

3) where did you write the props and transforms?

4) have you restarted splunk instance after updating props and transforms ?

Re: Fields Extraction from raw events with regex

pchintha — Fri, 08 Apr 2022 13:53:12 GMT

1) please share more sample raw events

04-08-2022 06:31 Server: O85XDC7, Userid: PKDPRW8, Alias: o34xda3pkspaw8, Return Code: 400, Password Len: 8, Host: hello1625.winki.com, Execution ID: ccms, Directory: /usr/local/scripts, Program: CoWXatrix.jar, Elapsed Time: 0, Bypass Cache: false, Type: Unix, Version: 2.0

04-08-2022 06:31 Server: O87XDC0, Userid: TOWDHU1, Alias: o73xda3sppklxl, Return Code: 400, Password Len: 25, Host: hello1625.winki.com, Execution ID: ccms, Directory: /usr/local/scripts, Program: CoWXatrix.jar, Elapsed Time: 0, Bypass Cache: false, Type: Unix, Version: 2.0

2) share the props and transforms that you have wrote
We tried in all different ways but no luck

props.conf
[ssc_cloakware]

REPORT-extractions = extractions

transforms.conf

[extractions]

DELIMS = ",", ":"

#SOURCE_KEY = _raw

#REGEX = \s([^\:]+)\:\s+([^\,]+)

#FORMAT = $1::$2

#MV_ADD = true

#REPEAT_MATCH = true

#CLEAN_KEYS = false

3) where did you write the props and transforms?

In Search Head Deployer

4) have you restarted splunk instance after updating props and transforms ?

Yes, we restarted if any changes done we do always restart and bundle push as well

Actually this is we are taking sample events from Prod and uploading in UAT there we are trying. And index we created in indexer for this custom app

Re: Fields Extraction from raw events with regex

pchintha — Fri, 08 Apr 2022 14:44:53 GMT

Any update by chance

Re: Fields Extraction from raw events with regex

mayurr98 — Fri, 08 Apr 2022 14:58:03 GMT

Hi for the search to work you would have to write this :

<your base search> | extract kvdelim=":" pairdelim="," | rex "Server\s*:\s*(?<Server>[^\,]+)"

props.conf and transforms.conf is best to put on the heavy forwarder if you have one or the indexing layer.

The regex that I provided is for transforms only and it works well for all the events that you have given.

https://regex101.com/r/0rdToo/1

use below configuration on HF or Indexers.

props.conf [ssc_cloakware] REPORT-extractions = field_extractions transforms.conf [field_extractions] REGEX = \s([^\:]+)\:\s+([^\,]+) FORMAT = $1::$2

Restart the instance after editing the configuration.

I am gonna test this configuration on my lab instance. meanwhile you do the same.

Re: Fields Extraction from raw events with regex

pchintha — Mon, 11 Apr 2022 07:33:22 GMT

Hi Sorry for late responding, as you said i did same its working in the props.conf and transforms.conf but except the Server field in the logs, can you please share the extract field for the Server for transforms.conf

Re: Fields Extraction from raw events with regex

pchintha — Tue, 12 Apr 2022 03:04:39 GMT

Hi, please update for this

Re: Fields Extraction from raw events with regex

mayurr98 — Tue, 12 Apr 2022 05:04:45 GMT

Put below in props.conf

props.conf [ssc_cloakware] REPORT-extractions = field_extractions EXTRACT-server = Server\s*:\s*(?<Server>[^\,]+)

This is search time field extraction so make sure you write this regex in SH.

OR simply go to search head:

Settings » Fields » Field Extractions » Add new

Destination App: <Choose appropriate app> Name: Server Apply to: Sourcetype: <sourcetype_name> Extraction/Transform: Server\s*:\s*(?<Server>[^\,]+)

Please upvote/accept to close this question if it works for you.

Re: Fields Extraction from raw events with regex

youngsuh — Fri, 28 Oct 2022 20:09:28 GMT

@mayurr98

Here is my search that I want to put into props & transform.

index=nsips_horizon sourcetype="vmware:uag:esmanager" | rex "^(?:[^ \n]* ){3}(?P<UAG_hostname>[a-z]+\d+)\s+(?P<app_name>[^:]+)[^:\n]*:\s+\[(?P<thread_id>\S+?)\](?P<log_level>[^ ]+)\s+(?P<file_name>[^\[]+)\[(?P<function_name>[^:]+):\s+(?P<line_num>[^\]]+)[^\[\n]*\[(?P<client_IP>[^\]]+|.?)\]\[(?P<username>.+|.?)\]\[(?P<session_type>.+|.?)\]\[(?P<session_id>[^\]]+)[^\]\n]*\]\s+\-\s+(?P<message>.+)"

here is sample log:

Sep 9 05:36:55 UAG Name UAG-ESMANAGER: [Curator-QueueBuilder-0]INFO utils.SyslogManager[start: 355][][][][] - Edge Service Manager : started Sep 9 05:36:54 UAG Name UAG-ESMANAGER: [Curator-QueueBuilder-0]INFO utils.SyslogManager[stop: 1071][][][][] - Edge Service Manager : stopped

based on what I read the props.conf

[vmware:uag:esmanager] REPORT-esmanager = esmanager

Transform.conf

[esmanager] REGEX = ^(?:[^ \n]* ){3}(?P<UAG_hostname>[a-z]+\d+)\s+(?P<app_name>[^:]+)[^:\n]*:\s+\[(?P<thread_id>\S+?)\](?P<log_level>[^ ]+)\s+(?P<file_name>[^\[]+)\[(?P<function_name>[^:]+):\s+(?P<line_num>[^\]]+)[^\[\n]*\[(?P<client_IP>[^\]]+|.?)\]\[(?P<username>.+|.?)\]\[(?P<session_type>.+|.?)\]\[(?P<session_id>[^\]]+)[^\]\n]*\]\s+\-\s+(?P<message>.+) FORMAT = $1::$2

Right? What's $1::$2 doing?