topic Re: regex field extraction not from _raw in Splunk Search

regex field extraction not from _raw

tamir — Mon, 18 Mar 2024 14:02:42 GMT

hey guys
did someone ever happed to come through this problem. I'm using Splunk Cloud
I'm trying to extract a new field using regex but the data are under the source filed

| rex field=source "Snowflake\/(?<folder>[^\/]+)"

this is the regex I'm using when i use it in the search it works perfect. but the main goal is to save this search as a permanent field.
i know that the the field extraction draw from the "_raw" there is an option to direct the Cloud to pull from the source and save it a permanent field.

Re: regex field extraction not from _raw

gcusello — Tue, 12 Mar 2024 13:19:12 GMT

Hi @tamir,

you have to create a new field using the following syntax:

Snowflake\/(?<folder>[^\/]+) in source

in few words you have to add "in" and the firld to use for the extraction.

ciao.

Giuseppe

Re: regex field extraction not from _raw

richgalloway — Tue, 12 Mar 2024 13:22:03 GMT

If the rex command works perfectly then you should have a field called "folder" with the extracted data in it. Is that what is happening? If not, please describe how the rex command is not acting as expected. Note that the "folder" field will be present only within the query that extracted it. If you need the field to be available to all queries then it will have to be extracted at index-time using a transform.

Re: regex field extraction not from _raw

tamir — Wed, 13 Mar 2024 09:25:06 GMT

hey @gcusello thanks for your replay.
It seems like the capture do not capture any of the fields i needed, I've tried to save it an even to play a bit with the syntax. but still no success.

Re: regex field extraction not from _raw

tamir — Wed, 13 Mar 2024 09:21:39 GMT

@richgalloway thanks for replay the | rex is working as it should the problem start when I'm trying to save the Regex. and this is cause by the fact i need to save the regex from the "source" field and no from the "_raw" field.
The main goal is to add another field in all searches without using the | rex command every time.

Re: regex field extraction not from _raw

gcusello — Wed, 13 Mar 2024 09:55:24 GMT

Hi @tamir ,

my solution is to save the extraction in an field extraction,

if you want to use the regex in a search, you have to add it to a search:

index=your_index | rex field=source "Snowflake\/(?<folder>[^\/]+)"

Ciao.

Giuseppe

Re: regex field extraction not from _raw

isoutamo — Wed, 13 Mar 2024 12:38:36 GMT

you must use transforms to get this done.

Create Field transformations on your app e.g.
1. get_directory_from_linux_audit_source
2. Type regex-based
3. Regular expression like: /log/([^/]+)/
4. Format: directory::$1
5. Source key: source
6. Save and give needed permissions like app and roles which can use it
Create Field extractions
1. Name: <what ever you want to call it>
2. Apply to: sourcetype named: e.g. linux_audit or what ever this is in your node
3. Type: Uses transforms
4. Extraction/Transforms: <from above like get_directory_from_linux_audit_source>
5. Save and give needed permission like above
Wait that this will be applied to all needed places on SCP
Use it like: index=<your index> sourcetype=<your sourcetype> <Your field name>=*

r. Ismo

Re: regex field extraction not from _raw

tamir — Mon, 18 Mar 2024 12:50:40 GMT

@isoutamo hey thanks for the replay.
I've been trying to create the following two you shared, but somehow i still don't see the the field in the field section I'm sharing the process I've taking.
let me know if I'm missing something.

Re: regex field extraction not from _raw

isoutamo — Mon, 18 Mar 2024 13:54:27 GMT

Are you sure that your transformation's name is same on both places (e.g. sonw vs snow etc.)?