topic Re: How to extract values from a field instead of _raw? in Splunk Search

How to extract values from a field instead of _raw?

soumyacharya91 — Tue, 24 Apr 2018 09:40:42 GMT

I am getting below log and want to extract the data/values from the field using props.conf / transforms.conf.

Field_name:  [
  "value", "value", "value", "value"
]

In _raw format I'm getting the below logs from the same

"Field_name": "[\r\n  \"value\",\r\n  \"value\",\r\n  \"value\",\r\n  \"value\"]"

Any help will be much appreciated.

Thanks

Re: How to extract values from a field instead of _raw?

p_gurav — Tue, 24 Apr 2018 13:53:47 GMT

Can you share whole event? Also what rex your using? You can use | rex field=<field_name> max_match=0 "reg_exp"

Re: How to extract values from a field instead of _raw?

maciep — Wed, 25 Apr 2018 01:49:14 GMT

I don't understand your examples, but there are at least 2 ways to extract new fields from existing fields in props/transforms. Let's say you have already extracted a field called "my_field".

Then using EXTRACT in props, you can tell splunk the field to run the regex against by adding " in myfield" after your regex.

props

[your_sourcetype]
EXTRACT-some_new_field = <your regex> in my_field

Or if you're using props/transforms, then specify the existing field as the source key in transforms

props

[your_sourcetype]
REPORT-some_new_field = extract_new_field

transforms

[extract_new_field]
SOURCE_KEY = my_field
REGEX = <your regex>

Re: How to extract values from a field instead of _raw?

soumyacharya91 — Wed, 25 Apr 2018 08:56:09 GMT

Hi I have tried the solution using props.conf and transforms.conf

Seems it is working properly when I search using extract reload=t. But without this command in search query it is not working.

Transforms.conf

[name]
SOURCE_KEY=
REGEX =\"(?[^\s]+)\"

props.conf

REPORT-classname=name

Re: How to extract values from a field instead of _raw?

maciep — Wed, 25 Apr 2018 10:35:11 GMT

that doesn't make any sense. that extract command should force it to reload the config, which it will do on its own on some interval as well. that shouldn't make it work or not work on a consistent basis.

but i thought you said you were trying to extract a field from a field other than raw. But you leave SOURCE_KEY blank above? put the existing field in there instead of leaving it blank, so your regex runs against it instead of _raw.

Also, please use the code button when posting your config data. As you can see, answers is stripped out important characters from your comment.

with the code button:

\"(?<some_field>[^\s]+)\"

w/o code button:
\"(?[^\s]+)\"

Re: How to extract values from a field instead of _raw?

soumyacharya91 — Wed, 25 Apr 2018 10:57:14 GMT

Actually I provided the field but some how it is not updated here. let me explain you the scenario clearly.

_raw data

"Metadata": "{\r\n  \"GeneratedOnHost\": \"XXXXXX\"\r\n}"

As syntax highlight

Metadata:    {
  "GeneratedOnHost": "XXXXXX"
}

transforms.conf

[name]
SOURCE_KEY=Metadata
REGEX =\"GeneratedOnHost\"\:\s\"(?<extracted field name>[^\s]+)\"

props.conf

REPORT-name1=name

I used this configuration which can able to extract if I include extract reload=t in my search query every time I execute. But without that command it can't even populating the extracted field name.

Re: How to extract values from a field instead of _raw?

maciep — Wed, 25 Apr 2018 11:11:42 GMT

I'm confused as to whether splunk is actually extracting the metadata field already or if you're just assuming because it's syntax-highlighted that it is? You could extract that field yourself first if Splunk isn't doing it for you.

also, are you in fast mode? if you're relying on splunk to extract the metadata field for you, then ensure you're allowing it do that by choosing smart or verbose mode.

Re: How to extract values from a field instead of _raw?

maciep — Wed, 25 Apr 2018 22:02:32 GMT

any update on this? still not working?