topic Re: multivalue field search time extraction through UI in Splunk Search

multivalue field search time extraction through UI

nawazns5038 — Wed, 22 Apr 2020 05:38:48 GMT

Hi,

I would like to extract field values from UI using the field transformations and field extractions from settings.

I have added the field extractions and referenced it to use a field transform. I follow the same naming conventions for other normal field extractions with transforms and it works well.

  props.conf
 [sourcetype]
 REPORT-IP = REPORT-IP

transforms.conf 
[REPORT-IP]
FORMAT = IP::$1
MV_ADD = 1
REGEX = c=IN IP4 (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
SOURCE_KEY = list

I have verified the regex and it captures well when checked in general, however I don't see the required multi value field in Splunk with the values extracted.

Is anything else missing ?

Thanks

Re: multivalue field search time extraction through UI

DalJeanis — Fri, 24 Apr 2020 02:27:27 GMT

These two are always different values. I don't know what the effect of this would be. The second one should match the stanza in transforms.conf

 [sourcetype]
 REPORT-IP = REPORT-stanza

  [REPORT-stanza]
 your code

Re: multivalue field search time extraction through UI

michael_bates_1 — Fri, 24 Apr 2020 05:51:26 GMT

Hi, Are you able to provide a sample of the data you are attempting to extract from.

Re: multivalue field search time extraction through UI

nawazns5038 — Mon, 27 Apr 2020 23:35:28 GMT

I can understand what you mean but it does not effect anything , I use the same format for other field transforms as well in order to avoid confusion and it works well.

Re: multivalue field search time extraction through UI

michael_bates_1 — Mon, 27 Apr 2020 23:49:23 GMT

Hi
The most likely suspect is the field "list" does not exist when the transform attempts to execute.
Search time order of operations are

Inline extraction from props
Transform-based extraction
Auto KV fields
Field-Alias
Calc fields
Lookups
Eventtypes
Tags

Can you please provide some insight into how/when the field "list" used as the SOURCE_KEY is extracted.

Thanks.

Re: multivalue field search time extraction through UI

nawazns5038 — Mon, 27 Apr 2020 23:57:57 GMT

zsdfgdsgfgsdfg dfgkh sdfl;askgh a;sdfkghjak gah fkgakfgakgj ha;sdghkjasgh as;kgkahsjkghao[YOWERT    [Y  [OYTUWEH ASDGH IQHIWURHTGIWQUHO HEOW    ROIouoasdfoutyet uertyuqewr tyouyqwrtuootptypowreqyturwet wqrtu8 sdfgsdfg sdfg w['aE QWE[T QWETJHTJKERHWTERHWKTH EJRTHKJEWRHTUII    auioprtywerryweuortyew c=IN IP4 10.33.22.11ghghfjgfhj ghjgjedkj wrktjyhoe oqw4u5yohwueirthywi hqiuehrtqtp[[qh erhtjqehrtueqhrtuiqwyhtkjewhgiuhdefjkaslhguqirotuyqpuyt quywertuipyp  y] oiq  ytuoqerytu htuqerhuthuqeirhtuy4238[ ]2u5]r  iuwifhdjgfhjrhgtu
=-+c=IN IP4 10.44.33.22+t334543 34q535asdtesdft

Re: multivalue field search time extraction through UI

to4kawa — Tue, 28 Apr 2020 00:10:49 GMT

try REGEX options.

props.conf

[yourSourcetype]
REPORT-IP = IP

transforms.conf

 [IP]
 FORMAT = IP::$1
 MV_ADD = 1
 REGEX = (?ms)c=IN\sIP4\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
 SOURCE_KEY = list

Sample:

| makeresults
| eval list="zsdfgdsgfgsdfg dfgkh sdfl;askgh a;sdfkghjak gah fkgakfgakgj ha;sdghkjasgh as;kgkahsjkghao[YOWERT    [Y     [OYTUWEH ASDGH IQHIWURHTGIWQUHO    HEOW    ROIouoasdfoutyet uertyuqewr tyouyqwrtuootptypowreqyturwet wqrtu8 sdfgsdfg sdfg w['aE QWE[T QWETJHTJKERHWTERHWKTH EJRTHKJEWRHTUII    auioprtywerryweuortyew c=IN IP4 10.33.22.11ghghfjgfhj ghjgjedkj wrktjyhoe oqw4u5yohwueirthywi hqiuehrtqtp[[qh erhtjqehrtueqhrtuiqwyhtkjewhgiuhdefjkaslhguqirotuyqpuyt quywertuipyp    y] oiq    ytuoqerytu htuqerhuthuqeirhtuy4238[    ]2u5]r    iuwifhdjgfhjrhgtu
 =-+c=IN IP4 10.44.33.22+t334543 34q535asdtesdft"
 | rex field=list max_match=0 "(?ms)c=IN\sIP4\s(?<IP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"

Re: multivalue field search time extraction through UI

nawazns5038 — Tue, 28 Apr 2020 20:20:01 GMT

I figured out the issue here.

The SOURCE_KEY = list , the field list here is an already search time extracted field. And I am trying to apply one more search time field extraction or a search time multi value field extraction to it and so it could not work.

If I change the SOURCE_KEY = _raw , it works perfect but does not solve my case. As I want the extraction particular only to that field.

I did not change any other parameters in my settings shown in the question except for the source_key.

Re: multivalue field search time extraction through UI

nawazns5038 — Tue, 28 Apr 2020 20:20:47 GMT

Thanks for the regex.

I figured out the issue here.

The SOURCE_KEY = list , the field list here is an already search time extracted field. And I am trying to apply one more search time field extraction or a search time multi value field extraction to it and so it could not work.

If I change the SOURCE_KEY = _raw , it works perfect but does not solve my case. As I want the extraction particular only to that field.

Re: multivalue field search time extraction through UI

nawazns5038 — Tue, 28 Apr 2020 20:24:11 GMT

I figured out the issue here.

If I change the SOURCE_KEY = _raw , it works perfect but does not solve my case. As I want the extraction particular only to that field.

I did not change any other parameters in my settings shown in the question except for the source_key.

Re: multivalue field search time extraction through UI

nawazns5038 — Tue, 28 Apr 2020 20:24:58 GMT

So you mean we cannot perform search time extractions again on already search time extracted fields. ??

Re: multivalue field search time extraction through UI

michael_bates_1 — Wed, 30 Sep 2020 05:14:49 GMT

Please pay attention to the "order of operations" posted earlier.
Fields from a previous step are available in subsequent steps except for calculated fields (they run in parallel), so,

If you configure either an inline field extraction, or an additional Transforms that is executed first, then you would be able to use the field.

E.g
props.conf

[sourcetype]
EXTRACT-list = regex_to_extract_the_list
REPORT-ip = ip_from_list

transforms.conf
[ip_from_list]
MV_ADD = 1
SOURCE_KEY = list
REGEX = c=IN\sIP4\s(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})
FORMAT = IP::$1

Re: multivalue field search time extraction through UI

to4kawa — Tue, 28 Apr 2020 22:34:13 GMT

So there's a problem with the other settings.