multivalue field search time extraction through UI

[nawazns5038]

Hi,

I would like to extract field values from UI using the field transformations and field extractions from settings.

I have added the field extractions and referenced it to use a field transform. I follow the same naming conventions for other normal field extractions with transforms and it works well.

  props.conf
 [sourcetype]
 REPORT-IP = REPORT-IP

transforms.conf 
[REPORT-IP]
FORMAT = IP::$1
MV_ADD = 1
REGEX = c=IN IP4 (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
SOURCE_KEY = list

I have verified the regex and it captures well when checked in general, however I don't see the required multi value field in Splunk with the values extracted.

Is anything else missing ?

Thanks

[nawazns5038]

I figured out the issue here.

The SOURCE_KEY = list , the field list here is an already search time extracted field. And I am trying to apply one more search time field extraction or a search time multi value field extraction to it and so it could not work.

If I change the SOURCE_KEY = _raw , it works perfect but does not solve my case. As I want the extraction particular only to that field.

I did not change any other parameters in my settings shown in the question except for the source_key.

[to4kawa]

try REGEX options.

props.conf

[yourSourcetype]
REPORT-IP = IP

transforms.conf

 [IP]
 FORMAT = IP::$1
 MV_ADD = 1
 REGEX = (?ms)c=IN\sIP4\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
 SOURCE_KEY = list

Sample:

| makeresults
| eval list="zsdfgdsgfgsdfg dfgkh sdfl;askgh a;sdfkghjak gah fkgakfgakgj ha;sdghkjasgh as;kgkahsjkghao[YOWERT    [Y     [OYTUWEH ASDGH IQHIWURHTGIWQUHO    HEOW    ROIouoasdfoutyet uertyuqewr tyouyqwrtuootptypowreqyturwet wqrtu8 sdfgsdfg sdfg w['aE QWE[T QWETJHTJKERHWTERHWKTH EJRTHKJEWRHTUII    auioprtywerryweuortyew c=IN IP4 10.33.22.11ghghfjgfhj ghjgjedkj wrktjyhoe oqw4u5yohwueirthywi hqiuehrtqtp[[qh erhtjqehrtueqhrtuiqwyhtkjewhgiuhdefjkaslhguqirotuyqpuyt quywertuipyp    y] oiq    ytuoqerytu htuqerhuthuqeirhtuy4238[    ]2u5]r    iuwifhdjgfhjrhgtu
 =-+c=IN IP4 10.44.33.22+t334543 34q535asdtesdft"
 | rex field=list max_match=0 "(?ms)c=IN\sIP4\s(?<IP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"

[nawazns5038]

Thanks for the regex.

I figured out the issue here.

The SOURCE_KEY = list , the field list here is an already search time extracted field. And I am trying to apply one more search time field extraction or a search time multi value field extraction to it and so it could not work.

If I change the SOURCE_KEY = _raw , it works perfect but does not solve my case. As I want the extraction particular only to that field.

[to4kawa]

So there's a problem with the other settings.

[michael_bates_1]

Please pay attention to the "order of operations" posted earlier.
Fields from a previous step are available in subsequent steps except for calculated fields (they run in parallel), so,

If you configure either an inline field extraction, or an additional Transforms that is executed first, then you would be able to use the field.

E.g
props.conf

[sourcetype]
EXTRACT-list = regex_to_extract_the_list
REPORT-ip = ip_from_list

transforms.conf
[ip_from_list]
MV_ADD = 1
SOURCE_KEY = list
REGEX = c=IN\sIP4\s(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})
FORMAT = IP::$1

[michael_bates_1]

Hi
The most likely suspect is the field "list" does not exist when the transform attempts to execute.
Search time order of operations are

1. Inline extraction from props
2. Transform-based extraction
3. Auto KV fields
4. Field-Alias
5. Calc fields
6. Lookups
7. Eventtypes
8. Tags

Can you please provide some insight into how/when the field "list" used as the SOURCE_KEY is extracted.

Thanks.

[nawazns5038]

So you mean we cannot perform search time extractions again on already search time extracted fields. ??

[nawazns5038]

I figured out the issue here.

The SOURCE_KEY = list , the field list here is an already search time extracted field. And I am trying to apply one more search time field extraction or a search time multi value field extraction to it and so it could not work.

If I change the SOURCE_KEY = _raw , it works perfect but does not solve my case. As I want the extraction particular only to that field.

I did not change any other parameters in my settings shown in the question except for the source_key.

[michael_bates_1]

Hi, Are you able to provide a sample of the data you are attempting to extract from.

[nawazns5038]

zsdfgdsgfgsdfg dfgkh sdfl;askgh a;sdfkghjak gah fkgakfgakgj ha;sdghkjasgh as;kgkahsjkghao[YOWERT    [Y  [OYTUWEH ASDGH IQHIWURHTGIWQUHO HEOW    ROIouoasdfoutyet uertyuqewr tyouyqwrtuootptypowreqyturwet wqrtu8 sdfgsdfg sdfg w['aE QWE[T QWETJHTJKERHWTERHWKTH EJRTHKJEWRHTUII    auioprtywerryweuortyew c=IN IP4 10.33.22.11ghghfjgfhj ghjgjedkj wrktjyhoe oqw4u5yohwueirthywi hqiuehrtqtp[[qh erhtjqehrtueqhrtuiqwyhtkjewhgiuhdefjkaslhguqirotuyqpuyt quywertuipyp  y] oiq  ytuoqerytu htuqerhuthuqeirhtuy4238[ ]2u5]r  iuwifhdjgfhjrhgtu
=-+c=IN IP4 10.44.33.22+t334543 34q535asdtesdft

[DalJeanis]

These two are always different values. I don't know what the effect of this would be. The second one should match the stanza in transforms.conf

 [sourcetype]
 REPORT-IP = REPORT-stanza

  [REPORT-stanza]
 your code

[nawazns5038]

I can understand what you mean but it does not effect anything , I use the same format for other field transforms as well in order to avoid confusion and it works well.