topic Re: In what phase are discovered fields available (automatic field extraction)? in Splunk Search

In what phase are discovered fields available (automatic field extraction)?

vxsplunk — Tue, 29 Sep 2020 11:29:03 GMT

I want to add a field to my events that is derived from a discovered field at search time. The new field wil be a primary field used in searches, therefore I can't use "| eval new=substr(..)".

Example logging:

2016-10-18 11:22:33.456 ERROR session=12akeife0wkefs0e835i5l0iwef

When searched for 'ERROR', the field 'session' is automatically discovered. That's good.

Now I want to create a new field 'ses' and fill it with the first 10 characters of 'session'. This is how I think this should be done:

transforms.conf:    
[ses]
REGEX=(.{10})
FORMAT=ses::$1
SOURCE_KEY=session

props.conf:
[sourcetype_x]
REPORT-ses = ses

Now when I search for 'ERROR', I would expect to find the new field 'ses' with value '12akeife0w', but it isn't there.

I took notice of:

http://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Configurationparametersandthedatapipeline#How_configuration_parameters_correlate_to_phases_of_the_pipeline
How configuration parameters correlate to phases of the pipeline - Search phase

http://docs.splunk.com/Documentation/Splunk/6.5.0/admin/propsconf
Search-time field extractions: Why use REPORT if EXTRACT will do?
It's a good question. And much of the time, EXTRACT is all you need for
search-time field extraction. But when you build search-time field
extractions, there are specific cases that require the use of REPORT and the
field transform that it references. Use REPORT if you want to:
...
* Manage formatting of extracted fields, in cases where you are extracting
multiple fields, or are extracting both the field name and field value.

http://docs.splunk.com/Documentation/Splunk/6.5.0/Knowledge/WhenSplunkEnterpriseaddsfields
When field discovery is enabled, Splunk software:
• **Identifies and extracts the first 50 fields that it finds in the event data that match obvious key=value pairs. This 50 field limit is a default that you can modify by editing the [kv] stanza in limits.conf, if you have Splunk Enterprise.
• Extracts any field explicitly mentioned in the search that it might otherwise have found though automatic extraction, but is not among the first 50 fields identified.
• **Performs custom field extractions that you have defined, either through the Field Extractor, the Extracted Fields page in Settings, configuration file edits, or search commands such as rex.

My question(s):
In what phase are discovered fields available?
Are discovered fields available before a transform at search-time?

Re: In what phase are discovered fields available (automatic field extraction)?

sundareshr — Tue, 18 Oct 2016 13:08:34 GMT

Try this for your transforms. Removed SOURCE_KEY The source for your extraction is _raw (default). and modified your regex.

transforms.conf:    
 [ses]
 REGEX=session=(\w{10})
 FORMAT=ses::$1

Re: In what phase are discovered fields available (automatic field extraction)?

vxsplunk — Tue, 29 Sep 2020 11:29:58 GMT

I knew this workaround and it works, but that is not what I was trying to accomplish or to find out. Because of SOURCE_KEY=_raw I have to manually extract fields again that are also auto discovered/extracted and this gives me much more to configure.

To better understand field discovery I would like to know in what phase (see my intro to the question) field discovery is done?

Re: In what phase are discovered fields available (automatic field extraction)?

sundareshr — Wed, 19 Oct 2016 10:50:19 GMT

See if this answers your question. Field discovery is done in the search phase. Field discovery depends on your search mode. If you search in Fast Mode, Splunk does not auto discover fields. In Smart mode, only if you don't use any transforming commands. However, it will always auto-discover fields in the Verbose mode.

http://docs.splunk.com/Documentation/Splunk/6.5.0/Knowledge/WhenSplunkEnterpriseaddsfields

Re: In what phase are discovered fields available (automatic field extraction)?

vxsplunk — Fri, 21 Oct 2016 13:03:30 GMT

Splunk will also auto discover fields that are mentioned in the search and it doesn't matter in which search mode you are, but that shouldn't be the problem because this 'ses' field will be used in the search (ses=1234567890). It will be a key-part of the search I ultimately trying to use.

Some documentation about this (see the link I mentioned in the question and in you last comment http://docs.splunk.com/Documentation/Splunk/6.5.0/Knowledge/WhenSplunkEnterpriseaddsfields):

When field discovery is enabled, Splunk software:
- Identifies and extracts the first 50 fields that it finds in the event data that match obvious key=value pairs. This 50 field limit is a default that you can modify by editing the [kv] stanza in limits.conf, if you have Splunk Enterprise.
- Extracts any field explicitly mentioned in the search that it might otherwise have found though automatic extraction, but is not among the first 50 fields identified.
- Performs custom field extractions that you have defined, either through the Field Extractor, the Extracted Fields page in Settings, configuration file edits, or search commands such as rex.

Is Splunk following this order as mentioned (top-down)?
- Yes? Then I expect my example in the question to work
- No? What order is followed?