topic Re: What is the difference between index time extractions and search time extractions? in Splunk Search

What is the difference between index time extractions and search time extractions?

aatha89 — Tue, 09 Oct 2018 19:03:59 GMT

My question is what is the difference between an index time extraction and a search time extraction? Can anyone explain with some simple examples?

I have tried to read this :

one:https://docs.splunk.com/Documentation/Splunk/7.1.3/Indexer/Indextimeversussearchtime, but i stil cant understand.

Re: What is the difference between index time extractions and search time extractions?

kmaron — Tue, 09 Oct 2018 19:59:52 GMT

I'm not going to be able to explain much myself but maybe something in one of these will help?

https://answers.splunk.com/answers/5817/search-time-versus-index-time-field-extractions.html
https://answers.splunk.com/answers/151939/how-do-index-and-search-time-field-extractions-differ-and-which-is-better-for-search-performance.html
https://answers.splunk.com/answers/396276/should-i-use-an-index-time-field-extraction.html

Re: What is the difference between index time extractions and search time extractions?

Vijeta — Tue, 09 Oct 2018 20:08:59 GMT

Index time extractions are the field extractions done at the indexer end when the data is indexed.
Search time extractions are field extractions at Search Heads done at the search time for eg. extracting a field using rex command in your search or defining field extractions on search heads.

Re: What is the difference between index time extractions and search time extractions?

marycordova — Tue, 09 Oct 2018 20:32:46 GMT

splunk will create some default (metadata) fields such as _time, host, index, source, sourcetype etc and write those to disk on the file system along with the raw log event string

example raw log: time=1539116213 user=mary host=laptop ip=192.168.0.1

when you run the query index=main user=mary host=laptop splunk performs search time field extractions on the raw log looking for matches to your search for those fields that are not metadata and/or indexed and written to file system disk field value pairs

index time extractions are field value pairs written to disk just like the metadata fields but for the most part this is unnecessary and results greater disk usage on your indexers without providing additional benefit

there are of course use-cases for, and exceptions to, all of this

Re: What is the difference between index time extractions and search time extractions?

nilbak1 — Wed, 10 Oct 2018 06:17:16 GMT

*Index time Field extraction - *

It happens at index time when splunk indexes data.
At index time, it extracts some default fields like source, source types and hosts.
We can also define our custom source types, hosts so that it tags events with them.

*Search time field extraction- *

It happens at search time when we search through data.
It can extract additional fields other than default fields depending on its search settings.
It includes aliasing, tagging, addition of fields from lookup. But here, you cannot change host or source type assignments.

Hope i am able to answer your question properly 🙂

Re: What is the difference between index time extractions and search time extractions?

iamarkaprabha — Wed, 10 Oct 2018 18:53:05 GMT

Hi ,

It was mentioned earlier by Stephen Sorkin [Splunk] ♦ in one of his answers. I am just coping this remark on the same. i find it really useful

In general, we recommend search-time extractions rather than index-time extractions. There are relatively few cases where index-time extractions are better, and they come at the cost of brittleness of configuration and an increase in index size (which in turn makes searches slower).

The distinction in the UI of "uses transform" vs. inline doesn't have anything to do with search-time vs index-time. It is referring to where the regex itself is stored: in an EXTRACT- line in props.conf (for inline) as opposed to in a REPORT- line that refers to a stanza in transforms.conf (for uses transform).

Index time extractions are also set in props.conf and transforms.conf by means of the TRANSFORM- line. Again, they should rarely be used. They are appropriate when the heuristic of search for the value of the field fails (either because the value is ubiquitous outside of cases where the field equals the value, or because the value isn't an indexed token) or when you commonly search for field!=value without other terms to constrain the search.

Link of the answer : https://answers.splunk.com/answers/5817/search-time-versus-index-time-field-extractions.html