Splunk Search

What is the difference between index time extractions and search time extractions?

aatha89
Explorer

My question is what is the difference between an index time extraction and a search time extraction? Can anyone explain with some simple examples?

I have tried to read this :

one:https://docs.splunk.com/Documentation/Splunk/7.1.3/Indexer/Indextimeversussearchtime, but i stil cant understand.

Tags (1)
1 Solution

nilbak1
Communicator

*Index time Field extraction - *

  • It happens at index time when splunk indexes data.

  • At index time, it extracts some default fields like source, source types and hosts.

  • We can also define our custom source types, hosts so that it tags events with them.

*Search time field extraction- *

  • It happens at search time when we search through data.

  • It can extract additional fields other than default fields depending on its search settings.

  • It includes aliasing, tagging, addition of fields from lookup. But here, you cannot change host or source type assignments.

Hope i am able to answer your question properly 🙂

View solution in original post

iamarkaprabha
Contributor

Hi ,

It was mentioned earlier by Stephen Sorkin [Splunk] ♦ in one of his answers. I am just coping this remark on the same. i find it really useful

In general, we recommend search-time extractions rather than index-time extractions. There are relatively few cases where index-time extractions are better, and they come at the cost of brittleness of configuration and an increase in index size (which in turn makes searches slower).

The distinction in the UI of "uses transform" vs. inline doesn't have anything to do with search-time vs index-time. It is referring to where the regex itself is stored: in an EXTRACT- line in props.conf (for inline) as opposed to in a REPORT- line that refers to a stanza in transforms.conf (for uses transform).

Index time extractions are also set in props.conf and transforms.conf by means of the TRANSFORM- line. Again, they should rarely be used. They are appropriate when the heuristic of search for the value of the field fails (either because the value is ubiquitous outside of cases where the field equals the value, or because the value isn't an indexed token) or when you commonly search for field!=value without other terms to constrain the search.

Link of the answer : https://answers.splunk.com/answers/5817/search-time-versus-index-time-field-extractions.html

nilbak1
Communicator

*Index time Field extraction - *

  • It happens at index time when splunk indexes data.

  • At index time, it extracts some default fields like source, source types and hosts.

  • We can also define our custom source types, hosts so that it tags events with them.

*Search time field extraction- *

  • It happens at search time when we search through data.

  • It can extract additional fields other than default fields depending on its search settings.

  • It includes aliasing, tagging, addition of fields from lookup. But here, you cannot change host or source type assignments.

Hope i am able to answer your question properly 🙂

marycordova
SplunkTrust
SplunkTrust

splunk will create some default (metadata) fields such as _time, host, index, source, sourcetype etc and write those to disk on the file system along with the raw log event string

example raw log: time=1539116213 user=mary host=laptop ip=192.168.0.1

when you run the query index=main user=mary host=laptop splunk performs search time field extractions on the raw log looking for matches to your search for those fields that are not metadata and/or indexed and written to file system disk field value pairs

index time extractions are field value pairs written to disk just like the metadata fields but for the most part this is unnecessary and results greater disk usage on your indexers without providing additional benefit

there are of course use-cases for, and exceptions to, all of this

@marycordova

Vijeta
Influencer

Index time extractions are the field extractions done at the indexer end when the data is indexed.
Search time extractions are field extractions at Search Heads done at the search time for eg. extracting a field using rex command in your search or defining field extractions on search heads.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...