Splunk Search

Will having lots of extracted fields increase my index size?

Splunk Employee
Splunk Employee

I need to understand how adding fields to raw data will increase our index size growth. We are in the process of adding many fields to individual records. This could run to as much as hundreds per record / line.

Will each record contain a field label and a repeat of the data from the full raw record? This could cause the increase in our index size to be very substantial in extracting fields.

Tags (2)
1 Solution

Splunk Employee
Splunk Employee

Generally, the vast majority of field extractions are performed at search time, so configuring lots of fields per event will not increase your index size by any amount. It is possible to create index-time extractions, so that extra fields are written to the index but that has to be specifically configured and is only recommended in isolated cases where it's absolutely necessary for search-speed to be as fast as possible.

The IFX feature, rex command and regular 'EXTRACT' settings in props.conf all work at search-time. Unless you have purposely configured all of your fields to be created at index-time, they will not increase your index size.

View solution in original post

Splunk Employee
Splunk Employee

Generally, the vast majority of field extractions are performed at search time, so configuring lots of fields per event will not increase your index size by any amount. It is possible to create index-time extractions, so that extra fields are written to the index but that has to be specifically configured and is only recommended in isolated cases where it's absolutely necessary for search-speed to be as fast as possible.

The IFX feature, rex command and regular 'EXTRACT' settings in props.conf all work at search-time. Unless you have purposely configured all of your fields to be created at index-time, they will not increase your index size.

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!