topic Re: Create calculated field during the indexing in Getting Data In

Create calculated field during the indexing

rayar — Mon, 21 Mar 2022 10:07:43 GMT

We are considering to calculate specific filed (list) during the indexing

the calculation will be - | eval list=if(match(dhost,"\.[\w]{2,3}\.[\w]{2}:?[\d]?"),"mozilla","iana")

1. What is the performance impact ?

2. how it should be done ?

Re: Create calculated field during the indexing

gcusello — Mon, 21 Mar 2022 10:38:14 GMT

Hi @rayar,

the choose to use indexed fields (or not) depends on the volume of indexed data and searches:

if you have a large volume of logs to index, it isn't a good ideas because you overload your indexers and it's better to perform this extraction at search time when you already filtered yourdata in a search,
if you have many searches using those fields, it could be useful.

In other words, you have to decide if anticipate a work at index time or to use it at search time.

If you have intermediate Heavy Forwarders, they could be used to extract fields without overloading Indexers.

About the way to do this, you can see at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Configureindex-timefieldextraction

I usually don't extract fields at index time!

Ciao.

Giuseppe

Re: Create calculated field during the indexing

somesoni2 — Mon, 21 Mar 2022 13:57:19 GMT

As far as I know, there is no calculated indexed-time field extraction possible in Splunk. All indexed time field extractions are either static OR using regex on _raw OR metadata fields (host/source/sourcetype/_time) etc.