Splunk Search

## Search performance benefit from field inclusion versus exclusion

Explorer

Hi all,

First, I do apologise if this is clearly answered in Answers or Documentation; I have spent some time in both, and have still to find an answer.

Second, I am very new to Splunk. In fact, this question comes directly from Fundamentals One; a throw-away comment in Module 8, to be specific.

And so, my question: on the subject of search performance, and field extraction in particular, the instructor states that field inclusion can provide a boost, as it occurs before field extraction; he then goes on to say that field exclusion offers no such benefit, as it occurs after field extraction.

I'm trying to wrap my head around why this is the case; that is, why field exclusion differs so markedly from field inclusion, in terms of what Splunk knows about the entire search at the point of field extraction.

Thanks! And apologies for any stumbles re lexicon/vocabulary.

John

1 Solution
SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

Explorer

Yes, that's exactly the scenario I was raising, but my question is a bit more nuanced than that. (I absolutely take the point you make, though.)

The instructor implied that, to continue your example, the search piped to `| fields - d, e, f` performs exactly the same as the search without any pipe (or without a pipe to any field exclusions).

If Splunk knows about field inclusions when it's looking at field extraction, shouldn't it also know about field exclusions, and thereby get a -- very, very slight, in all likelihood, I'll grant -- performance boost from what it needn't consider?

John

SplunkTrust

In extraction case it the spl/splunk search optimizer can not utilize this information before it has extracted all fields. But after | fields - d, e, f | it can utilize that information on the next part of query as minimizing transferred data etc. if I had understood this correctly.

Ismo

Explorer

Ah, OK. So it's simply a matter of field inclusion being explicitly part of predicate optimisation -- as per "> Documentation > Splunk® Enterprise > Search Manual > Built-in optimization" -- and field exclusion isn't, correct?

John

Get Updates on the Splunk Community!

#### Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

#### Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

#### Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...