Search performance benefit from field inclusion versus exclusion

[johnjarvis]

Hi all,

First, I do apologise if this is clearly answered in Answers or Documentation; I have spent some time in both, and have still to find an answer.

Second, I am very new to Splunk. In fact, this question comes directly from Fundamentals One; a throw-away comment in Module 8, to be specific.

And so, my question: on the subject of search performance, and field extraction in particular, the instructor states that field inclusion can provide a boost, as it occurs before field extraction; he then goes on to say that field exclusion offers no such benefit, as it occurs after field extraction.

I'm trying to wrap my head around why this is the case; that is, why field exclusion differs so markedly from field inclusion, in terms of what Splunk knows about the entire search at the point of field extraction.

Thanks! And apologies for any stumbles re lexicon/vocabulary.

John

[isoutamo · Accepted solution]

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

[johnjarvis]

Yes, that's exactly the scenario I was raising, but my question is a bit more nuanced than that. (I absolutely take the point you make, though.)

The instructor implied that, to continue your example, the search piped to | fields - d, e, f performs exactly the same as the search without any pipe (or without a pipe to any field exclusions).

If Splunk knows about field inclusions when it's looking at field extraction, shouldn't it also know about field exclusions, and thereby get a -- very, very slight, in all likelihood, I'll grant -- performance boost from what it needn't consider?

John

[isoutamo]

In extraction case it the spl/splunk search optimizer can not utilize this information before it has extracted all fields. But after | fields - d, e, f | it can utilize that information on the next part of query as minimizing transferred data etc. if I had understood this correctly.

Ismo

[johnjarvis]

Ah, OK. So it's simply a matter of field inclusion being explicitly part of predicate optimisation -- as per "> Documentation > Splunk® Enterprise > Search Manual > Built-in optimization" -- and field exclusion isn't, correct?

John