Splunk Search

Search performance benefit from field inclusion versus exclusion

Explorer

Hi all,

First, I do apologise if this is clearly answered in Answers or Documentation; I have spent some time in both, and have still to find an answer.

Second, I am very new to Splunk. In fact, this question comes directly from Fundamentals One; a throw-away comment in Module 8, to be specific.

And so, my question: on the subject of search performance, and field extraction in particular, the instructor states that field inclusion can provide a boost, as it occurs before field extraction; he then goes on to say that field exclusion offers no such benefit, as it occurs after field extraction.

I'm trying to wrap my head around why this is the case; that is, why field exclusion differs so markedly from field inclusion, in terms of what Splunk knows about the entire search at the point of field extraction.

Thanks! And apologies for any stumbles re lexicon/vocabulary.

John

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

View solution in original post

0 Karma

Explorer

Yes, that's exactly the scenario I was raising, but my question is a bit more nuanced than that. (I absolutely take the point you make, though.)

The instructor implied that, to continue your example, the search piped to | fields - d, e, f performs exactly the same as the search without any pipe (or without a pipe to any field exclusions).

If Splunk knows about field inclusions when it's looking at field extraction, shouldn't it also know about field exclusions, and thereby get a -- very, very slight, in all likelihood, I'll grant -- performance boost from what it needn't consider?

John

0 Karma

SplunkTrust
SplunkTrust

In extraction case it the spl/splunk search optimizer can not utilize this information before it has extracted all fields. But after | fields - d, e, f | it can utilize that information on the next part of query as minimizing transferred data etc. if I had understood this correctly.

Ismo

0 Karma

Explorer

Ah, OK. So it's simply a matter of field inclusion being explicitly part of predicate optimisation -- as per "> Documentation > Splunk® Enterprise > Search Manual > Built-in optimization" -- and field exclusion isn't, correct?

John

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!