Splunk Search

Search performance benefit from field inclusion versus exclusion

johnjarvis
Explorer

Hi all,

First, I do apologise if this is clearly answered in Answers or Documentation; I have spent some time in both, and have still to find an answer.

Second, I am very new to Splunk. In fact, this question comes directly from Fundamentals One; a throw-away comment in Module 8, to be specific.

And so, my question: on the subject of search performance, and field extraction in particular, the instructor states that field inclusion can provide a boost, as it occurs before field extraction; he then goes on to say that field exclusion offers no such benefit, as it occurs after field extraction.

I'm trying to wrap my head around why this is the case; that is, why field exclusion differs so markedly from field inclusion, in terms of what Splunk knows about the entire search at the point of field extraction.

Thanks! And apologies for any stumbles re lexicon/vocabulary.

John

0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

View solution in original post

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

0 Karma

johnjarvis
Explorer

Yes, that's exactly the scenario I was raising, but my question is a bit more nuanced than that. (I absolutely take the point you make, though.)

The instructor implied that, to continue your example, the search piped to | fields - d, e, f performs exactly the same as the search without any pipe (or without a pipe to any field exclusions).

If Splunk knows about field inclusions when it's looking at field extraction, shouldn't it also know about field exclusions, and thereby get a -- very, very slight, in all likelihood, I'll grant -- performance boost from what it needn't consider?

John

0 Karma

isoutamo
SplunkTrust
SplunkTrust

In extraction case it the spl/splunk search optimizer can not utilize this information before it has extracted all fields. But after | fields - d, e, f | it can utilize that information on the next part of query as minimizing transferred data etc. if I had understood this correctly.

Ismo

0 Karma

johnjarvis
Explorer

Ah, OK. So it's simply a matter of field inclusion being explicitly part of predicate optimisation -- as per "> Documentation > Splunk® Enterprise > Search Manual > Built-in optimization" -- and field exclusion isn't, correct?

John

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...