Splunk Search

Search performance benefit from field inclusion versus exclusion

johnjarvis
Explorer

Hi all,

First, I do apologise if this is clearly answered in Answers or Documentation; I have spent some time in both, and have still to find an answer.

Second, I am very new to Splunk. In fact, this question comes directly from Fundamentals One; a throw-away comment in Module 8, to be specific.

And so, my question: on the subject of search performance, and field extraction in particular, the instructor states that field inclusion can provide a boost, as it occurs before field extraction; he then goes on to say that field exclusion offers no such benefit, as it occurs after field extraction.

I'm trying to wrap my head around why this is the case; that is, why field exclusion differs so markedly from field inclusion, in terms of what Splunk knows about the entire search at the point of field extraction.

Thanks! And apologies for any stumbles re lexicon/vocabulary.

John

0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

View solution in original post

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

If I understood correctly you are asking that why

| fields a, b, c

performs better than

| fields - d, e, f

The reason is that in 1st cases splunk knows what to look instead of 2nd where it needs to look every possible extractions and then remove those unwanted ones. You can imagine which kind of difference it could be e.g. for hundreds of fields.

R. Ismo

0 Karma

johnjarvis
Explorer

Yes, that's exactly the scenario I was raising, but my question is a bit more nuanced than that. (I absolutely take the point you make, though.)

The instructor implied that, to continue your example, the search piped to | fields - d, e, f performs exactly the same as the search without any pipe (or without a pipe to any field exclusions).

If Splunk knows about field inclusions when it's looking at field extraction, shouldn't it also know about field exclusions, and thereby get a -- very, very slight, in all likelihood, I'll grant -- performance boost from what it needn't consider?

John

0 Karma

isoutamo
SplunkTrust
SplunkTrust

In extraction case it the spl/splunk search optimizer can not utilize this information before it has extracted all fields. But after | fields - d, e, f | it can utilize that information on the next part of query as minimizing transferred data etc. if I had understood this correctly.

Ismo

0 Karma

johnjarvis
Explorer

Ah, OK. So it's simply a matter of field inclusion being explicitly part of predicate optimisation -- as per "> Documentation > Splunk® Enterprise > Search Manual > Built-in optimization" -- and field exclusion isn't, correct?

John

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...