Extracted field in fast mode

[jariw]

Hello,   i try to understand the "fast mode" compared to the "smart" and "verbose mode" in relation to field extractions.

Backgound:   I am investigating the slownes of Splunk searches on the "Enterprise  Security SHC" compared to non ES SHC. I tried below search:  

index=wineventlog earliest=1761217200 latest=1761220800

on both SHC.  It gives a difference. ES takes 30 sec's , non ES takes 17 sec's.

I noticed that the "interesting fields" are different. Besides the default fields on the ES SHC there is the "Messages" field.  That's no default field. On the non ES SHC there are only the default fields visible.

I checked all the props, transforms, viewstates... whatever to find the cause off this. 

If i do the search on the non ES SHC with extra "Messages=*" .  Then it gives the same search time as on the ES SHS.  So it looks like a extra extraction on the ES SHC, one which we don't know anything about. On which happens "under water".

Maybe it is the cause off the slowness, maybe not. But i want to now why this happens before looking anything further.

Can anybody explain the use off "fast mode" and when (c.q. why) this field extraction happens? WHy is this "Messages" in the interesting fields? What can cause this to happen?

Thanks in advance.

Greetings

Jari

[bowesmana]

On ES there will also be a ton of automatic lookups. If you look at the search.log you will see lines like

CsvDataProvider [2092202 searchOrchestrator] - Reading schema for lookup table=

how many of each do you have in each environment. That can change what fields you will see and it certainly can affect performance significantly. 

You can see the effect of this behaviour if you upload a private app that has an import = search statement in default.meta and then run your search in that uploaded app's context. The undocumented import statement appears to override which apps are evaluated to give the app its runtime context, so by removing all other apps, you will not get any of the field extractions provided by those TAs or any of the lookups.

I have an existing site where I have an app that has an import statement and the searches run several times faster in that app's context compared to any other.

[PickleRick]

OK. If a search is running in Fast mode, Splunk only extracts the fields explicitly named in the search (and reads the default metadata indexed fields). If you run in Verbose mode, Splunk extracts everything it can. Additionally it "back-maps" eventtypes and tags to events. That's where the difference in execution time comes from.

And since Splunk doesn't extract the fields when running in Fast mode it doesn't show you those fields in "Interesting fields".

EDIT: Splunk can be... "tamed" a bit around field extraction even in Smart/Verbose mode by using the fields command. This isn't that important when running the searches manually but since reports run in Smart mode, limiting extracted fields can have performance impact.

[jariw]

Hi @PickleRick 

The "strange" thing is.. both searches are in fast mode... even the one which gives the Message field.

I can't find where this field is extracted during the search, and it isn't a indexed field. (if it was, the fast mode should have showed it on both SHC).

greetz

Jari

[PickleRick]

Is it the same search?

Also - I'm not sure what happens if you "pin" a field into the "interesting fields" section manually. I don't use this functionality myself.

Anyway, you might want to compare the job details for both searches and see whether the field is explicitly requested somehow. (and go down into the job log if you're brave enough 😉 )

[jariw]

Yep... same search.

I am going the search properties way now, Dirst thing i noticed in the results:

ES SHC: eventFieldCount 7

non ES: eventFieldCount 6

greets

Jari

[jariw]

I've found something in de search.log from the indexer peer itself.

In the non ES search i found:

INFO SearchOperator:kv [2700279 StreamSearch] - Trying to disable extractor name="wel-message"
INFO SearchOperator:kv [2700279 StreamSearch] - Disabling extractor name="wel-message" - it does not extract any of the required fields

And on the ES search i found only:

INFO SearchOperator:kv [2495326 StreamSearch] - Trying to disable extractor name="wel-message"

Not disabling this extraction (and multiple other extractions. So it seems that in fast mode not all extractions are disabled on the ES SHC.

Question is why..

[PickleRick]

I would expect it to be the other way around. Splunk can optimize out some extractions so that not all are run if not all are needed. So if the fields extracted by a particular extraction are known and were not requested Splunk might not run the extraction during search at all.

[gcusello]

Hi @jariw ,

you're comparing pears with apples!

to compare two searches, you have to work:

• on the same machine and in the same time,
• not using earliest=now, but something in the past (e.g. from -70m@m to -10m@m), to be sure to search on the same events,
• run both the searches in the same app, because there could be different eventtypes, tags and field extractions that give different times,
• use the same mode: Fast, Smart or Verbose, isn't relevant, but use the same.

Anyway, the presence of ES on a SH gives a greater load on the server, so this server must have more resources (e.g. to run Splunk you need at least 12 CPUs, to run ES you need at least 16 CPUs or more!), so for this reason you cannot compare searches.

But, why do you need to do this comparison?

Ciao.

Giuseppe

[jariw]

Hi @gcusello 

I can understand the same machine. but about time the search itself has the time in it ( earliest=1761217200 latest=1761220800 ) so that must be no problem.

Both searches are run from Search.. so i think same app

And indeed the ES has much more extra overhead, and maybe there is something for the "Messages' field.

But the question is in fact what can cause the "Messages" field to apear in the interesting fields. after that i will look further for the slow searches.

greets

Jari

[jariw]

Hi @livehybrid 

Thanx for the quick response.

This fields.conf is new for me, but

/opt/splunk/bin/splunk btool fields list

gave no Messages field in the list with fields. 

So " Field Discovery is somewhat limited" , but to which degree ?

greets

Jari

[livehybrid]

Hi @jariw 

Is Messages an index-time field which is defined in fields.conf on the ES SH? This might explain why it appears in fast mode on the ES SH but no the other.

"Fast mode" in Splunk is designed to prioritise the speed of search results by minimising unnecessary processing and field extraction. Specifically, in fast mode Field Discovery is somewhat limited and you will probably find no search-time extractions appear, however indexed fields will.

The ES SH may have a bunch of additional automatic lookups which may be adding time to the processing.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing