All Apps and Add-ons

Indexed vs search fields,Indexed fields vs search time fields

charlesmeo
Explorer

Does KO explorer show which fields are indexed and which not? This has always been a challenge and anything which does this would be helpful.
I couldn't find a direct answer in the doc or screenshots, but may have just missed it.

This is handy to know if you want to manipulate lispy etc. and do other optimisations on searches.

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

No, the KO Explorer doesn't distinguish between indexed fields and search time fields. It translates your search into normalizedSearch, indexed fields don't play a role there - that's lispy (see https://conf.splunk.com/watch/conf-online.html?search=fn1003 for more).

From a search perspective it's not easy to answer "is this field indexed or not?", at least not generically. In principle the answer could be different from one event to the next. Things you can do:

  • check fields.conf for fields that have explicitly been marked as indexed
  • check props.conf for INDEXED_EXTRACTIONS
  • check transforms.conf for stanzas that generate indexed fields
  • use tstats or the :: syntax to check specific fields in specific events, e.g. | tstats count where index=foo sourcetype=bar by maybe_indexed_field will only return results if maybe_indexed_field is indexed. Similarly, if index=foo sourcetype=bar field=value yields results but index=foo sourcetype=bar field::value does not, field is likely not indexed [caveat: some weirdnesses exist with :: around complex values such as spaces, quotes, etc.]
  • use the walklex command (7.3+, CLI only below that) to list all index-time fields... probably the best approach for discovery: | walklex index=_internal type=field

Finally, you can promote https://ideas.splunk.com/ideas/E-I-70 ("Add an Explain/Debug Mode to display how fields were created") to maybe get an improvement added to core in some future release.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

No, the KO Explorer doesn't distinguish between indexed fields and search time fields. It translates your search into normalizedSearch, indexed fields don't play a role there - that's lispy (see https://conf.splunk.com/watch/conf-online.html?search=fn1003 for more).

From a search perspective it's not easy to answer "is this field indexed or not?", at least not generically. In principle the answer could be different from one event to the next. Things you can do:

  • check fields.conf for fields that have explicitly been marked as indexed
  • check props.conf for INDEXED_EXTRACTIONS
  • check transforms.conf for stanzas that generate indexed fields
  • use tstats or the :: syntax to check specific fields in specific events, e.g. | tstats count where index=foo sourcetype=bar by maybe_indexed_field will only return results if maybe_indexed_field is indexed. Similarly, if index=foo sourcetype=bar field=value yields results but index=foo sourcetype=bar field::value does not, field is likely not indexed [caveat: some weirdnesses exist with :: around complex values such as spaces, quotes, etc.]
  • use the walklex command (7.3+, CLI only below that) to list all index-time fields... probably the best approach for discovery: | walklex index=_internal type=field

Finally, you can promote https://ideas.splunk.com/ideas/E-I-70 ("Add an Explain/Debug Mode to display how fields were created") to maybe get an improvement added to core in some future release.

woodcock
Esteemed Legend

Au contraire mon frère: as of v7.3.0 walklex is also SPL!
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Walklex

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

That's exactly what I wrote.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Indexed fields can play a role in lispy, but the KO Explorer doesn't deal with lispy. Its purpose is knowledge objects, ie search time things. Tags, Eventtypes, calculated fields, field aliases, stuff like that... things that cause normalizedSearch to explode in bad cases. Indexed fields don't matter there.

A key word in that first sentence is can. Many indexed fields will work even if lispy treats them as search time, ie if field=value is translated to a lispy of [ AND value ] instead of [ AND field::value ] (see linked conf talk for more depth and breadth). As a result, fields.conf will only yield a small fraction of indexed fields.

I've added two more things to consider to the original answer.

0 Karma

charlesmeo
Explorer

Accepted the answer, which is of course correct but still leaves open the substance of your Idea.

I had forgotten about INDEXED_EXTRACTIONS. Of course fields.conf won't give you anywhere near complete coverage. You would potentially need to go out to all the forwarders and see what they're up to as well. Alternatively, this is a good use case for splunking your git repo of splunk configurations I would think.

But in general it seems, no magic bullets for this. C'est la vie..

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

TBH, walklex is close to a magic bullet 😛

0 Karma

charlesmeo
Explorer

I have voted for the Idea as suggested.

But there's still something here I don't get. I thought indexed fields DID play a role in the lispy. If you happen to know you are dealing with one, you can use the :: operator instead of =, and the lispy is different.
But since there is no easy way to figure out whether a field is indexed or not (for custom fields) you don't have this option.

Perhaps a future version of KO Explorer could at least inspect fields.conf and let the user know if a field might be indexed? Or is this not worth doing?
I am finding lispy quite confusing. I am a splunker of long standing but haven't looked under this particular rock before now.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...