Knowledge Management

Is it best practice to constrain searches by index name?


(That is: in a Splunk environment where data is stored in multiple indexes.)

I'm almost certain that the answer is "Yes". I've seen what could be interpreted as "Yes" answers inline in other questions. For example, this statement in 2016 by @somesoni2 :

it's always advisable, from performance perspective, that you specify the index you need to search, to cut down the number of indexes/buckets to be searches.

However, I wanted a specific question for this, preferably with affirmative answers from Splunk or prominent Splunk users, that I could refer to (and point other people to).

I'm asking this question because I have learned that, as a fix for failing the AppInspect check check_indexes_conf_does_not_exist, a fellow app developer both (a) removed indexes.conf from their app and (this is specifically why I'm asking this question) (b) removed all index=... constraints from the searches in the app. So the app now constrains its searches by sourcetype, but not index.

I suspect this means that Splunk will search all indexes that the user is allowed to search. Not ideal, from a performance perspective. Unless—this is just a thought bubble; I have no evidence for this in reality—Splunk somehow keeps track of which sourcetypes are stored in which indexes: when a search specifies a sourcetype, Splunk implicitly constrains the search to only those indexes that it knows contains those sourcetypes. I doubt that this is true, but I wanted to at least raise this as a possibility; preferably, for someone to explicitly state as false in their answer.


Filter early, filer often

Any time it's possible, use the form index=ndx sourcetype=srctp ...

Using merely index=ndx ..., and NOT including sourcetype=srctp will still allow Splunk to try to look in EVERY sourcetype (even the 291 that aren't in that index)

0 Karma

Ultra Champion

You don't get much more prominent than @somesoni2! 🙂

But the short answer is YES.

The faster (earlier in the search) you can exclude events, the faster your search will ultimately execute.
Since indexes 'tend' to be the largest aggregator of events, by specifying 1 index out of 10 potential indexes on your system, you could have just excluded upto 90% of events that would otherwise have to be searched.

So yes, ideally, you would specify indexes to make your searches more efficient.

However, changes that were made to the Splunkbase App-Inspect process somewhat goes against this approach.

The reason the App-Inspect changes were made, is that in some cases users were installing Splunk apps which contained an indexes.conf which defined a custom index for that app. In those cases, users would install the app, collect some data and perhaps at a later date they would remove the app. As you might expect, removing the app, removes indexes.conf, which removes the index and the data within it.

The App-Inspect process was changed so that apps must no longer contain an indexes.conf to protect against this outcome.

Since you (as an app creator) can't now specify the index you want to use, you may need to rely on the user to specify it, but that means you can't presuppose what index name they would choose.

There is however a suitable work around.
In your app create a macro like 'my_app_index_macro' define this macro as index=*

Then in your apps searches include that as part of the search string:
'my_app_index_macro' sourcetype=my_app_sourcetype ..whatever|whatever
(note the backticks are wrong in my examples - make sure you use the correct one)

This effectively turns your search into:
index=* sourcetype=my_app_sourcetype ..whatever|whatever
Not ideal, but no worse than sourcetype=my_app_sourcetype ..whatever|whatever

To get the optimisation to have any effect, in your apps documentation instruct the user (or implement a configuration step in your apps setup.xml) to modify the macro to suite their environment and replace index=* with index they have chosen to use index=my_chosen_index.

If my comment helps, please give it a thumbs up!
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!