Workload Management -- how to de-prioritize searches that don't include index specification

[wryanthomas]

We had someone bring down a search head cluster member the other day; the user had inadvertently used "ndex=myindex" instead of "index=myindex".  I.e., a typo that led Splunk to try to match against all default indexes for the user.

I'd like to be able to create a Workload Management rule that de-prioritizes such searches (i.e., whether the omission of index is intentional or not).

Obviously, matching on "index=*" doesn't work for this scenario.  And 'NOT index' would match instances of index way more than the intended instance at the beginning of the search. (Also recognizing that index specification is not always "index=" it can also be "index IN(".) Further, many legit searches don't include 'index' -- e.g., tstats, inputlookup, pivot, rest, makeresults, etc.  Additionally, folks are using macros which hide index designations.

Upshot: How can I create a Workload Management rule that would prevent the use of ndex (instead of index) from bringing down an SHC member?

[wryanthomas]

This definitely would improve the situation. The current defaults are "appropriate", but because this person has (appropriately) wide access to various indexes relevant to his team (in this case, email infrastructure), it's just too wide for a mistake like this.  So, changing our system-wide approach to index defaults... I like very much.  Thanks for this suggestion.

[richgalloway]

How about addressing the root cause of the problem?  Remove all default indexes from each role.  Not only will this prevent a wide-ranging search from killing a SH, but it will force users to include explicit "index=" in their searches, which should make them more efficient.

As you already know, there's no way to trigger a rule based on the absence of a thing and, even if there were, the absence of that thing doesn't mean the thing isn't specified someplace else.