Splunk Enterprise

Workload Management -- how to de-prioritize searches that don't include index specification

wryanthomas
Contributor

We had someone bring down a search head cluster member the other day; the user had inadvertently used "ndex=myindex" instead of "index=myindex".  I.e., a typo that led Splunk to try to match against all default indexes for the user.

I'd like to be able to create a Workload Management rule that de-prioritizes such searches (i.e., whether the omission of index is intentional or not).

Obviously, matching on "index=*" doesn't work for this scenario.  And 'NOT index' would match instances of index way more than the intended instance at the beginning of the search. (Also recognizing that index specification is not always "index=" it can also be "index IN(".) Further, many legit searches don't include 'index' -- e.g., tstats, inputlookup, pivot, rest, makeresults, etc.  Additionally, folks are using macros which hide index designations.

Upshot: How can I create a Workload Management rule that would prevent the use of ndex (instead of index) from bringing down an SHC member?

Labels (1)
0 Karma

wryanthomas
Contributor

This definitely would improve the situation. The current defaults are "appropriate", but because this person has (appropriately) wide access to various indexes relevant to his team (in this case, email infrastructure), it's just too wide for a mistake like this.  So, changing our system-wide approach to index defaults... I like very much.  Thanks for this suggestion.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

How about addressing the root cause of the problem?  Remove all default indexes from each role.  Not only will this prevent a wide-ranging search from killing a SH, but it will force users to include explicit "index=" in their searches, which should make them more efficient.

As you already know, there's no way to trigger a rule based on the absence of a thing and, even if there were, the absence of that thing doesn't mean the thing isn't specified someplace else.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...