Deployment Architecture

Run saved searches on a single indexer

L_Petch
Explorer

Hello,

 

A client went mad on how many saved searches they require and wont get rid of them. Due to this it is hammering rw on the indexers to the point the indexers can cope and remove themselves from the cluster and then re adds which and more resource strain.
adding more indexers isnt an option

The current setup is 3vm multisite search head cluster and a 4vm multisite indexer cluster.

 

As they only require 3rf and 3sf i am wondering if there is a way to use only 1SH and 1 Indexer for all saved searches to run so that the load doesnt affect the other 3 indexers?

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @L_Petch ,

you can use the Monitoring Console App to have this information.

Ciao.

Giuseppe

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Saved searches are scheduled across the whole search head cluster. (with some additional conditions like a cluster member being in detention). That's what search head is for. Also - limiting searches to just one SH would inevitably lead to delayed/skipped searches. It won't solve the performance issue.

Even if you have multiple indexers holding the same bucket, the indexers holding primary copies respond with results from those primaries - it's by design and lets you distribute the search. Even if you had a possibility to get results from just one indexer, there would be no guarantee that you'd get all events from given time range because with sf=rf=3 and 4 indexers you'd still probably hit (actually would _not_ hit) some buckets which are not present at that chosen indexer.

So your idea is not a very good one.

You can use site affinity to force search heads to use only one site. But again - especially if you already have performance problems - that's counterproductive.

And from experience - it's often not the _number_ of searches but _how_ they're written.

Get Updates on the Splunk Community!

Buttercup Games: Further Dashboarding Techniques

Hello! We are excited to kick off a new series of blogs from SplunkTrust member ITWhisperer, who demonstrates ...

Message Parsing in SOCK

Introduction This blog post is part of an ongoing series on SOCK enablement. In this blog post, I will write ...

Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a ...