Deployment Architecture

Why are my searches hitting only one indexer in a cluster ??

ramarcsight
Explorer

Hello everyone,
I have a two indexers IDX 01, IDX 02 in a cluster connected to a search head cluster.

What I observed is IDX01 is having high CPU usage (like 100 %) many times in a day, but IDX02 does not have any alerts.

When I looked into DMC, IDX01 has more scheduled searches running on it whereas IDX02 shows less scheduled searches running on it.

I can clearly see that searches are running only on IDX01 but not on IDX02.

What can the problem be?

Cluster Master shows the indexer's health is fine.

How can I troubleshoot.........any suggestions, please.

what I see is DMC under these sections :

Median CPU Usage by Process Class
Maximum Search Concurrency
Maximum Resource Usage of Searches

more above all sections ............I can clearly see the IDX01 have high usage when compared to IDX02

0 Karma

MuS
Legend

Just guessing here: could it be you have a multi site cluster? If so check this http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisitesearchaffinity - this could explain why only one indexer gets hit by your SHC.

But again, this question is missing the level of detail that is needed to be able to help ...

cheers, MuS

0 Karma

FrankVl
Ultra Champion

You'd almost think this question is about the same issue, perhaps colleagues? https://answers.splunk.com/answers/684277/why-are-my-searches-only-hitting-one-indexer-in-a.html

And there it is indeed mentioned that it is a multi site set up.

0 Karma

FrankVl
Ultra Champion

Poor data distribution perhaps? How have you configured your forwarders to spread the data over your indexers? Also: how much data is on each indexer for the relevant indexes that those searches are hitting? If for some reason most/all data is flowing to IDX1, that could easily explain why that indexer is much more busy.

You've also made sure that both indexers are search peers of the search head and search head is successfully able to connect to both for running searches?

0 Karma

ramarcsight
Explorer

Both have same disk usage, almost same number of buckets ,if it's not hitting another indexer that is a problem , but it is hitting but hitting less when compared to IDX01

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...