Deployment Architecture

Why are my searches hitting only one indexer in a cluster ??

ramarcsight
Explorer

Hello everyone,
I have a two indexers IDX 01, IDX 02 in a cluster connected to a search head cluster.

What I observed is IDX01 is having high CPU usage (like 100 %) many times in a day, but IDX02 does not have any alerts.

When I looked into DMC, IDX01 has more scheduled searches running on it whereas IDX02 shows less scheduled searches running on it.

I can clearly see that searches are running only on IDX01 but not on IDX02.

What can the problem be?

Cluster Master shows the indexer's health is fine.

How can I troubleshoot.........any suggestions, please.

what I see is DMC under these sections :

Median CPU Usage by Process Class
Maximum Search Concurrency
Maximum Resource Usage of Searches

more above all sections ............I can clearly see the IDX01 have high usage when compared to IDX02

0 Karma

MuS
SplunkTrust
SplunkTrust

Just guessing here: could it be you have a multi site cluster? If so check this http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisitesearchaffinity - this could explain why only one indexer gets hit by your SHC.

But again, this question is missing the level of detail that is needed to be able to help ...

cheers, MuS

0 Karma

FrankVl
Ultra Champion

You'd almost think this question is about the same issue, perhaps colleagues? https://answers.splunk.com/answers/684277/why-are-my-searches-only-hitting-one-indexer-in-a.html

And there it is indeed mentioned that it is a multi site set up.

0 Karma

FrankVl
Ultra Champion

Poor data distribution perhaps? How have you configured your forwarders to spread the data over your indexers? Also: how much data is on each indexer for the relevant indexes that those searches are hitting? If for some reason most/all data is flowing to IDX1, that could easily explain why that indexer is much more busy.

You've also made sure that both indexers are search peers of the search head and search head is successfully able to connect to both for running searches?

0 Karma

ramarcsight
Explorer

Both have same disk usage, almost same number of buckets ,if it's not hitting another indexer that is a problem , but it is hitting but hitting less when compared to IDX01

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!