Hello everyone,
I have a two indexers IDX 01, IDX 02 in a cluster connected to a search head cluster.
What I observed is IDX01 is having high CPU usage (like 100 %) many times in a day, but IDX02 does not have any alerts.
When I looked into DMC, IDX01 has more scheduled searches running on it whereas IDX02 shows less scheduled searches running on it.
I can clearly see that searches are running only on IDX01 but not on IDX02.
What can the problem be?
Cluster Master shows the indexer's health is fine.
How can I troubleshoot.........any suggestions, please.
what I see is DMC under these sections :
Median CPU Usage by Process Class
Maximum Search Concurrency
Maximum Resource Usage of Searches
more above all sections ............I can clearly see the IDX01 have high usage when compared to IDX02
Just guessing here: could it be you have a multi site cluster? If so check this http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisitesearchaffinity - this could explain why only one indexer gets hit by your SHC.
But again, this question is missing the level of detail that is needed to be able to help ...
cheers, MuS
You'd almost think this question is about the same issue, perhaps colleagues? https://answers.splunk.com/answers/684277/why-are-my-searches-only-hitting-one-indexer-in-a.html
And there it is indeed mentioned that it is a multi site set up.
Poor data distribution perhaps? How have you configured your forwarders to spread the data over your indexers? Also: how much data is on each indexer for the relevant indexes that those searches are hitting? If for some reason most/all data is flowing to IDX1, that could easily explain why that indexer is much more busy.
You've also made sure that both indexers are search peers of the search head and search head is successfully able to connect to both for running searches?
Both have same disk usage, almost same number of buckets ,if it's not hitting another indexer that is a problem , but it is hitting but hitting less when compared to IDX01