Why are my searches only hitting one Indexer in a cluster ?

[sairam1444]

Hello everyone.
I have a multisite Indexer cluster. 2 IDX (IDX01, IDX02) and CM
2 SH with a deployer and a VIP to SH cluster

site 1
SH1
IDX01
CM

site2
SH2
IDX02

search affinity is enabled.

For example on SH1 if I run:

|tstats c where splunk_server=IDX02 earliest=-24h by index

I don't see any results. But I get results when I use

splunk_server=IDX01

as both SH1 and IDX01 are on the same site = site1

Again on SH2 if I run:

|tstats c where splunk_server=IDX01 earliest=-24h by index

I don't see any results. But I get results when I use

splunk_server=IDX02

as both SH2 and IDX02 are on the same site = site2
In the same way, on CM

|tstats c where splunk_server=IDX02 earliest=-24h by index

I don't see any results but I get results when I use

splunk_server=IDX01

as both CM and IDX01 are in same site = site1.

My Problem :

IDX01 has High CPU usage alerts and has been almost hitting 100% for a long time.

When I look in DMC
under DMC
Median CPU Usage by Process Class
Maximum Search Concurrency
Maximum Resource Usage of Searches

it clearly shows that searches are hitting this IDX 01 then other IDX02.

My doubts :
1. Is search affinity playing a role here?
2. If searches are more dispatching from SH1, is there a chance that more searches are running on IDX01 and causing high cpu problems?

Please help me. Thank you! (edited)

[mstjohn_splunk]

Hi @sairam1444,

Did @harsmarvania57 's answer help you solve your problem? If so, please approve their answer below. But, if you still are having an issue, go ahead and provide us with some more information on your problem. That way, the community knows that you still need help.

Thanks for posting!

[harsmarvania57]

Hi @sairam1444,

Please find below answers:
1.) Yes, search affinity is playing role here.

If you look at documentation http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Howclusteredsearchworks#Search_locally_in_... , it clearly say that In a multisite cluster, you typically put search heads on each site. This allows you to take advantage of search affinity. In search affinity, searches normally run across only peers on the same site as the requesting search head. Search affinity is always enabled with multisite clusters
2.) When search affinity is enabled, searches from SH will run locally on that site which means SH1 will run all searches against IDX01 (Because both SH1 and IDX01 belongs to same site). To understand how searches will run in multisite cluster with search affinity enabled please read documentation on link which I have provided in point 1.

I hope this helps.

Thanks,
Harshil

[gjanders]

If you want the search heads to balance their searches across site1 and 2, site0 is the setting used in search head clusters, this allows them to search indexers from either site, however this may not make sense depending on your setup.

[esalesapns2]

You have to set "site = site0" in the "[clustering]" and "[clustermaster:..]" stanzas to get the search head to search across all sites.