Splunk Search

Why are my searches only hitting one Indexer in a cluster ?


Hello everyone.
I have a multisite Indexer cluster. 2 IDX (IDX01, IDX02) and CM
2 SH with a deployer and a VIP to SH cluster

site 1


search affinity is enabled.

For example on SH1 if I run:

|tstats c where splunk_server=IDX02 earliest=-24h by index

I don't see any results. But I get results when I use


as both SH1 and IDX01 are on the same site = site1

Again on SH2 if I run:

|tstats c where splunk_server=IDX01 earliest=-24h by index

I don't see any results. But I get results when I use


as both SH2 and IDX02 are on the same site = site2
In the same way, on CM

|tstats c where splunk_server=IDX02 earliest=-24h by index

I don't see any results but I get results when I use


as both CM and IDX01 are in same site = site1.

My Problem :

IDX01 has High CPU usage alerts and has been almost hitting 100% for a long time.

When I look in DMC
under DMC
Median CPU Usage by Process Class
Maximum Search Concurrency
Maximum Resource Usage of Searches

it clearly shows that searches are hitting this IDX 01 then other IDX02.

My doubts :
1. Is search affinity playing a role here?
2. If searches are more dispatching from SH1, is there a chance that more searches are running on IDX01 and causing high cpu problems?

Please help me. Thank you! (edited)

0 Karma

Splunk Employee
Splunk Employee

Hi @sairam1444,

Did @harsmarvania57 's answer help you solve your problem? If so, please approve their answer below. But, if you still are having an issue, go ahead and provide us with some more information on your problem. That way, the community knows that you still need help.

Thanks for posting!

0 Karma


Hi @sairam1444,

Please find below answers:
1.) Yes, search affinity is playing role here.

If you look at documentation http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Howclusteredsearchworks#Search_locally_in_... , it clearly say that In a multisite cluster, you typically put search heads on each site. This allows you to take advantage of search affinity. In search affinity, searches normally run across only peers on the same site as the requesting search head. Search affinity is always enabled with multisite clusters
2.) When search affinity is enabled, searches from SH will run locally on that site which means SH1 will run all searches against IDX01 (Because both SH1 and IDX01 belongs to same site). To understand how searches will run in multisite cluster with search affinity enabled please read documentation on link which I have provided in point 1.

I hope this helps.


0 Karma


If you want the search heads to balance their searches across site1 and 2, site0 is the setting used in search head clusters, this allows them to search indexers from either site, however this may not make sense depending on your setup.

0 Karma

Path Finder

You have to set "site = site0" in the "[clustering]" and "[clustermaster:..]" stanzas to get the search head to search across all sites.

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...