I recently inherited this splunk system, and I am gradually working out how it is set up. When running a search yesterday, I noticed something. We have 10 indexers, 5 at site1, 5 at site2. We have 4 ...
See more...
I recently inherited this splunk system, and I am gradually working out how it is set up. When running a search yesterday, I noticed something. We have 10 indexers, 5 at site1, 5 at site2. We have 4 search heads, all assigned to Site0. When inspecting my search job, I saw that my results were only pulled from a single site's peers, not from both. Here are some pictures to explain:
My rep factor tells me I should have 2 copies at each site.
My search factor tells me I should have 2 searchable copies at each site.
This would imply that when I run a search across my 10 indexers, it would be pulling data from both sites.
So then i run a search on a specific index, and I see this:
I expected to see data pulled equally from both sites, but I see Site k is left completely alone.
Even if a single indexer was the ingest point for all the data, it would still be scattered across the 10 indexers as it worked to meet the replication/search factors. There is no reason everything should be stuck on one site.
Am I way off base here, or is something configured wrong?