So I am in somewhat of a fun situation where we have multiple instances of Splunk installed each with their own index clusters and search head clusters. I know you can configure search heads to search multiple index clusters, but not all my index clusters have the "same" data in a named index (mainly, index=main). So what I was wondering is if I install all the apps from all of the instances onto the search head cluster that is configured to connect to all index clusters, I can tell those apps to only look to the appropriate index cluster that has the data they want? I think I could accomplish this with sites maybe if I can tie an app to a site. But I am not finding either index cluster or site configurations for individual apps. The point would be to provide a single place to login and be able to see all the splunk data and to eventually retire the now extraneous search head clusters without the apps having to search multiple "main" indexes in clusters that don't have the data they are looking for.
While there is no feature to really easily do this, you can use search groups and macros to do this...
These do require use training and understanding when the craft their searches..
Above being said, in large scale deployment planning there are some points you can take into account for..
Do note that in the latest versions of Splunk, we don't distribute searches to clusters where the index names dont exist (the cm will report this back in SH.) So searching index=security* would be similar to searching index=security* splunk_server_group=securitycluster.
There's no such feature and I hope there never is. While I can see the benefit, I would not want to be the admin that has to keep track of what data is on each cluster so I can point newly-installed apps to the right place(s).
And how can we be sure app A that's currently just accessing data from cluster 1 can't benefit from data in cluster 2 or 3?
The only way I can think of to get the performance improvement you're thinking of is to have a different set of indexes on each cluster. Then, when an app needs to access data in a certain index the request will only go to the one cluster that has it.
This idea https://ideas.splunk.com/ideas/EID-I-103 exists to solve this. Please up vote if interested