Automatic upvote for question regarding multi-tenant deployments... 🙂
We're in a pretty similar situation as yours in that we have totally different "customers" with completely separate environments, probably even more so than in your case, so even IP addresses, hostnames, critical assets, privileged users etc could theoretically have the same name at two different customers but still be completely different things. Yet we run most things from central search heads.
Unfortunately I don't have a "silver bullet" for you in how to deal with this - I can only verify that multi-tenancy with Splunk IS a pain and the more you look at it the more it seems like doing multi-tenancy is really bending how you're supposed to use Splunk.
In the case of apps that explicitly use a certain index - yes, you will have to rewrite the app to take that away, and instead set the app to not care about the index at all, using each user or role's index access to define which data will be visible to which user. For users with access to several customers' data (like your own admin user) you'd have to rewrite your app to include something like dropdowns for choosing the correct index.
With more and more apps making use of Splunk's high performance datastore via tsidx files this becomes tricky as well, as these have separate permission models - I'm actually unsure if Splunk is enable to enforce access to these at all right now.
It gets hairier if you want to run environment-dependent lookups too, but I don't know if you're trying to do this right now. In short, this becomes a problem if you want to use the same lookup for different environments and have different environments return different data.
Sorry for not giving you better news...
... View more