We are deploying hosting to various organizations in our "company". Each organization in our company may consist of numerous apps (100+ and 5,000+ employees). Our intention is to provide these organizations with an AWS Account, which would be consumed into our AWS deployment infrastructure. Each VPC/AWS Account will hold various apps and types of data.
My query is should I be looking to treat each of these accounts as a separate Splunk site (Multisite deployment) and searches are local to that VPC?
Or instead, should I route log traffic to a separate "master" VPC deployment as a larger clustered deployment?
Qty of apps/users is a sliding scale as our project grows. Today it's 1 app only - next year it could be 100 per organization.
I had initially intended to route logs securely to a single Splunk Enterprise cluster made up of say 1 search head & 2-3 indexes and grow out as demand grows. But on reading about multisite, there seems to be quite a lot of benefits. However, suspect costs saved via VPC traffic cost vs oodles of nodes/indexers/search heads per AWS account will be lost.
Or would it be better to view Multisite as a longer term strategy deployment of Splunk — as the project grows etc.. — and then migrate deployment at a later date?
Security - what are the security aspects of the data, access controls, etc. If you are concerned about this, there should be no mixing of data across accounts. You can use VPC's to segregate data as well, depending on how paranoid you are about the boundaries and how good you are at AWS management.
Data resiliency - do you want to preserve your data irrespective of what happens with the underlying AWS services? If so, create multiple VPC's in different AZ's and put one Splunk site in each.
I wouldn't consider separate accounts as separate Splunk sites. If you're going multisite for redundancy within a cluster, then set up separate VPC's in different AWS AZ's or regions. Splunk sites don't have to map to multiple VPCs though if you don't want to route traffic and pay for that, but you won't gain from the segregation of copies of data in different locations. Again, depends on how paranoid you are about your data being safe (if an AZ goes down, you still have access to your data on the other site if you split your sites between AZs).
The cost of inter-VPC traffic will be the same as inter-account traffic, so you're not going to save anything on that.
1 search head and 2-3 indexers doesn't sound like a lot for 5000+ employees, but of course depends on the amount of data being collected and who's actually using Splunk.
If everyone in this "company" should have access to data from each organization, then going centralized would make sense. It really depends on the details around what the data is, who should have access to it, who shouldn't have access to it, and how secure you want that data to be (both in terms of resiliency and access security). There's a lot of info you didn't provide that would lead the answer into one direction rather than another...