How to set up Splunk to monitor logs and configure distributed search across 4 different development environments (Dev > Tst > Stg > Prod) in AWS

[larryliang]

We have four AWS accounts to host different development environments:
Dev -> Tst -> Stg -> Prod

Requirements:
We want to set up Splunk to index/monitor logs across all accounts and provide a single endpoint for searching using GUI.

We are thinking about doing the following:
- setting up dedicated indexer for each account (which individual forwarders communicate to)
- Configure distributed search (search head instance) to search across all indexers to provide an aggregated view across all accounts.
- Each indexers will be set up with the same internal DNS name across all accounts. In this case we can bake the forwarder with the same configuration into AMI and promote that AMI across accounts.
- As I understand, the search head needs to have network access to the individual indexers (search peers). We're thinking of using VPC peering. We do not need to worry about cross region connectivity as we will be using only one region across multiple accounts.

Can someone from Splunk please provide some inputs to this proposed design and comment on if this is the endorsed way of using Splunk with AWS?

Thanks,

[s2_splunk]

Running Splunk on AWS is no different from deploying a similar architecture on-prem; there is really no endorsement of any kind. 😉
As long as you have network connectivity from your search head to your indexers AND your SH can distinguish indexers in the distributed search setup, this should work fine. Not sure how that would work with all indexers having the same DNS name, but I am not an AWS expert by any means.