We have been challenged to spin up a small Splunk Enterprise environment,
I would like to have three servers and cluster them all in an indexer cluster and search head cluster,
Server A-> Indexer and search head
Server B-> Indexer and search head
Server C -> Indexer and search head
Server D -> Cluster Master and deployer.
How would you be able to logically separate the server roles? Would anyone know of where I can find documentation on this subject?
Splunk recommends that you don't merge different roles in one instance, specially for clusters, so not sure if you'll find documentation on the same.
If you're aware of all the painful implications of running multiple roles on one machine, you should be able to stand up two instances per machine with entirely separate port ranges. Check server.conf, web.conf, inputs.conf, and outputs.conf for where to specify ports... maybe more.
Then you should be able to configure one of those instances as a search head cluster member, and the other instance as an indexer cluster peer.
That being said, don't.
You will likely have a hard time getting support from Splunk with this configuration. However, this might be possible if you run two instances of Splunk on each server, one indexer instance and one search head instance. This configuration might be supported, you'll definitely want to reach out to your sales engineer to make sure you're not getting yourself into something that you can't get out of later.
If virtualization is an option you might want to go that route.
Like stated already, this is not a recommended approach. You are talking about a "small Splunk environment". Can you say a bit more about what that means? What will your daily ingest volume be? What do you expect your search workload to be (scheduled searches, ad-hoc searches)? How do you intend to replicate data in your indexer cluster? What are your availability requirements?
And most importantly, what are the specs for the servers you will have available? Unless they are really beefy machines with 24 or more cores, you will likely not be successful with the approach you outline (besides the administrative pain and potential support issues). So yes, don't do it is good advice.
If you require both a SHC (minimum of 3 members) and an indexer cluster (minimum of 2 cluster peers), a supported deployment would require a minimum of six servers (3 SHs, 2 Indexers, 1 CM/Deployer). But if you do that, you need to also ensure that your indexers have enough cores to handle the potential search workload a 3-node SHC can generate, so likely you would need more indexers than just 2, unless you can get high core-count servers to use for your indexers.
If you want to share more details about what your environment needs to be able to do and what your HA requirements are, we would be better able to guide you towards a sound deployment architecture.