This question cannot be credibly answered without knowing answers to some key questions:
What is your expected daily data ingest?
What is your data retention policy, i.e. how many days of log data do you need to keep?
What is your expected concurrent search volume?
What HDD drives do these boxes have? 5k/7k/10k rpm?
Are you planning on using Premium Apps, like Splunk App for Enterprise Security?
While possible, we typically do not recommend running multiple Splunk instances on the same physical server anymore, since our more recent product versions feature multiple parallelization settings that can be used to exploit available hardware better. Also, administering this, especially in a clustered environment, comes with one or three headaches (if the hardware dies, you lose two peers, different IPs/different ports, etc.). Also, running Splunk Enterprise in Docker is not supported at this point in time, so probably not your best choice for production unless you don't expect Splunk support to help you when issues arise.
Your key constraint is likely going to be available disk IOPS (specifically random read). Even if your drives are 10k drives, a RAID10 array with 10 drives would give you around 900 IOPS max (at an assumed 60:40 R/W rate). That's about what you need for a single Splunk indexer instance. Splunk has a very unique and demanding I/O profile with constant streaming writes (indexing) and completely unpredictable and often random read operations. Any disk subsystem that provides less than 1000 IOPS is likely not do well with multiple Splunk instances, whether they are containerized or not.
If you have a data high ingest rate with a relatively low search workload, you may be able to configure two ingestion pipelines and process upwards of 600GB/day per indexer (indexing is largely a streaming write operation). Two indexing pipelines at full capacity will utilize between 8 and 10 cores, so you still have a ton of cores left for search processes (remember, indexers are also search peers and search is what stresses your I/O most).
If you have a relatively low data ingest rate, but a very high concurrent search volume, you will be able to utilize cores for searches as long as your disk can keep up. A lot depends on the types of searches as well. Dense searches that require a lot of buckets to be retrieved from disk and unzipped are taxing CPU cores more than sparse searches that can use index files to filter down the number of buckets we need to open to satisfy the search.
What will help you a lot here is the amount of RAM your system has. The more RAM, the bigger your OS file system cache and the higher the likelihood that searches over recent data can be satisfied out of the cache vs. having to bother the disk.
To summarize: While it is these servers have a ton of memory and CPU resources, the disk subsystem is likely your limiting factor. As soon as you hit I/O waits, your performance will take a nose dive, no matter how many CPU cores are ready to do work.
Any workaround (virtualization, multiple instances, containers) is dependent on that shared resource and likely not going to improve your overall system performance.
If your servers had SSDs instead of "spinning rust", the conversation would be completely different.
I hope this helps to shed some light onto the question you raised.
... View more
