Questions on Migration From All-in-one Stand-alone Deployment

[gearmstrong]

All,

In process of trying to figure out architecture/hardware requirements for upgrade our current all-in-one deployment. 

Current Architecture
Azure Windows 2016 56GB RAM, 16 vCPUs with 2 TB Data Drive (7500 Max IOPs) Data drive (SSD) @98% capacity.

Roles - Indexer, Deployment Server, Search Head Web Server

Current Data Ingestion Rate is 123.97 GB/Day

Deployment Goals

Use as few Azure Windows 2019 Server(s) as possible with minimum hardware to reduce costs.  Increase 2TB Drive to 4TB GPT Drive (or delete older unnecessary data and stay at 2TB).  Preserve existing knowledge objects, index data and increase performance.

Questions

1) Can the Search Head, Deployment Server and First Indexer be on its own server (as it is now) and Second Indexer be on a  second server (by itself)?  In other words require only two servers to be utilized or do I have to have three servers as most documents depict?

2) If deploying a second Indexer does it have to have the same size disk as current indexer?  And if so, how is current data on almost full 2TB Drive dealt with now that there is a secondary indexer?   

3) Anyone have experience with or know the feasibility of leveraging Search Head (or other components) in a Virtual (Azure) Containerized usage?  Again, in order to reduce Hardware Costs.

4) Any other suggestions appreciated to aid in cost reduction.

Best regards,

Greg

[richgalloway]

1) In a distributed environment, search heads and indexers are on separate [virtual] machines.  You can have two servers, but one of them will be a SH and the other will be an indexer.  The usual procedure in this migration is to make the standalone server become the indexer so no data migration is needed.

2) Non-clustered indexers do not need to have the same storage.  They must have enough to hold whatever data they will ingest over the expected retention period, plus about 10-15% for overhead.  Keeping all indexers the same will make for easier management.

3) Yes, I'm sure someone does.  I am not one of those people, however.

4) You can reduce the number of CPUs or the amount of memory to save money as long as performance meets your expectations.  Understand, however, that if you ask Splunk for support they may ask you to bring any under-provisioned servers to minimum specs before assisting you.