Splunk Architectural Questions - DMC, CM, LM, DS, SHCD?

[quihong]

First some quick background, I have new but fairly complex Splunk Enterpirse ES environment with HA Index Clustering and two Search Head Cluster (one for ES and one for core splunk). All servers are physical with 20 cores and 32GB.

• Can I have more than one DMC? Currently we have the DMC on our Cluster Master (CM) and License Master (LM). I want to move the DMC to my Deployment Server + ES Seach Head Cluster Deployer. Any issues with that plan? So server #1 would have DMC+DS+ES SHCD. Server #2 will have CM + LM.
• For server #2 (CM+LM), I would like to use it to host Splunk Apps such as DBConnect, Stream, eStreamer. I'm not convince this is a good idea, but wanted to get the community opinion on this.

My thought is that I'm always using the DS and DMC and would prefer them to be on the same server. Secondly, I need a place to install Splunk Apps such as DBConnect and eStreamer that needs to pull data into Splunk. If I move the DMC off the CM+LM, I feel that the server is really under utilize just acting as a CM+LM. My other thought is to move the CM+LM to virtual if it's an issue.

I appreciate any thoughts around this. Thank you.

[esix_splunk]

So a few thoughts on this...

You can have multiple DMCs, but you need to understand the requirements. Typically these are run on the CM because the CM is aware of all the members of the cluster and DMC has full functionality because of this. If you offload this to a different server, you will need to add all of the Peers on that host in order for the DMC to query them via REST. This works well and is perfectly doable.

Running DMC off the deployer and DS is not a problem assuming the resources are available for both of the other components. Depending on the number of clients the DS manages, you need to watch your disk I/o.. Deployer isnt a heavily used function, but disk and network should be considered also for this instance..

And for the CM and LM, virtual is the way to go. These require very little resources, and in regards for HA / DR, having these as virtual is beneficial for minimizing your down time. And you could reclaim that server into your IDX tier or one of your SHC..

[Richfez]

Comment because it's only a little part of the answer - maybe if we all add our two cents it'll add up to a dollar someday!

In my environment, which is ES but not ES SHC.

Box 1: CM and DMC, virtual. Makes the Indexer pieces of DMC work easier since it's on the CM.

Box 2: DS and LM, virtual. LM is such a light load, DS requires at least a modicum of beef. But not too much in my environment.

Box 3: Virtual HF I use for DBConnect, Stream and eStreamer, again a VM.

Box 4: I have an ad-hoc (non ES) SH that's a virtual.

Boxes 5, 6 and 7: Indexers and the ES SH which are all physical.

We will be adding a pair (perhaps 3) more indexers in a second cluster site and probably a couple of physical search heads as our next "enhancements".

So, again, it's not quite answering your question, but I figure it at least gives you one more data point to think about.