What is the best way to combine a License Master, Distributed Management Console, Deployment Server, and a SHC Deployer on 2 dedicated Splunk servers?

[mfrost8]

Hi. We're in the process of redoing a lot of our existing Splunk infrastructure. Currently we have a license master that co-resides in the same instance as an indexer. We also have a deployment server that runs as a separate instance on a separate indexer server. (We learned early on that deployment servers don't play nice with streams of events coming in on the same port. We now have a few hundred deployment clients.)

In the new infrastructure, we want to move those 2 functions off to dedicated servers. Or at least semi-dedicated. Or at least not search heads or indexers. However, since we're also going to convert to search head clustering we'll need to run a "deploy server". We're also going to use this opportunity to setup the distributed management console.

This means we have 4 functions

• Deployment server
• Deployer for clustered search heads
• License Master
• Distributed management console

and we have 2 (new) servers to do it on -- 2 servers that exist for no purpose other than to provide these 4 functions.

My problem is, I'm not exactly sure how best to combine them. As I'm pretty sure that the deployment server won't work well with any of the other 3 functions, it seems reasonable to make a single instance with just that. It would be kind of nice to avoid having to have multiple instances on a single server as that's a minor pain to manage (multiple $SPLUNK_HOME's, etc) and I thought it might be possible to run the deployment server on one host and then all of the remaining 3 (deploy server, license master and DMC) in a single instance on the other host. I don't know if that's reasonable though.

Absolute worst case, I can run 2 instances on each of the 2 servers and provide one instance per function, but I'd rather not if I don't have to.

Does anyone have any experience and/or opinions about how best to combine these functions across 2 hosts with or without multiple instances on them?

Thanks!

[hexx · Accepted solution]

In your situation, I would recommend to set up two instances - one per server - with the following role distribution:

1) One instance that takes on the roles of:

• Cluster Master
• Search-Head Cluster App Deployer
• Distributed Management Console

2) One instance dedicated to the role of Deployment Server

[mfrost8]

Thanks, this is where is the way I was leaning.

[MuS]

Hi mfrost8,

I don't know your problems with the deployment server; but I have and had a lot of costumers running all the functions your talking about on one single instance installed on a small VM without problems. Take a look at this wiki http://wiki.splunk.com/Things_I_wish_I_knew_then to read more about Architecture and Deployment related hints.

Hope this helps ...

cheers, MuS

[mfrost8]

Thanks, Mus.

Perhaps things have changed, but when we started with the deployment server (Splunk 4.x days), Splunk indicated that if were going beyond about 50 or so clients you'd start having problems if you were sharing it with something like an indexer. Things worked, but we got a lot of errors in splunkd.log where the two components were clashing in the same instance and trying to use/overuse 8089. Perhaps Splunk has made this better, but in the DS wiki (http://wiki.splunk.com/Deploy:DeploymentServer) I see splitting this into its own instance for 30+ as a recommendation.

The requirement in the docs for the Deployer also seem to indicate that it shouldn't co-reside with anything else that's too busy (i.e. a low volume deployment server is OK and so is an indexer cluster master, but otherwise a dedicated instance): http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/SHCsystemrequirements#Deployer_requirem...

I don't fine any specific requirements for the license master which leads me to believe it is a "good neighbor".

Per the DMC recommendations, http://docs.splunk.com/Documentation/Splunk/6.2.5/Admin/ConfiguretheMonitoringConsole#Which_instance... it seems as if this should probably be a separate instance as well (we don't do indexer clustering).

This leads me to believe that we should probably have 3 instances on 2 servers where the license master and DMC co-reside on one of those instances. But that's a bit of a guess based on the docs.

Thanks for the link reference. I've seen this before and found it to be a very useful resource.

[hexx]

Splunk indicated that if were going beyond about 50 or so clients you'd start having problems if you were sharing it with something like an indexer

This is still valid today.