Deployment server planning

[hectorvp]

I've around 400 servers distributed over 2 datacenters.(Datacenters are physically close to each other and may have some dedicated LAN channels as well)

I'm supposed to control their UF using deployment server.(Linux based deployment server)

Should I use only 1 deployment server at a particular datacenter to manage UFs of other datacenter + his own data center.

Or shall I keep 2 Deployment servers one at each datacenter.

Their is a VPN connection between 2 datacenters DMZ where this deployment server will be hosted. And number of firewall punching would be same independent of UFs connecting to other data centers deployment server or same.

What should I prefer??

When we say dedicated deployment server in Splunk enterprise does it mean search head and indexing is disabled.

I need to have search head and indexer as well with deployment server to capture only internal UF logs for health checkup daily routines. Other logs would be forwarded to other dedicated indexers not in our control.

In above case having search head and indexer shall I consider the capacity of deployment server to manage only 50 clients?? As per Splunk docs....or  can I sufficiently handle 400 servers with single or double deployment server?

[thambisetty]

@hectorvp 

When we say dedicated deployment server in Splunk enterprise does it mean search head and indexing is disabled.

There is nothing to disable. the functionalities should not be used.

I need search head and indexer as well along with deployment server stacked in single virtual server. This indexer and search head would be needed to capture Splunk internal logs for routine health checkup.

you can do selective forwarding, you can keep _internal logs at deployment server level as you mentioned you don't have control on search head. but best practice is to monitor everything search head by collecting all logs to indexers.

As per Splunk docs it says having DS with search head should handle only upto 50 clients as a best practise.

I don't think you will have more than 10 alerts configured on deployment server with _internal logs.

Do I need to consider this as well, or will it work fine with having only 1 DS for 400 servers and this DS having search head & indexer???

Having 12 CPUs & 12 GB Memory will server your purpose of managing 400 servers and SH & Indexer.

[hectorvp]

Thanks @thambisetty .

How much serverss can I manage approximately with one deployment server ...if servers scale from 400 to 2000??

Considering I'm having deployment server and search head with at maximum 2 users using search head and indexer on same box for storing internal UF logs for health checkup.

I saw with some other answers they state of nearly 5000 servers, will it be same case in my scenario?

[thambisetty]

@hectorvp 

can you please share server specs?

[hectorvp]

@thambisetty 

Linux based system with 12GB RAM and 12 CPU and 500GB/1TB disk space.

Note: I've two region where servers are distributed but they are under same VPN.

I guess deployment server is si gle threaded right, what are your thoughts about it.

[thambisetty · Accepted solution]

Totally depends on how much app size you deploy to how many clients.

splunk says that deployment server With standard configurations (12 gb memory and 12 cpu with 64-bit) can manage upto 8k clients with dedicated deployment server functionality.

scaling up to manage 400-2000 clients with specs you have for deployment server might not be a problem. The problem is only when you index more data & having more concurrent searches running.

make sure you will not have many schedule searches and only _internal data is getting indexed locally.

I recommend phone homing interval to 300 seconds to increase performance of DS.

you can calculate the time taken to deploy app to client using below link.

https://docs.splunk.com/Documentation/Splunk/8.0.6/Updating/Calculatedeploymentserverperformance

[isoutamo]

Hi

with your node amount I prefer to use only one server in one location. DS don’t support any HA features at this time, so with two you must manually configure both of those and also keep those in sync. It’s much easier to use one and if needed cold backup (e.g. automatic rsync) if you think that you cannot have situation when DS can be down couple of hours. Normally this is not an issue as all UFs are working with local configurations as long as DS come up again.

r. Ismo

[hectorvp]

Hi @isoutamo ,

Thanks for responding this query.

I need search head and indexer as well along with deployment server stacked in single virtual server. This indexer and search head would be needed to capture Splunk internal logs for routine health checkup.

Actual system & app logs would be forwarder to some other dedicated indexers.

As per Splunk docs it says having DS with search head should handle only upto 50 clients as a best practise.

Do I need to consider this as well, or will it work fine with having only 1 DS for 400 servers and this DS having search head & indexer???