Deployment Architecture

Is there a way to limit the number of clients a Splunk deployment server pushes a new/updated app to at the same time?

jbresciani
New Member

Is there a way to limit the number of clients a Splunk deployment server pushes a new/updated app to at the same time? We've run into issues where to the Splunk universal forwarder is restarted on hundreds of virtual machines simultaneously causing massive IO delays on shared storage.

I'm looking for a way to have a Splunk deployment server stage out the deployment of app changes so that only X number of nodes are deployed to at a time.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

There is no way to do this with Splunk native functionality aside from groups in serverclasses [ group by subnet / server location / server role / etc.] The best option is to adjust the phonehomeinterval to a level that reduces your i/o to acceptable levels. The caveat to this is that it takes longer for your clients to update.

Outside of Splunk, there are some other things you could potentially do, although not recommended. One option Ive seen is to run with multiple dns cnames for your deployment server. Push your updates in a staggered manny by dropping the cnames out while pushing updates. The UF will still phone home, but wont be able to connect until you put the cnames back in. You need to be aware of the ttl values in dns in this case though and have it set really low.

Another option would be firewall rules, block inbound to the DS for portions of your UF's subnets while updates are being pushed...

sobrien
Splunk Employee
Splunk Employee

Hi Jacob,

A good way to achieve this would be to break up your fleet into smaller serverclasses:

https://docs.splunk.com/Splexicon:Serverclass

You can then confirm the successful install of your apps and updates per serverclass prior to rolling out to the rest of the fleet.

Hope this helps.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...