Which tools/methods are considered the best for orchestrated deployment of universal forwarder installations?

[adnankhan5133]

We have Chef in our environment, as well as a Deployment Server. Is there any guide out there on how to deploy UF’s across hundreds of Linux and Windows servers?

[richgalloway]

You should use Chef to install and upgrade the UFs.  As part of the installation process, have Chef delete the deploymentclient.conf and outputs.conf files from $SPLUNK_HOME/etc/system/local and define custom versions of the those files in an app.  This allows the deployment server to manage those configurations.

Once that's done, allow the DS to control which apps are installed on which forwarders.  You probably could do that with Chef, but like using the DS because it has insight into the state of the forwarders.  In some shops, agent installations are done by a different team so using the DS gives the Splunk admin more control over how the forwarders are managed.