What would be considered the Splunk best practice when managing multiple deployment servers in different locations that also act as log collectors, while wanting to keep configurations on each consistent as possible?
Would it be a tiered deployment setup, utilizing /opt/splunk/etc/apps/ as the repositoryLocation in the serverclass deployed out to the HF/DS systems and also using this serverclass.conf to deploy out to Universal Forwarders that check into them, and then controlling which apps are enabled/disabled when they're distributed to those DS's on a central, top-tier DS?
Or does Splunk recommend that something like Ansible or a similar technology be introduced to do this type of configuration deployment?
You are able to specify different repository locations for the same host by assigning one host to multiple server classes. So in this example you might have one top-level DS that feeds to three tiered HF/DS's (as a side note, depending on how many clients the tiered DS's will have and how much work the Heavy Forwarder will be doing this might be a bad idea, see https://docs.splunk.com/Documentation/Splunk/8.1.1/Updating/Planadeployment):
- DS:
serverclass.conf
[serverClass:heavy_forwarders]
whitelist.1 = <insert hostname regex or ip regex>
targetRepositoryLocation = $SPLUNK_HOME/etc/apps/
[serverClass:heavy_forwarders:app:app_for_hf]
#Here you can include any apps that will manage the tiered DS itself, such as the serverclass.conf that will be used to send configs to the tiered DS's clients
[serverClass:tiered_ds]
whitelist.1 = <insert hostname regex or ip regex, will be same as above>
targetRepositoryLocation = $SPLUNK_HOME/etc/deployment-apps/
[serverClass:tiered_ds:app:app_for_uf_through_tiered_ds]
#Here you can include anything that will go out to the tiered DS's clients
That makes sense.
Would this be considered a Splunk best practice in this given architecture? There isn't a lot of material out there on the situations where nesting deployment servers would be a requirement, and whether Splunk suggests using its own methods to manage multiple deployment servers (nesting) or if an alternate technology should be used to manage these configurations.
It sort of depends on your situation. How many deployment clients do you anticipate having? Are there network segregation concerns? A properly provisioned Deployment Server can handle thousands of clients, so while there are some cases that a tiered deployment makes sense, it's usually not necessary.