Brief Summary Git...Git...Git and Ansible. We've got multiple git repositories in which changes are proposed, tracked, and codified, which are then distributed by a mixed means of either git sync or Ansible push. This helps our team of multiple engineers all have a hand the work without stepping on each other. Combined with a git-shell profile, we have visibility into when local changes have been made, and educate us on what files (*cough* app.conf *cough) are created automatically. In Depth: In an attempt to standardize our TA and app experience, we've built an architecture around the utilization of a DS. The Cluster Manager, Search Head Deployers, Single Purpose Distributed search heads, and modular heavy forwarders all check into a centralized enterprise deployment server. That deployment server pushes out apps to the top tier management systems into their respective directories, like the Search Head Deployer's shcluster/apps. We rely on Splunk's tiered DS options to make sure everything runs properly without unnecessary restarts. This allows us to update a single repository with TAs for the environment and distribute them to the search and indexing tier without having to track and remember that "Oh the indexer version of the Cisco TA is located and maintained here and the search version is over here, and they have slightly different regex for this new field extraction the customer wanted. We do something similar with our deployment servers for UFs. In the end, we maintain 4 repositories, and utilize Ansible to push out a majority of our configurations. Instead of the 8 or 9 repositories if we handled this by tier.
... View more