We are trying to achieve one-click deployments with Splunk applications. Our desired workflow is below:
1) we develop the app and push the changes to the develop branch
2) we have a pipeline listening for any changes on the branch and when changes are pushed, deployment is triggered on a WIP environment
3) we pack the Splunk applications for the search head and deployment server in tar.gz archives which are extracted on the search head/deployment server (in etc/apps on the search head and etc/deployment-apps on the deployment server) and rsynced using the --delete flag so that if an app has been removed from git, it will also be removed from the search head/deployment server apps - this process is being executed by Chef
4) we manually reload the deployment server or, in the case of adding a new index, we manually apply the cluster bundle
We are facing the following difficulties:
There are just some of the challenges we've faced so far. Is anyone using CI/CD (Continuous Integration / Continuous Deployment) to deploy Splunk applications? We are using GoCD if this helps.
Thanks in advance.
For your DEV/QA environment, you should try using data cloning at the forwarder level over to those indexers instead of snapshot copying. Then just set the data retention in the DEV/QA environment to a much smaller level. You should be able to have the same apps on the search head and indexer as normal but they would be separate from your prod environment (much more cost effective if you're not using a cluster / multisite cluster as DEV/QA as well).
You can deploy apps on the search head using the deployment server (I use the DS to deploy tech addons and additional configurations to the search heads, however be careful of deploying apps to a prod environment where you allow users to edit dashboards, reports, and saved searches with global permissions, as the deployment server likes to keep the apps' synced).