Anyone have and recommendations or best practices to follow when deploying splunk apps in medium to large deployments?
I have a bunch of all-in-one apps that contains inputs, savedsearches, indexes, props (index-time and search-time settings), and custom views. As we're looking to split out our splunk deployment across multiple systems (Forwarders/indexer/search head), it seems like a good idea to split out the configurations so only the relevant parts are deployed to each component.
I'm looking for advice on how to split-up, name, and manage separate apps.
Selfishly, I would recommend you look at the unix app and TA (technology add-on). We do a few things in most of our apps that you can see examples of there:
I'm sure there is stuff I missed or that you have questions on. Please feel free to add comments and I will update the answer.
Thanks for your feedback. We do use version control for the apps, but we deploy all the apps from a single deployment server, and there are never customized on individual systems. The idea of using a build process to automatically split up apps into different chunks is interesting, but probably more work than what I can justify...
Any thoughts on naming conventions for the apps?
MyApp-IDX? How do you handle the difference between deploying to a heavy forwarder and a universal forwarder? (Do you deploy the all the props setting to the universal forwarder and to the indexer?)
We tend to just use
TA-myapp. In general, the TA will have all the bits necessary to deploy on a heavy forwarder - if deployed on a light forwarder or on an indexer where the data is coming in cooked, they will have no impact.
With regard to our very simple python build system, we have talked about open sourcing it, but there is not really clarity around what we will do with it at this time.