What are recommendations for app creation and deployment in an environment with distributed search and indexer clustering?

[jbouch03]

Hi,

We just recently redeployed our Splunk implementation with a indexer clustering and distributed search heads, and want to recreate our core Splunk application from the ground up. Indexer clustering and distributed search heads are still pretty new to us, so I was wondering if anyone had any tips for app creation on a clustered instance.

Our primary question is this:
Do we start the app creation in the cluster master? And if so as we make changes to the application do we just push the bundle down to the indexers, or do we have to create our app on each indexer and cluster master until we have finished the app, then deploy it through the cluster master?

Our app will need to meet the following constraints:
Its own index
Its own props.conf and transforms.conf
Access to other indexers
Ability to be searched through the primary search head and not just from the indexers
Use the Splunk web framework with Django bindings

Any assistance provided would be greatly appreciated. Thank you in advance.

[emiller42 · Accepted solution]

Create it outside of your cluster entirely. Set up a dev instance where you can get it ready for deployment. Once everything is good, put it on the index cluster master (etc/master-apps/) and deploy it to the cluster from there. Same with the search head cluster. Deploy it on the deployer (etc/shcluster/apps/) and then deploy from there. Use that pattern to push changes and updates.

EDIT: Since you mention web frameworks in your tags: I recommend a separate app bundle for indexers which only contains what they need to index data. Indexes, props, and transforms. (Basically, a TA to follow established app nomenclature) There's no need to push anything affecting the UI or any search-time configurations to the indexing layer.

[jbouch03]

So if I'm reading this right, it's best to create the app on a non clustered dev Splunk implementation, then once its mostly complete, move it to the production clustered environment? Then just package app updates? I was really hoping we can do it in production. We are required to use Redhat Enterprise and we already burned up all our licenses moving from 1 server to 6. It's not a big deal, but that means we have to purchase another license.

[jbouch03]

That's a really good point. We already have a TA in the works for the app.

[emiller42]

You can just run Splunk locally on your workstation. (Set up a Centos VM if you want to stay on Linux)

The nasty thing is that, while you can have your UI app just be developed on your cluster deployer (in etc/apps/) you're going to need to distribute the indexer configs from the outset via the cluster master.

Since your Indexers just need index-time configs though, you can focus on those first, get them deployed, and then focus on the SH app. Develop on the cluster deployer in etc/apps/, then when ready, copy the folder to etc/shcluster/apps/ and push to the cluster.

[jbouch03]

This has been great information. Thanks for all your help. I really appreciate it.