since index replication we have got another way to distribute configuration. I already found out by troubleshooting that any index definition in _cluster overrules my deployment apps.
Furthermore deploying via the cluster gracefully restarts splunkd.
So my question is: what are the best practices?
If you're deploying apps to the peer nodes on a cluster, you must use this method:
This puts the apps under $SPLUNK_HOME/etc/slave-apps
Configuration file precedence rules are here:
For a quick summary, see this subsection of that topic:
It's not that you "must use this method", its that you can use this method and if you want to,... here's how you do it... but beware that the cluster bundle will supersede any configurations you send via deployment servers.
The configuration bundle method is the only way to ensure that certain key configurations stay in sync across all peer nodes, which is necessary for the proper operation of the cluster.
Regarding use of deployment server or third-party tools:
It's against best practices to use a deployment server to push apps to indexers. Mainly because it doesnt gracefully restart the indexer. That being said, it can be done but with caution.
Best practices is to use the Cluster Master's slave-apps to deploy to your indexers. This keeps parity among the indexers and cluster master and can reduce load on the cluster that would be associated to peers that dont cleanly restart.