We have set up a distributed architecture for splunk app for vmware.
Architecture components: 1 Master node, 1 SH (which has scheduler setup), 2 Indexers, 1 Forwarder (which is the DCN).
While we try to push TAs from the master node to the indexers, we get errors particularly for SA-Utils and Splunk_TA_vmware. Rest all TAs - Splunk_TA_vcenter , Splunk_TA_esxilogs and SA-Hydra - can be distributed without any issue.
Error for Splunk_TA_vmware states
Error for SA-Utils
If we try forceful pushing (skipping validation through CLI), the indexer then stops working and keeps on prompting error "No app servers running. Server had an unexpected error."
However in the configuration manual, it is mentioned to remove the SA_Utils if forwarding configuration bundle through CML
http://docs.splunk.com/Documentation/VMW/3.1.4/Configuration/Deploytheappinaclusterdeployment
We have tried the same and this works, but Utils is one of the important components which is required on indexer (also mentioned in documentation:
http://docs.splunk.com/Documentation/VMW/3.1.4/Configuration/Componentreference
So now what steps should be followed to move SA_Utils to the indexer?
As a work around for now, we have manually dropped the required components on the indexer in /opt/splunk/etc/apps/ , but then there is no point in doing this because we will not be able to auto sync configuration changes in future.
Is this appropriate way of setting up VMware in distributed architecture? Or we are missing anything? Please advise!
It is a little complicated design issue between apps and core structure. That makes us deploying the app to Indexer Clustering environment.
If you use CLI to avoid validation at Cluster Master, and deploy SA-Utils, SA-Hydra and the TA components, it is supposed to work.
Can you also double-check if those TA/SA packages are deployed in $SPLUNK_HOME/etc/slave-apps directory in each Cluster Peer? Sometimes, a user unzipped the app package and see etc directory under $SPLUNK_HOME/etc/apps directory, and that cause unexpected error.
If it is still not working, can you please file a Support case for further investigation?