Since the recommended best practice is for ES to run on its own cluster, I have several ES customers that run both an ES and non-ES search head cluster.
Is there a recommended method for keeping configs that are applicable on both clusters in synch? When users create content that gets saved in the "local" directory for the app, what is the best way to get those changes into the other cluster? Can I just copy that directory to the other search head cluster and have it synchronize? Or does the search cluster only recognize that new content if it is created by a user in the GUI?
If I take the local content and push it out with the search deployer, it will end up in the default directory on the search heads rather than the local directory. Can the local directories then be deleted or are those changes stored in the Raft repository and will then get reapplied?
Take a look at http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/HowconfrepoworksinSHC. Specifically, "The cluster does not replicate any configuration changes that you make manually, such as direct edits to configuration files.".
These changes can however be pushed from the deployer. So for instance, on shcluster1, user1 makes a change via GUI. That change is replicated to all shcluster1 search heads. Depending on what kind of changes you're looking for, maybe change to dashboards, you could push that to the deployer in shcluster2, then deploy the change. Maybe speak to what sort of changes you're looking to keep in sync.
hello there, @hortonew comment is spot on and very valid imho.
would like to suggest a different approach.
if i understand your requirement, you are asking to keep an ES SHC and a non ES SHC (of the same client) in sync. obviously, each SHC has their own deployer.
will recommend to have a "dev" app on each of the SHCs, or even better a dev app in a SH that sees all data but does not belong to any of the SHC.
rsync that app to deployers on a regular cron basis and push to clusters from each deployer.
that will keep your items in sync.*
with that being said, why would you want items from regular SHC on ES SHC and vice versa?
Hope it helps
*note: you might not be able to use that method on kv store items