How to synchronize app between search head clusters?

[jbrandtelastica]

We are running two entirely separate Splunk Enterprise clusters in two AWS regions. We'd like to synchronize alerts and saved searches and so on between these regions-- ideally, automatically, so that updates in the primary cluster are copied to the secondary cluster automatically on some schedule (daily or weekly).

Currently all of our production saved searches and alerts are stored in an app on the primary cluster.

We've been thinking of two potential methods for doing this.

1. Copy the app directory from one of the primary search heads to an S3 bucket; copy the contents of that S3 bucket to the deployment server on the secondary cluster and run a cron job to update search heads and run a rolling restart of them.

Or:

1. Use the REST API to update searches remotely on one of the secondary search heads.

The first method seems pretty fragile, with a few places it could fail. The second seems like it's more complex to implement but would be more robust.

Is there a recommended way to synchronize apps between search head clusters like this?

[jcrabb_splunk]

Have you considered simply having members of the SHC in both regions? Here is a doc that discusses this:

https://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/DeploymultisiteSHC

[jbrandtelastica]

There are indexer clusters in each region, and due to some data separation requirements we can't have search heads in one region able to search indexers in other regions. Otherwise I'd just set up a single SHC. If search heads could be made site-aware and limited to a single indexer cluster while retaining multi-site configuration, that would work, though (if that makes sense).

[jcrabb_splunk]

Sorry it took me a bit to get back to you. SHC is not site aware so I can understand your approach. If you are wanting to "merge" the changes in /etc/apps/ and /etc/users/ between the environments, there isn't an out of the box approach without using SHC or Search Head Pooling (deprecated). Rysnc may be something to look at.

If the "sync" is one way, so to speak, and you just want to take changes to apps in one environment and apply them to another, you could run a script that copies the relevant apps to the Deployer's "/etc/shcluster/apps/" directory each day/week and then deploy them (as you alluded to) to the other environment. Obviously I've not tested this but something to consider.

You may also want to stop by IRC (#splunk on EFNet) and see if anyone has a suggestion, then come back and post anything you feel would be helpful.

[jbrandtelastica]

Okay, it does only have to be 1-way, and we're not worrying about stuff in /etc/users, just /etc/apps/our-app. Your rsync suggestion is pretty much what we're thinking of doing (except with s3sync instead of plain old rsync). I'm mostly worried about keeping it robust. I'll check out the irc channel, thanks.