Deployment Architecture

How to synchronize app between search head clusters?

jbrandtelastica
New Member

We are running two entirely separate Splunk Enterprise clusters in two AWS regions. We'd like to synchronize alerts and saved searches and so on between these regions-- ideally, automatically, so that updates in the primary cluster are copied to the secondary cluster automatically on some schedule (daily or weekly).

Currently all of our production saved searches and alerts are stored in an app on the primary cluster.

We've been thinking of two potential methods for doing this.

  1. Copy the app directory from one of the primary search heads to an S3 bucket; copy the contents of that S3 bucket to the deployment server on the secondary cluster and run a cron job to update search heads and run a rolling restart of them.

Or:

  1. Use the REST API to update searches remotely on one of the secondary search heads.

The first method seems pretty fragile, with a few places it could fail. The second seems like it's more complex to implement but would be more robust.

Is there a recommended way to synchronize apps between search head clusters like this?

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

Have you considered simply having members of the SHC in both regions? Here is a doc that discusses this:

https://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/DeploymultisiteSHC

Jacob
Sr. Technical Support Engineer

jbrandtelastica
New Member

There are indexer clusters in each region, and due to some data separation requirements we can't have search heads in one region able to search indexers in other regions. Otherwise I'd just set up a single SHC. If search heads could be made site-aware and limited to a single indexer cluster while retaining multi-site configuration, that would work, though (if that makes sense).

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

Sorry it took me a bit to get back to you. SHC is not site aware so I can understand your approach. If you are wanting to "merge" the changes in /etc/apps/ and /etc/users/ between the environments, there isn't an out of the box approach without using SHC or Search Head Pooling (deprecated). Rysnc may be something to look at.

If the "sync" is one way, so to speak, and you just want to take changes to apps in one environment and apply them to another, you could run a script that copies the relevant apps to the Deployer's "/etc/shcluster/apps/" directory each day/week and then deploy them (as you alluded to) to the other environment. Obviously I've not tested this but something to consider.

You may also want to stop by IRC (#splunk on EFNet) and see if anyone has a suggestion, then come back and post anything you feel would be helpful.

Jacob
Sr. Technical Support Engineer
0 Karma

jbrandtelastica
New Member

Okay, it does only have to be 1-way, and we're not worrying about stuff in /etc/users, just /etc/apps/our-app. Your rsync suggestion is pretty much what we're thinking of doing (except with s3sync instead of plain old rsync). I'm mostly worried about keeping it robust. I'll check out the irc channel, thanks.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...