How do we sync user created dashboards and saved searches between 2 search heads that are accessed using a load balanced URL?

[shahzadarif]

We've got 2 search heads in our environment and users access them using a load balanced URL. That's all working as expected, however, we've got an issue and I just don't know how to resolve it.

Users are creating dashboards and saving searches which obviously are staying on the machine they're currently logged in. Once they log out and back in, if the load balancer takes them to the second search head, they don't see their saved searches or dashboards. I fixed this issue by creating a cron job which rsyncs apps directory once a day, but obviously this isn't as frequent as they want. I had it set to run every 2 minutes but that was causing issues like if a user deleted a saved search it would get sync'd.

Is there another way to resolve this issue besides rsync? Can I mount apps directory on a shared NFS storage or something?

[jplumsdaine22]

What you want is search head clustering. It will replicate knowledge objects between search heads. Unfortunately you're going to need another search head, as you need at least three in a cluster.

Check out the docs on search head clustering: http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/AboutSHC

[shahzadarif]

Thanks for getting back to me so quickly.
I've got Cluster Manager, can I use that as a 3rd Search Head? Thanks

[jplumsdaine22]

All the answers are in the documentation. It's fairly involved so I recommend having a good read through it. It's not as simple as adding A records to DNS as you would for simple web servers, as there is a great deal of state replication (not to mention the distribution of scheduled searches between cluster members)

The benefit of doing this work is great though. You get reliable failover and replication, and it doesn't cost you a cent extra. In my experience, adding members to a cluster seems to be fairly easy once the cluster is set up, so you can scale out easily in the future.

Also I strongly recommend putting a load balancer in front of your cluster instead of a round robin dns entry. Check out HAproxy if you don't have one already.