Evening Splunk community,
My organization practices Blue / Green data-centers and requires us to switch production data centers every quarter.
In my environment I manage two standalone Search Heads. One Search Head in each data-center, separated by region. I'm trying to determine a clean solution for keeping user knowledge artifacts (saved searches, reports, alerts..ect) synced across the two Search Heads without having to implement Search Head Cluster Replication.
Does anyone have any tips, advice, or general best practices when it comes to keeping knowledge objects synced between two or more standalone Search Heads? I've read a few forum posts that cover this topic and I've detailed some of the solutions I'm brainstorming, but wanted to get everyone's opinion before I start down the wrong path.
For starters I believe the knowledge artifacts on the Search Head reside under the following directories. Not including the saved searches within /etc/apps.
$SPLUNK_HOME/etc/system/local/authentication.conf $SPLUNK_HOME/etc/system/local/authorize.conf $SPLUNK_HOME/etc/users/* Everything in local folders and local.meta files under splunkhome/etc/apps
I think search head clustering is the solution they you want. It allows you to share knowledge objects amongst the heads. And if you have four or more you can lose a head without undue pain.
After you have it setup, the day to day administering should take less time than monitoring each individual head. You could use a monitoring console to monitor the SHC. You could use it to setup all kinds of alerts.