KVStore data between Standalone search heads

[VK18]

Hi All,
I currently have a primary standalone Enterprise Security (ES) search head located in the main data center. Every day, a cronjob is executed to copy the entire /opt/splunk/etc/apps directory to the secondary standalone Enterprise Security search head, which is located in the DR site.

Now, the question arises: should I also copy the primary KVStore data, located in the var/lib directory, to the secondary ES search head? Currently, I'm only syncing the apps folder and not the var/lib directory.

In the event of an issue with the primary search head in the future, I plan to bring up the secondary search head. Will there be any issues with the KVStore data if I'm not syncing the var/lib directory between the primary and secondary search heads?

Note :Since we're not using any custom-made KVStore lookups and only depend on the default ones generated by different Enterprise Security apps, it makes us wonder if syncing the var/lib directory between the primary and secondary search heads is essential.

Regards
VK

[VK18]

Hi Giuseppe,

Thank you for your reply. 

But just wanted to understand bit more on this, All default collections.conf and transforms.conf will be synced with Secondary SH from Primary SH. Once we bring up the services on Secondary SH in future, It should populate all the data as is in primary as we got default KVstore in secondary as well ? Is my understanding correct ? 

Regards
VK

[gcusello]

Hi @VK18,

the easiest and effective approach for your requirement is to use a Search Head Cluster that replicates all the configurations and app data (as KV-Store) between SHs.

If you don't have a SH-Cluster (also because you need at least three SHs and a Deployer) so you created a workarounf to align configurations.

You have two choices:

• use an hardware platform as VxRail that automatically cyncronize the two instances (bot from Primary to secondary and from Secondary on primary).
• Otherwise you have to create some scheduled scripts that copy conf files and KV-Store from one to the other.

The firt One runs without problems.

Instead, in the second case, put attention to the application of this script, because if the Primary is down and you are using the Secondary, when the Primary will come up and running, you have to copy from Secondary to Primary and not the usual versus, in other words, you have to add many checks to your script before copy execution.

As you can understand, yhe manage of this process isn't so easy and sure!

For this reason I hint to use a Search Head Cluster that guarantees the complete and correct replication of all the objects.

Ciao.

Giuseppe 

[isoutamo]

Hi

you could use this https://splunkbase.splunk.com/app/5328 to make backups for KVstore on your primary and then restore those into secondary. I'm not sure is this still working configuration or not.

As @gcusello said, SHC do this automatically for you and without issues which other solutions definitely will generate for you.

So 1st you must understand why you need this secondary SH and based for that decide which is best / less worst solution to implement it.

r. Ismo

[gcusello]

Hi @VK18,

yes, to have a running copy of the primary Search Head, you have to copy also the $SPLUNK_HOME/var/lib/splunk/kvstore folder from the primary to the secondary.

Ciao.

Giuseppe