How to sync apps and configurations without a deployment server in my distributed search environment?

[bondbig]

Hi!

I have 4 Splunk servers (one per each geographical location), each with combined Indexer and Search Head roles (yes, I know that it's not good, but I'm limited with number of servers), and each server gets its own portion of events. Servers are united as search peers, so whatever search head you use, all data is searchable. However, all configurations are done manually on each server: index creation, listeners, apps and so on.

I can't use indexer clustering because it doubles (or even quadruples) required storage and consumes bandwidth of links between locations. And currently I cannot use Deployment server, because it requires a separate machine (I'm going to have about 2000 forwarders).

Are there any tricks on how to sync at least some configuration in this scenario? I was thinking about a shell script, which will do regular sync and server restart/reload, but I'm sure there are some other (better) ideas in this community.

[jchampagne_splu]

There is nothing stopping you from using Index Clustering and just setting the SF:1 and RF:1. You'll get all of the advantages configuration management across your indexers AND all of your buckets will be ready for clustering should you decide to turn it on someday.

With regard to other configurations, you can use something like rsync to synchronize $splunk_home/etc/users and $splunk_home/etc/apps that aren't synchronized by the cluster master.

For your forwarders, if you're not using Deployment server, you'll need to use some other kind of configuration management tool like SCCM, Chef, or Puppet.

[bondbig]

Thank you! I've already tried to set SF and RF to 1 in test environment, this seems to be best option.

Regarding deployment server: what is the actual real-life HW requirements for a deployment server which manages 2000 forwarders? They all will be installed on windows servers and mostly have pretty basic configuration: collect application, system and security log.
Does heartbeat interval impacts performance? If I set it to 12 hours, for example, would it lower load to Deployment server?

[jchampagne_splu]

A single deployment server running on our reference hardware can handle 10,000+ clients. It all comes down to the size of the bundles being distributed and how quickly you'd like to distribute the configurations.

We have a formula to help you figure out how long it will take a single deployment server to distribute configurations to all of your clients:
http://docs.splunk.com/Documentation/Splunk/6.3.1511/Updating/Calculatedeploymentserverperformance

Yes, if you reduce the phone home interval for deployment clients, that would lower the load on the deployment server and allow you to support more clients. However, it also increases the distribution time for configurations.

[bondbig]

Thank you!
What about this:

The deployment server is not supported as a means to distribute configurations or apps to cluster members.

I still want to have apps and other stuff (saved searches, dashboards, etc) synced between my 4 servers.

[jchampagne_splu]

If all of your servers are acting as Search Heads AND Indexers (not recommended, as you already pointed out), then Index clustering would keep everything in sync automatically. The only thing that wouldn't be kept in sync would be user-specific changes that are saved to $SPLUNK_HOME/etc/users, you'd need to use a script or something like rsync to synchronize those folders.

One thing to note here, if you're synchronizing scheduled searches across search heads manually, then each of your 4 search heads will be running those scheduled searches independently. You'll be wasting a lot of resources running the same job multiple times. The search artifacts generated by your searches will also not be synchronized. These are all things you get out of the box with proper Search Head Clustering.

[bondbig]

thank you!

Index clustering would keep everything in sync automatically
including apps and users?

I couldnt find a detailed list of what is synced by index clustering and what is not.

And it looks like index clustering is not an option for me because of this:

Cluster nodes cannot share Splunk Enterprise instances. The master node, peer nodes, and search head must each run on its own instance.
it means that i will need at least 5 servers (4 indexers/receivers for each location and one search head), but i'm limited with 4.

[jchampagne_splu]

No, Index clustering will only keep the items you place in master-apps on the Cluster Master in sync. For anything in $SPLUNK_HOME/etc/apps or $SPLUNK_HOME/etc/users, you'll need to use a script.