Hi, I am expanding from a single server install to 2 servers, each identical with half the index data on each (odd & even buckets). I have lots of custom and external apps on the first server and need to keep these in sync on both new servers,
What is the suggested mechanism and which directories / files do I need to sync?
I have tried / experimented with a few options and can't get it 100% right so looking for some guidance on what I am missing.
The goal is to have 2 indexes sharing the load but not in a cluster.
Thnaks
The built-in mechanism is an index cluster. If you want specific indexes searchable across two or more indexers, you should set them up as an index cluster. Legacy data will not be synched once you enable clustering, so if you need all historical data available, you may need to reindex this data once you've set up the indexers as a cluster. This does require a system acting as Cluster Manager, which easily can be a lightweight VM in a small environment with a small data volume indexed per day.
Just rsyncing or using other methods to copy the data back and forth is fraught with problems in accuracy of your search results at any particular time.
UPDATE:
If you aren't duplicating data, then configure your data sources to send their data to both (not duplicated but splitting your sources to send to different indexers). This would be a largely manual process if you want to distribute the data volume between the two indexers. However, this won't really do proper load balancing between the two indexers as individual searches may or may not need to hit both indexers. It is entirely possible in this setup that the search load is weighted more heavily on one indexer over the other.
Also, use Splunk Deployment Server to manage the apps and configurations. This is not a cluster setup; this is a configuration management solution for managing Splunk configurations and installed apps across multiple Splunk systems. You can manage your forwarders this way, as well, to control how the data gets to the two indexers
I am in a situation where the client does not want a cluster, I have tried to advise against it but they are adamant.
Although I agree with your point of view, given the desire of the client, what files should I be looking to sync other than etc/apps ?
Incidentally, the driver behind not wanting a cluster is data duplication. Data resiliency is not the goal, however distributing load is. Each index will have odd or even buckets on manually split initially and then automatically through the universal forwarding agent so the search load is split across both indexes without the requirement for twice the storage.
Why do you want to do this not using the built-in mechanisms for this?
I'm new to splunk, and if there is a way of doing this without creating search heads or index clusters, please point me in the right direction.