How to setup a load balancer between search head and users ? I do have a VIP address with a stick protocol enabled on two ports.
Paraphrased from http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Configuresearchheadpooling#Use_a_load_balance...
One reason for using a load balancer in front of multiple search heads is to enable users to access one of any of the multiple search heads using a single point of access.
Using a load balancer will also ensure access to search artifacts and results if one of the search heads becomes inaccessible.
Keep in mind the following gotcha:
The load balancer must employ layer-7 (application-level) processing.
Configure the load balancer so that user sessions are "sticky" or "persistent".
This ensures that the user remains on a single search head throughout their session. "
Edit 2016: You might be interested in checking out the benefits of Search Head Pooling as well: https://answers.splunk.com/answers/330572/how-do-we-sync-user-created-dashboards-and-saved-s.html
2016 nobody should be using search head pooling in a new deployment. The new hotness (as of Splunk 6.2) is search head clustering.
You need to look into search head pooling
http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Configuresearchheadpooling
We ran into issues were we couldn't do this because our NFS mount wasn't fast enough. It needs to be pretty fast in order to perform well.