HA search head configuration - load balancer

[sonicZ]

We currently have two dedicated search heads that have been running well but would like to move to a number of HA VMs for better failover redundancy and (user, app) scaling for the future.

Just a few concerns i have:
We've already created the shared etc $SPLUNK_HOME/etc dir for search head pooling.
Can we just use a load balanced vip to keep the connection state alive or would i need to use something like HA proxy/ Keep aliveD to maintain a client connection?
Would a general RR DNS record for the cluster of VM search heads work also?

I've also run bonnie++ to make sure our disk is up to par, but being that these are search heads not indexers it's not a major issue i am guessing.
Bonnie++ run#1 details below, the random seeks are < 800 - I know those are indexer specs though.

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

spweb1sb-w1-inf

8096M

21540

3

9690

1

20456

1

267.7

4

16

18461

18

+++++

+++

8370

6

4025

3

+++++

+++

11822

9

Latency

22997ms

3444ms

1613ms

555ms

Latency

714us

110us

133us

88us

12us

30us

And bonnie++ run #2 a bit better results

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

spweb1sb-w1-inf

8096M

9835

1

9311

1

22739

1

333.9

4

16

20599

19

+++++

+++

+++++

+++

20530

19

+++++

+++

+++++

+++

Latency

52577ms

3887ms

1595ms

833ms

Latency

96us

113us

134us

78us

14us

51us

[gkanapathy · Accepted solution]

You need to maintain session/cookie state between client and search head. You can use a LB VIP, but only if it's session aware on the session cookie. This works pretty much like any session-dependent web application. You search heads are probably okay with those IOPs, but lower IOPs can affect maximum user load if not necessarily single-search performance on an unloaded system.

[Lucas_K]

What version of splunk are you running on those search heads?

Using an F5 vip on a v4.3.4 search head pool results in this : http://splunk-base.splunk.com/answers/51844/excessive-session-lock-files-using-up-inodes?page=1&focu...

[sonicZ]

Thanks for the details GK!