Deployment Architecture

HA search head configuration - load balancer

sonicZ
Contributor

We currently have two dedicated search heads that have been running well but would like to move to a number of HA VMs for better failover redundancy and (user, app) scaling for the future.

Just a few concerns i have:
We've already created the shared etc $SPLUNK_HOME/etc dir for search head pooling.
Can we just use a load balanced vip to keep the connection state alive or would i need to use something like HA proxy/ Keep aliveD to maintain a client connection?
Would a general RR DNS record for the cluster of VM search heads work also?

I've also run bonnie++ to make sure our disk is up to par, but being that these are search heads not indexers it's not a major issue i am guessing.
Bonnie++ run#1 details below, the random seeks are < 800 - I know those are indexer specs though.





Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
spweb1sb-w1-inf8096M21540396901204561267.74161846118++++++++8370640253++++++++118229
Latency22997ms3444ms1613ms555msLatency714us110us133us88us12us30us

And bonnie++ run #2 a bit better results





Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
spweb1sb-w1-inf8096M9835193111227391333.94162059919++++++++++++++++2053019++++++++++++++++
Latency52577ms3887ms1595ms833msLatency96us113us134us78us14us51us

0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

You need to maintain session/cookie state between client and search head. You can use a LB VIP, but only if it's session aware on the session cookie. This works pretty much like any session-dependent web application. You search heads are probably okay with those IOPs, but lower IOPs can affect maximum user load if not necessarily single-search performance on an unloaded system.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

You need to maintain session/cookie state between client and search head. You can use a LB VIP, but only if it's session aware on the session cookie. This works pretty much like any session-dependent web application. You search heads are probably okay with those IOPs, but lower IOPs can affect maximum user load if not necessarily single-search performance on an unloaded system.

Lucas_K
Motivator

What version of splunk are you running on those search heads?

Using an F5 vip on a v4.3.4 search head pool results in this : http://splunk-base.splunk.com/answers/51844/excessive-session-lock-files-using-up-inodes?page=1&focu...

0 Karma

sonicZ
Contributor

Thanks for the details GK!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...