I have a distributed splunk environment where I have 1 search head and 3 indexers.
I would like to install second search head for maintenance reasons, so when I need to do kernel or splunk updates on first search head, second search head is still available for users.
How can I accomplish this. ? Any links to an how to would be great too.
Are you planning to use Search Head Pooling, optionally with both heads behind a load balancer so your users can transparently be failed over to another head (during maintenance) ?
This link has some good info.
A few key points :
-you'll need shared storage(ie: NAS) so the search heads can share the same etc/apps , etc/users directorys
-each head maintains its own etc/system directory
-enable pooling on each head (simple to do using the CLI)
-if using local users, the etc/passwd file must be maintained on each search head.I prefer using LDAP authentication.
-if using a load balancer and alerting , setup the load balancer host name as the alert link hostname.
The steps are pretty much the same for your 2nd/3rd/4th search heads. You will, however, want to make sure that you copy/replicate your config apps/bundles to the additional search head so they use the same field extractions, lookups and such.