How to set up 2 search heads behind 1 Apache reverse proxy with load balancing?

[strangelaw]

Howdy!

I have wondered around the topic within Splunk community for a while, but did not find a definite answer. So this time I hope there's at least 1 decent setup around. Well, that is - after this is discussed through.

Anywho, here is my config. See, I have tried a bunch of configs. I do know there's some overlap on 001-proxy.conf for directories.

At the moment, the problem seems to be that session gets immediately expired (at least it says so on web browser). It allows me to give credentials and so on, but after hitting enter on password it yells "Session Expired".

My goal is simple: have 2 search heads behind 1 apache reverse proxy. Loadbalancing is optional 🙂

001-proxy.conf:

ProxyRequests Off
ProxyPreserveHost On

BalancerMember http://firstsearchhead.ip:8000
BalancerMember http://secondsearchhead.ip:8000

AllowOverride None
Order deny,allow
Allow from all

ProxySet lbmethod=byrequests
ProxyPass /en-US balancer://searchheads/en-US
ProxyPassReverse /en-US balancer://searchheads/en-US
ProxyPass /static balancer://searchheads/static
ProxyPassReverse /static balancer://searchheads/static
ProxyPass /manager balancer://searchheads/manager
ProxyPassReverse /manager balancer://searchheads/manager
ProxyPass / balancer://searchheads/
ProxyPassReverse / balancer://searchheads/
ProxyPassReverseCookiePath / /

web.conf Search head configuration on - both heads have the same config except the tools proxy base IP, as below:

[settings]
x_frame_options_sameorigin = False
#tools.proxy.on = True
#root_endpoint = /splunk
#root_endpoint = /
#SSOMode = permissive
#trustedIP = 127.0.0.1, proxy.ip.here.too
appServerPorts = 0
enableSplunkWebSSL = 0
tools.sessions.timeout = 4320
tools.proxy.on = True
SSOMode = permissive
tools.proxy.base = http://searchhead.ip:8000  (<- This one is actually different/unique per search head)

Anyone who really knows how this should be implemented, please shine with your thoughts here!

I would appreciate this much.

[shahzadarif]

This is how I've achieved this on Splunk 6.3.3:

My web-proxy conf:

ProxyPass /splunk http://myVIP.com:8000/splunk
ProxyPassReverse /splunk http://myVIP:8000/splunk

My inputs.conf on Search Heads $SPLUNK_HOME/etc/system/local:

[default]

host = upsks01b

host = VIPurl

web.conf on Search Heads $SPLUNK_HOME/etc/system/local

[settings]
root_endpoint = /splunk

What version of Splunk are you running on Search Heads?

[strangelaw]

Version is 6.3.2

So you have not actually configured Apache as LB proxy for those 2 search heads? Well, thing you describe above is what I am running now - but what I need is 2 search heads running behind Apache so that Apache really runs LB for them (or failover).

USER --> APACHE --> head1/head2/head3...nn --> SPLUNK indexes (3...nn)

Whilst it's not the most scalable, the apache portion itself is easier to scale in parallel.

M