Archive
Highlighted

One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

Motivator

Hi All, Currently I am facing the above issue, ours is distributed system with search head pooling configuration setup. Before the search head F5 load balancer is configured to balance the User traffic hitting the search head.
Splunk Version 6.0.3.

Issue -

I have validate the splunkd process running in both the search head instances, but still in F5 load balancer it showing down and also when one of the splunkd process were stopped its not switching to the other splunk instances automatically. But not sure what algorithm or configuration has been done in F5 side. But before going to F5 team I need to check from splunk side for this issue. So kindly guide me what are the configuration file should verified in Splunk.

Tags (1)
0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

Builder

If you can load the search head by hostname in a broswer the same as the others then odds are the F5 simply cannot talk to the search head. My guess, firewall/vlan rules.

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

SplunkTrust
SplunkTrust

Was this working earlier? Check if the F5 is redirecting traffic to correct Splunk web port.

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

SplunkTrust
SplunkTrust

Unfortunately you need to know exactly how the F5 is configured because it sounds like it isnt properly configured.

You should have a "front end" VIP listening on some port (usually 8000, but could be whatever you desire) with a backend pool of splunk search head instances listening on your splunk web port (usually 8000), and load balancing based on cookie persistence. You should be using the sessionid{splunk web port number} cookie for the application cookie based persistence.

You should also have health checks based on your splunk web port number.

View solution in original post

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

Motivator

Hi All thanks for your inputs on this issue, I had tried to access the URL individually but it failed, I am able to access one of the Splunk search head URL instance which is showing up in the F5 balancer.

https ://hostname.xxx.com:8443 --- splunk search head 1 (Status showing UP in F5)
https://hostname.xxx.com:8443 --- splunk search head 2 (Status showing down in F5)

I am getting page can not be displayed error

How / where to find the web port /URL configuration details for accessing the search head.

thanks in advance

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

SplunkTrust
SplunkTrust

you can log onto the search head and run a few commands that will shed light.

./splunk list web-port

Will show what the current configured web port is

netstat -an

Will show if the server is indeed listening on that web port

./splunk btool web list --debug

Will help to show which web.conf file is setting the server port

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

Motivator

thanks jkat54, after executing the below command on both the server, I got this output

Search head 1 :

./splunk btool web list --debug
/splunksearchpool/etc/apps/ADMIN-allsearchheads/default/web.conf httpport = 8443
/splunksearchpool/etc/apps/ADMIN-allsearchheads/default/web.conf trustedIP = 10.140.x.x,10.140.x.x,127.0.0.1,168.133.x.x,168.133.x.x,168.133.x.x,168.133.x.x,168.133.x.x,168.133.x.x

search head 2 :

./splunk btool web list --debug

/opt/splunk/etc/system/default/web.conf httpport = 8000

After executing the above commands, I could see that search head 1 is listening to the port 8443 and where as search head 2 listening to the port 8000.

when executed this url https://search head1.xxxx.com:8443 - able to access and login
where as for search head 2, i am able to access using http://search head2.xxxx.com:8000 but unable to login.

Similarly when execute nestat -an , I could see that search head is listening to the port 8443 where as the search head2 its not listening to any ports mentioned in web.conf file.

Kindly guide me how to fix this issue so that I can able to access both the search head and sync with each other.

thanks in advance.

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

Motivator

Hi All, Can you guide us on this issue, as we need to make both the splunk search head up URL link up and running in our environment. Currently only one search head is accessible. thanks in advance.

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

SplunkTrust
SplunkTrust

looks like search head 2 is down

Try these commands as the splunk user
/opt/splunk/bin/splunk status <- might tell you its not running
/opt/splunk/bin/splunk restart <- might tell you an error during the startup

Any reason why sh1 has sso enabled and runs on different port? Shouldnt they be in sync?

0 Karma
Highlighted

Re: One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

Motivator

Hi Jkat thanks for your effort on this, I have checked both splunk status and found to be up and running fine. Even checked with the F5 team and they told me that its configured with Round Robin method and its listen to 8443.

Status of search head 1 (root user)
./splunk status
splunkd is running (PID: 26667).
splunk helpers are running (PIDs: 26668).
splunkweb is running (PID: 26702).

status of search head 2 (root user)

./splunk status
splunkd is running (PID: 31308).
splunk helpers are running (PIDs: 31309).
splunkweb is running (PID: 31372).

I am not sure why the search head is listening to different port. Can you guide me how to make this sync with the same port number.

thanks in advance.

0 Karma