One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

[Hemnaath]

Hi All, Currently I am facing the above issue, ours is distributed system with search head pooling configuration setup. Before the search head F5 load balancer is configured to balance the User traffic hitting the search head.
Splunk Version 6.0.3.

Issue -

I have validate the splunkd process running in both the search head instances, but still in F5 load balancer it showing down and also when one of the splunkd process were stopped its not switching to the other splunk instances automatically. But not sure what algorithm or configuration has been done in F5 side. But before going to F5 team I need to check from splunk side for this issue. So kindly guide me what are the configuration file should verified in Splunk.

[Hemnaath]

Hi Jkat, could you guide me on this problem ?

thanks in advance.

[Hemnaath]

Hi Jkat, can u guide me in getting this issue fixed.

[Hemnaath]

thanks Jkat, But kindly guide me know whether the below steps are right to fix this issue.

Steps :
1) Stop the splunk service running in the search02 server by executing the command
/opt/splunk/bin

./splunk stop

2) Remove the partially created folder under the mount point /splunk_search_pool from search02 server by executing the command
rm -rf /splunk_search_pool
3) Use the below command to mount back to the correct mount point as it is in search01
mount -t nfs splunfs01:/opt/splunk_shp /splunk_search_pool

4) Then restart the splunk service by executing the below command
/opt/splunk/bin/
./splunk start
5) Need to check whether both the URL is working by hitting the respective URLS
https://search01.com:8443
https://search02.com:8443

Thanks in advance.

[Hemnaath]

can you guide me whether these steps should be followed to make this issue get fixed.

Search02 - Currently showing down in F5

Steps to change the mount point in search02

1) stop the splunk services /opt/splunk/bin
./splunk stop
2) Remove the directory which contain partial data available in seach02 under the mount point
/splunk_search_pool
3) Using the below command mount the FS to the same mount point as in search01

mount -t nfs splunfs01:/opt/splunk_shp /splunk_search_pool

4) Restart the splunk service

Kindly guide whether above steps are right way to fix this issue.

[Hemnaath]

thanks Jkat for putting your effort on this. As guide I had executed the df -k and got this outputs.

[user@search01 /]$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_search01-lv_root
20513276 11111620 8358472 58% /
tmpfs 1960364 0 1960364 0% /dev/shm
/dev/mapper/vg_search01-lv_opt
51475068 8949172 39904840 19% /opt
/dev/mapper/vg_search01-openv
3030800 912664 2118136 31% /usr/openv
/dev/sda1 499656 100452 372992 22% /boot
syna02:/security_splunk
16777216 1111808 15665408 7% /splunk_search_pool_old

splunfs01:/opt/splunk_shp
20517888 5558528 13917184 29% /splunk_search_pool ---> This server belongs to lisence manager come Deployment manager server. where I could see the splunk_search_pool directory which is missing in the search head 2.

[user@search02 ~]$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_search02-lv_root
14987656 10774928 3444728 76% /
tmpfs 1960364 0 1960364 0% /dev/shm
/dev/sda1 499656 100472 372972 22% /boot
/dev/mapper/vg_search02-optvol
20511356 14354112 5108668 74% /opt

Kindly provide me some steps to fix this issue, thanks in advance

[JDukeSplunk]

If you can load the search head by hostname in a broswer the same as the others then odds are the F5 simply cannot talk to the search head. My guess, firewall/vlan rules.

[somesoni2]

Was this working earlier? Check if the F5 is redirecting traffic to correct Splunk web port.