After failing over from the active cluster master to the redundant node (which holds the same configuration), 15 buckets report now    slave bucket=XXXXXX has unexpected mask=11 expected=0   This results in search factor not met for the corresponding indices.  I can see the cluster master running the "CMChangeMasksJob" in regular intervals but it looks to me it just can't handle those 15 buckets.  I am looking for any hints how to tackle this. First and foremost, what is a bucket mask? Are my buckets corrupted? Can I try to update the mask manually?   ... View more

We have noticed that during the HFW shutdown procedure (e.g. caused by Parsing app deployment) there is a sequence of events for what seems to be closing the active TCP incoming connections. An example as follows:   09-08-2020 15:12:38.606 +0100 INFO TcpInputProc - Running shutdown level 1. Closing listening ports. 09-08-2020 15:12:38.606 +0100 INFO TcpInputProc - Done setting shutdown in progress signal. 09-08-2020 15:12:38.606 +0100 INFO TcpInputProc - Shutting down listening ports 09-08-2020 15:12:38.606 +0100 INFO TcpInputProc - Stopping IPv4 port 9997 09-08-2020 15:12:38.606 +0100 INFO TcpInputProc - Setting up input quiesce timeout for : 90.000 secs 09-08-2020 15:12:38.942 +0100 INFO TcpInputProc - Waiting for connection from src=172.18.18.185:64536, 172.30.194.1:58219, 172.16.57.76:49451, 172.16.218.6:52112, 172.30.50.1:34143, 172.16.36.20:50702, 172.18.13.28:39612, 172.30.66.2:47563, 172.16.57.79:54330, 172.16.165.70:57168 ... to close before shutting down TcpInputProcessor. ... 09-08-2020 15:14:19.103 +0100 WARN TcpInputProc - Could not process data received from network. Aborting due to shutdown 09-08-2020 15:14:20.123 +0100 WARN TcpInputProc - Could not process data received from network. Aborting due to shutdown 09-08-2020 15:14:21.138 +0100 WARN TcpInputProc - Could not process data received from network. Aborting due to shutdown 09-08-2020 15:14:22.172 +0100 WARN TcpInputProc - Could not process data received from network. Aborting due to shutdown     Now, what worries me is that the number of "TcpInputProc - Could not process data received from network. Aborting due to shutdown" events can vary anything from 20 (which in total takes 15-20 seconds) to 100s (which can be as long as 4 minutes). The more those events I have, the longer the shutdown procedure takes and the longer the HFW remains inactive (unable to process data). Eventually, the shutdown process is forced after the default 360s. Questions: 1. Why do we see different number of "TcpInputProc - Could not process data received from network. Aborting due to shutdown" events on different occasions? 2. Is there any way of limiting them and generally speeding up the shutdown procedure? Maybe there is some tuning we can do on the HFW nodes? ... View more