We have a clustered environment with 3 SHC and 2 indexers. We have been using KV Store with great success as an alternative to csv files for lookups. However, we had a DNS issue with one of the indexers and after this the requesting data from the KV Store takes 10.2 seconds regardless of the collection size.
From what I can tell the cluster is healthy and the mongod.log does not show any problems. When I make a request for data from a collection I see the 10 seconds are spent at the same in search log:
01-24-2019 10:37:57.355 INFO SearchStatusEnforcer - Enforcing disk quota = 10485760000
01-24-2019 10:37:57.355 INFO PreviewExecutor - Preview Enforcing initialization done
01-24-2019 10:37:58.356 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-24-2019 10:38:07.629 INFO PreviewExecutor - Stopping preview triggers since search almost finished
01-24-2019 10:38:07.629 INFO DispatchExecutor - END OPEN: Processor=noop
01-24-2019 10:38:07.635 INFO ReducePhaseExecutor - Ending phase_1
Is there anyone that have an idea what might be causing this?