Why does splunkd.exe has failed connections and high latency towards our Splunk servers?

[Doxie]

Hi,

Let me start by saying that that i have a very limit knowledge about Splunk, its normally not my area of expertise.

I made some performance investigations and accidently came across some interesting finding for Splunk.
With one of the tool i'm using i could see that splunkd.exe had a very high latency towards our Splunk servers, 700ms-1000ms and more than 20% failed connections.

I cant really verify those numbers, because if i do a normal ping towards the same servers, i get around 20ms, so its only splunkd.exe that have the high latency.

I was wondering if anyone could point me in the right direction, where to look, to get an understanding of this "issue".

outputs.conf

[tcpout]
defaultGroup = primary_heavy_forwarders
maxQueueSize = 7MB
useACK = true
forceTimebasedAutoLB = true
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_heavy_forwarders]
server = NAME1:9997, NAME2:9997, NAME3.com:9997

#clientCert = $SPLUNK_HOME/etc/auth/server.pem
#sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
#sslPassword = ********
#sslVerifyServerCert = true

splunkd.log (part of the log file for today, from a client)

02-09-2022 10:48:20.841 +0100 INFO ApplicationLicense - app license disabled by conf setting.
02-09-2022 10:48:26.777 +0100 WARN TcpOutputProc - Cooked connection to ip=IP1:9997 timed out
02-09-2022 10:48:50.836 +0100 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
02-09-2022 10:48:56.568 +0100 WARN TcpOutputProc - Cooked connection to ip=IP1:9997 timed out
02-09-2022 10:49:09.291 +0100 INFO TcpOutputProc - Closing stream for idx=IP2:9997
02-09-2022 10:49:09.291 +0100 INFO TcpOutputProc - Connected to idx=IP1:9997, pset=0, reuse=0. using ACK.
02-09-2022 10:50:17.238 +0100 ERROR TcpOutputFd - Read error. En befintlig anslutning tvingades att stänga av fjärrvärddatorn.
02-09-2022 10:50:17.238 +0100 INFO TcpOutputProc - Connection to IP2:9997 closed. Read error. En befintlig anslutning tvingades att stänga av fjärrvärddatorn.
02-09-2022 10:50:17.238 +0100 WARN TcpOutputProc - Possible duplication of events with channel=source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log|host::807|splunkd|2728, streamId=0, offset=0 on host=IP2:9997
02-09-2022 10:50:17.238 +0100 WARN TcpOutputProc - Possible duplication of events with channel=source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log|host::807|splunkd|2727, streamId=0, offset=0 on host=IP2:9997
02-09-2022 10:50:17.238 +0100 WARN TcpOutputProc - Possible duplication of events with channel=source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log|host::807|splunkd|2721, streamId=0, offset=0 on host=IP2:9997
02-09-2022 10:50:17.238 +0100 WARN TcpOutputProc - Possible duplication of events with channel=source::C:\Program Files\SplunkUniversalForwarder\var\log\splunk\health.log|host::807|splunkd|2713, streamId=0, offset=0 on host=IP2:9997
02-09-2022 10:50:17.238 +0100 WARN TcpOutputProc - Possible duplication of events with channel=source::WinEventLog:Security|host::807|XmlWinEventLog:Security|, streamId=3264402492634740844, offset=200186306 on host=IP2:9997
02-09-2022 10:50:17.238 +0100 WARN TcpOutputFd - Connect to IP1:9997 failed. En socketåtgärd försökte utföras till ett nätverk som inte går att kontakta.
02-09-2022 10:50:17.238 +0100 ERROR TcpOutputFd - Connection to host=IP1:9997 failed
02-09-2022 10:50:17.238 +0100 WARN TcpOutputFd - Connect to IP2:9997 failed. En socketåtgärd försökte utföras till ett nätverk som inte går att kontakta.
02-09-2022 10:50:17.238 +0100 ERROR TcpOutputFd - Connection to host=IP2:9997 failed
02-09-2022 10:50:17.238 +0100 WARN TcpOutputProc - Applying quarantine to ip=IP2 port=9997 _numberOfFailures=2
02-09-2022 10:50:17.238 +0100 WARN TcpOutputFd - Connect to IP3:9997 failed. En socketåtgärd försökte utföras till ett nätverk som inte går att kontakta.
02-09-2022 10:50:17.238 +0100 ERROR TcpOutputFd - Connection to host=IP3:9997 failed
02-09-2022 10:50:17.238 +0100 WARN TcpOutputFd - Connect to IP1:9997 failed. En socketåtgärd försökte utföras till ett nätverk som inte går att kontakta.
02-09-2022 10:50:17.238 +0100 ERROR TcpOutputFd - Connection to host=IP1:9997 failed
02-09-2022 10:50:17.238 +0100 WARN TcpOutputProc - Applying quarantine to ip=IP1 port=9997 _numberOfFailures=2

limits.conf

# [thruput]
# maxKBps = 0

The only thing i tested by myself so far is to add the servers to the host file, without any success.
I also noticed that from the outputs.conf its DNS names and in the log file its IP, but maybe that does not matter.

Any help would be much appreciated, 

Thanks in advance