9.1.3/9.2.1 onwards slow indexer/receiver detection capability is fully functional now (SPL-248188, SPL-248140). https://docs.splunk.com/Documentation/Splunk/9.2.1/ReleaseNotes/Fixedissues You can enable it on forwarding side in outputs.conf maxSendQSize = <integer>
* The size of the tcpout client send buffer, in bytes.
If tcpout client(indexer/receiver connection) send buffer is full,
a new indexer is randomly selected from the list of indexers provided
in the server setting of the target group stanza.
* This setting allows forwarder to switch to new indexer/receiver if current
indexer/receiver is slow.
* A non-zero value means that max send buffer size is set.
* 0 means no limit on max send buffer size.
* Default: 0 Additionally 9.1.3/9.2.1 and above will correctly log target ipaddress causing tcpout blocking. WARN AutoLoadBalancedConnectionStrategy [xxxx TcpOutEloop] - Current dest host connection nn.nn.nn.nnn:9997, oneTimeClient=0, _events.size()=20, _refCount=2, _waitingAckQ.size()=4, _supportsACK=1, _lastHBRecvTime=Thu Jan 20 11:07:43 2024 is using 20214400 bytes. Total tcpout queue size is 26214400. Warningcount=20 Note: This config works correctly starting 9.1.3/9.2.1. Do not use it with 9.2.0/9.1.0/9.1.1/9.1.2( there is incorrect calculation https://community.splunk.com/t5/Getting-Data-In/Current-dest-host-connection-is-using-18446603427033668018-bytes/m-p/678842#M113450).
... View more