Hi. What version of Splunk are you running? I ran into a bad bug with both Splunk Enterprise 9.3.7 and 9.4.5. The heavy forwarders sending to DNS load balanced indexers get TCPOUT blocked. This bug does not appear to be on the known issues despite many attempts by me trying to get it added there. It does not happen with 9.2.4. The Splunk JIRA that was opened is SPL-288904 The bug is said to be fixed in the upcoming releases 9.3.9 , 9.4.7, 10.0.3, 10.1 Hopefully soon. A workaround is the setting of dnsResolutionInterval in outputs.conf dnsResolutionInterval = <integer>
* The base time interval, in seconds, at which indexer Domain Name Server
(DNS) names are resolved to IP addresses.
* This is used to compute runtime dnsResolutionInterval as follows:
Runtime interval =
'dnsResolutionInterval' + (number of indexers in server settings - 1) * 30.
* The DNS resolution interval is extended by 30 seconds for each additional
indexer in the server setting.
* Default: 300 seconds (5 minutes) Splunk had recommended we set dnsResolutionInterval =480 (tcpout blocked). I tried 1000 (also blocked). I have set it to 10000 (ie 10,000) and after ~ 3 days this seems to be working.
... View more