HF doesn't accept traffic from UF

[slipinski]

Hi Splunk chaps, 

I'm facing problem with feeding HF from UF (HF is sending data to the cloud and this works fine).  I can exclude network or firewall issue - both servers are reachable from opposite side. 

Below is a chunk of log errors from UF : 

11-15-2021 11:12:57.024 +0000 INFO DC:DeploymentClient [6735 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-15-2021 11:13:09.024 +0000 INFO DC:DeploymentClient [6735 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-15-2021 11:13:10.140 +0000 WARN HttpPubSubConnection [6734 HttpClientPollingThread_97C72192-9F2D-4883-830A-776376593AC1] - Unable to parse message from PubSubSvr:
11-15-2021 11:13:10.140 +0000 INFO HttpPubSubConnection [6734 HttpClientPollingThread_97C72192-9F2D-4883-830A-776376593AC1] - Could not obtain connection, will retry after=70.985 seconds.
11-15-2021 11:13:17.695 +0000 WARN TcpOutputProc [3551 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=172.23.11.216 inside output group default-autolb-group from host_src=ldcrapnvvip10 has been blocked for blocked_seconds=446600. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Please see output debug from UF. 

/opt/splunkforwarder/etc/system/default/outputs.conf [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunkforwarder/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunkforwarder/etc/system/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunkforwarder/etc/system/local/outputs.conf defaultGroup = default-autolb-group
/opt/splunkforwarder/etc/system/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunkforwarder/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunkforwarder/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf useClientSSLCompression = true
/opt/splunkforwarder/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout-server://172.23.11.216:9997]
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout:default-autolb-group]
/opt/splunkforwarder/etc/system/local/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/local/outputs.conf server = 172.23.11.216:9997

Any ideas what blocks it? 

thanks in advance,

Sz

[slipinski]

Below is input config of HF.

/opt/splunk/etc/system/default/inputs.conf [SSL]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf allowSslRenegotiation = true
/opt/splunk/etc/system/default/inputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
/opt/splunk/etc/system/default/inputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/inputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/run/splunk/search_telemetry/*search_telemetry.json]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _introspection
/opt/splunk/etc/system/default/inputs.conf log_on_completion = 0
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = search_telemetry
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/...stash_hec]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = stash_hec
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/...stash_new]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf queue = stashparsing
/opt/splunk/etc/system/default/inputs.conf sourcetype = stash_new
/opt/splunk/etc/system/default/inputs.conf time_before_close = 0
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/tracker.log*]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = splunkd_latency_tracker
/opt/splunk/etc/system/default/inputs.conf [blacklist:/opt/splunk/etc/auth]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf [blacklist:/opt/splunk/etc/passwd]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf [fschange:/opt/splunk/etc]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf delayInMills = 100
/opt/splunk/etc/system/default/inputs.conf disabled = false
/opt/splunk/etc/system/default/inputs.conf filesPerDelay = 10
/opt/splunk/etc/system/default/inputs.conf followLinks = false
/opt/splunk/etc/system/default/inputs.conf fullEvent = false
/opt/splunk/etc/system/default/inputs.conf hashMaxSize = -1
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf pollPeriod = 600
/opt/splunk/etc/system/default/inputs.conf recurse = true
/opt/splunk/etc/system/default/inputs.conf sendEventMaxSize = -1
/opt/splunk/etc/system/default/inputs.conf signedaudit = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf [http]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf ackIdleCleanup = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf allowSslCompression = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf allowSslRenegotiation = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf dedicatedIoThreads = 2
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf disabled = 1
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf enableSSL = 1
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf maxSockets = 0
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf maxThreads = 0
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf port = 8088
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf sslVersions = *,-ssl2
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf useDeploymentServer = 0
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/etc/splunk.version]
/opt/splunk/etc/system/default/inputs.conf _TCP_ROUTING = *
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/system/default/inputs.conf sourcetype = splunk_version
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf [monitor:///opt/splunk/var/log/introspection]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf index = _introspection
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/splunk]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/splunk/license_usage_summary.log]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _telemetry
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/splunk/splunk_instrumentation_cloud.log*]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _telemetry
/opt/splunk/etc/system/default/inputs.conf sourcetype = splunk_cloud_telemetry
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/watchdog/watchdog.log*]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/apps/search/local/inputs.conf [monitor:///var/log/secure]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/search/local/inputs.conf disabled = false
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/apps/search/local/inputs.conf index = discol
/opt/splunk/etc/system/default/inputs.conf [script]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf interval = 60.0
/opt/splunk/etc/system/default/inputs.conf start_by_shell = true

[gcusello]

Hi @slipinski,

a very stupid question:

did you enabled Receiving [Settings -- Forwarding and Receiving -- Receiving] and Forwarding [Settings -- Forwarding and Receiving -- Forwarding] on the HFs?

Ciao.

Giuseppe

[slipinski]

Hi @gcusello. I think I did, but currently doesn't have access to webgui. Can I confirm these settings in CLI?

regards,

Sz 

[gcusello]

hi @slipinski,

you can check receiving vieving in $SPLUNK_HOME/etc/system/local/inputs.conf if you have the stanza 

[splunktcp://9997]
disabled = 0

you can check forwarding  vieving in $SPLUNK_HOME/etc/system/local/outputs.conf if you have the stanza 

[tcpout]
defaultGroup=my_indexers

[tcpout:my_indexers]
server=mysplunk_indexer1:9997, mysplunk_indexer2:9996

[tcpout-server://mysplunk_indexer1:9997]

Ciao.

Giuseppe

[slipinski]

Either input and output seem to be ok.

[gcusello]

Hi @slipinski,

did you checked local firewalls (iptables) on HFs?

Ciao.

Giuseppe

[slipinski]

Yes, I allowed traffic on these 3 ports, just to be on safe side

firewall-cmd --zone=public --permanent --add-port=8000/tcp
firewall-cmd --zone=public --permanent --add-port=9997/tcp
firewall-cmd --zone=public --permanent --add-port=8089/tcp

[gcusello]

Hi @slipinski,

using Telnet on one Universal Forwarder, what does it happen if you run:

telnet <HF_IP_Address> 9997

Ciao.

Giuseppe

[slipinski]

It works.

[gcusello]

Hi @slipinski,

what does it happen if you run the following searches on your Splunk Cloud:

index=_internal host=<hostname_Heavy_Forwarder>
index=_internal host=<hostname_Universal_Forwarder>
index=* host=<hostname_Universal_Forwarder>

if you haven't results, probablky the problem is in the connection between HFs and Splunk Cloud.

Ciao.

Giuseppe

[slipinski]

I added one logfile to being monitored under HF and can see results in a cloud.

[gcusello]

Hi @slipinski,

what does it happen if you run the above searches on Splunk Cloud?

Ciao.

Giuseppe

[slipinski]

index=_internal host=<hostname_Heavy_Forwarder>  OK
index=_internal host=<hostname_Universal_Forwarder> Nothing
index=* host=<hostname_Universal_Forwarder> Nothing

@gcusello  Only data from HF are visible in the cloud. 

[gcusello]

Hi @slipinski,

ok, summarizing the analysis:

• HF data are sent to Cloud,
• UF data aren't sent to Cloud (both internal and external data),
• firewall routes between UFs and HFs are open (telnet test),
• HFs are open to receive data,
• UFs have the correct outputs.conf (addressing the HFs in outputs.conf).

The strange thing is that you haven't neither _internal and external logs from UFs.

Could you share the outputs.conf and inputs.conf of UFs?

Ciao.

Giuseppe

[slipinski]

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = 172.23.11.216:9997

[tcpout-server://172.23.11.216:9997]

 172.23.11.216 is obviously address of HW. 

[isoutamo]

If you cannot found nothing from HFs logs related to UF connections then I propose that it's time for tcpdump to check if there is any traffic towards HF.

r. Ismo

[PickleRick]

Exactly. That's the first step with any connection problems. Dump the traffic on the appropriate interface and see whether any connection tries even take place.

tcpdump/wireshark is your greatest friend with network/connection troubleshooting.

[slipinski]

Yes, there is. However, HF replys to UF with [RST,ACK] packets. This generally means that port is closed, but in reality it isn't. As I mentioned before, I can telnet to HF on port 8089.

[PickleRick]

You get ACK/RST as first response or after initial 3-way handshake and some trafifc exchange?

Maybe you have some mismatch in TLS settings between HF and UF?

[slipinski]

@PickleRick ACK/RST packet turns up after 3-way handshake. 

I caught some TLSv1 packets. Looks like TLS hello packet from UF towards HF, so your guess related to TLS mismatch can be right.