topic Re: HF doesn't accept traffic from UF in Getting Data In

HF doesn't accept traffic from UF

slipinski — Mon, 15 Nov 2021 11:54:47 GMT

Hi Splunk chaps,

I'm facing problem with feeding HF from UF (HF is sending data to the cloud and this works fine). I can exclude network or firewall issue - both servers are reachable from opposite side.

Below is a chunk of log errors from UF :

11-15-2021 11:12:57.024 +0000 INFO DC:DeploymentClient [6735 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-15-2021 11:13:09.024 +0000 INFO DC:DeploymentClient [6735 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-15-2021 11:13:10.140 +0000 WARN HttpPubSubConnection [6734 HttpClientPollingThread_97C72192-9F2D-4883-830A-776376593AC1] - Unable to parse message from PubSubSvr:
11-15-2021 11:13:10.140 +0000 INFO HttpPubSubConnection [6734 HttpClientPollingThread_97C72192-9F2D-4883-830A-776376593AC1] - Could not obtain connection, will retry after=70.985 seconds.
11-15-2021 11:13:17.695 +0000 WARN TcpOutputProc [3551 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=172.23.11.216 inside output group default-autolb-group from host_src=ldcrapnvvip10 has been blocked for blocked_seconds=446600. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Please see output debug from UF.

/opt/splunkforwarder/etc/system/default/outputs.conf [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunkforwarder/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunkforwarder/etc/system/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunkforwarder/etc/system/local/outputs.conf defaultGroup = default-autolb-group
/opt/splunkforwarder/etc/system/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunkforwarder/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunkforwarder/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf useClientSSLCompression = true
/opt/splunkforwarder/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout-server://172.23.11.216:9997]
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout:default-autolb-group]
/opt/splunkforwarder/etc/system/local/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/local/outputs.conf server = 172.23.11.216:9997

Any ideas what blocks it?

thanks in advance,

Re: HF doesn't accept traffic from UF

slipinski — Mon, 15 Nov 2021 11:56:44 GMT

Below is input config of HF.

/opt/splunk/etc/system/default/inputs.conf [SSL]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf allowSslRenegotiation = true
/opt/splunk/etc/system/default/inputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
/opt/splunk/etc/system/default/inputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/inputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/run/splunk/search_telemetry/*search_telemetry.json]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _introspection
/opt/splunk/etc/system/default/inputs.conf log_on_completion = 0
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = search_telemetry
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/...stash_hec]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = stash_hec
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/...stash_new]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf crcSalt = <SOURCE>
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf queue = stashparsing
/opt/splunk/etc/system/default/inputs.conf sourcetype = stash_new
/opt/splunk/etc/system/default/inputs.conf time_before_close = 0
/opt/splunk/etc/system/default/inputs.conf [batch:///opt/splunk/var/spool/splunk/tracker.log*]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/system/default/inputs.conf move_policy = sinkhole
/opt/splunk/etc/system/default/inputs.conf sourcetype = splunkd_latency_tracker
/opt/splunk/etc/system/default/inputs.conf [blacklist:/opt/splunk/etc/auth]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf [blacklist:/opt/splunk/etc/passwd]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf [fschange:/opt/splunk/etc]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf delayInMills = 100
/opt/splunk/etc/system/default/inputs.conf disabled = false
/opt/splunk/etc/system/default/inputs.conf filesPerDelay = 10
/opt/splunk/etc/system/default/inputs.conf followLinks = false
/opt/splunk/etc/system/default/inputs.conf fullEvent = false
/opt/splunk/etc/system/default/inputs.conf hashMaxSize = -1
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf pollPeriod = 600
/opt/splunk/etc/system/default/inputs.conf recurse = true
/opt/splunk/etc/system/default/inputs.conf sendEventMaxSize = -1
/opt/splunk/etc/system/default/inputs.conf signedaudit = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf [http]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf ackIdleCleanup = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf allowSslCompression = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf allowSslRenegotiation = true
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf dedicatedIoThreads = 2
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf disabled = 1
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf enableSSL = 1
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf maxSockets = 0
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf maxThreads = 0
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf port = 8088
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf sslVersions = *,-ssl2
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf useDeploymentServer = 0
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/etc/splunk.version]
/opt/splunk/etc/system/default/inputs.conf _TCP_ROUTING = *
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/system/default/inputs.conf sourcetype = splunk_version
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf [monitor:///opt/splunk/var/log/introspection]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf index = _introspection
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/splunk]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/splunk/license_usage_summary.log]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _telemetry
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/splunk/splunk_instrumentation_cloud.log*]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _telemetry
/opt/splunk/etc/system/default/inputs.conf sourcetype = splunk_cloud_telemetry
/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/watchdog/watchdog.log*]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = _internal
/opt/splunk/etc/apps/search/local/inputs.conf [monitor:///var/log/secure]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/search/local/inputs.conf disabled = false
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/apps/search/local/inputs.conf index = discol
/opt/splunk/etc/system/default/inputs.conf [script]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf host = $decideOnStartup
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/system/default/inputs.conf interval = 60.0
/opt/splunk/etc/system/default/inputs.conf start_by_shell = true

Re: HF doesn't accept traffic from UF

gcusello — Mon, 15 Nov 2021 12:04:45 GMT

Hi @slipinski,

a very stupid question:

did you enabled Receiving [Settings -- Forwarding and Receiving -- Receiving] and Forwarding [Settings -- Forwarding and Receiving -- Forwarding] on the HFs?

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

slipinski — Mon, 15 Nov 2021 12:24:46 GMT

Hi @gcusello. I think I did, but currently doesn't have access to webgui. Can I confirm these settings in CLI?

regards,

Re: HF doesn't accept traffic from UF

gcusello — Mon, 15 Nov 2021 13:04:35 GMT

hi @slipinski,

you can check receiving vieving in $SPLUNK_HOME/etc/system/local/inputs.conf if you have the stanza

[splunktcp://9997] disabled = 0

you can check forwarding vieving in $SPLUNK_HOME/etc/system/local/outputs.conf if you have the stanza

[tcpout] defaultGroup=my_indexers [tcpout:my_indexers] server=mysplunk_indexer1:9997, mysplunk_indexer2:9996 [tcpout-server://mysplunk_indexer1:9997]

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

slipinski — Mon, 15 Nov 2021 14:28:08 GMT

Either input and output seem to be ok.

Re: HF doesn't accept traffic from UF

gcusello — Mon, 15 Nov 2021 14:33:54 GMT

Hi @slipinski,

did you checked local firewalls (iptables) on HFs?

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

slipinski — Mon, 15 Nov 2021 14:36:34 GMT

Yes, I allowed traffic on these 3 ports, just to be on safe side

firewall-cmd --zone=public --permanent --add-port=8000/tcp
firewall-cmd --zone=public --permanent --add-port=9997/tcp
firewall-cmd --zone=public --permanent --add-port=8089/tcp

Re: HF doesn't accept traffic from UF

gcusello — Mon, 15 Nov 2021 14:39:46 GMT

Hi @slipinski,

using Telnet on one Universal Forwarder, what does it happen if you run:

telnet <HF_IP_Address> 9997

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

slipinski — Mon, 15 Nov 2021 14:41:08 GMT

It works.

Re: HF doesn't accept traffic from UF

gcusello — Mon, 15 Nov 2021 14:47:00 GMT

Hi @slipinski,

what does it happen if you run the following searches on your Splunk Cloud:

index=_internal host=<hostname_Heavy_Forwarder> index=_internal host=<hostname_Universal_Forwarder> index=* host=<hostname_Universal_Forwarder>

if you haven't results, probablky the problem is in the connection between HFs and Splunk Cloud.

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

slipinski — Mon, 15 Nov 2021 14:51:26 GMT

I added one logfile to being monitored under HF and can see results in a cloud.

Re: HF doesn't accept traffic from UF

gcusello — Mon, 15 Nov 2021 15:21:41 GMT

Hi @slipinski,

what does it happen if you run the above searches on Splunk Cloud?

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

slipinski — Tue, 16 Nov 2021 08:03:43 GMT

index=_internal host=<hostname_Heavy_Forwarder> OK index=_internal host=<hostname_Universal_Forwarder> Nothing index=* host=<hostname_Universal_Forwarder> Nothing

@gcusello Only data from HF are visible in the cloud.

Re: HF doesn't accept traffic from UF

gcusello — Tue, 16 Nov 2021 08:37:58 GMT

Hi @slipinski,

ok, summarizing the analysis:

HF data are sent to Cloud,
UF data aren't sent to Cloud (both internal and external data),
firewall routes between UFs and HFs are open (telnet test),
HFs are open to receive data,
UFs have the correct outputs.conf (addressing the HFs in outputs.conf).

The strange thing is that you haven't neither _internal and external logs from UFs.

Could you share the outputs.conf and inputs.conf of UFs?

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

slipinski — Tue, 16 Nov 2021 08:45:03 GMT

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = 172.23.11.216:9997

[tcpout-server://172.23.11.216:9997]

172.23.11.216 is obviously address of HW.

Re: HF doesn't accept traffic from UF

isoutamo — Tue, 16 Nov 2021 09:13:40 GMT

If you cannot found nothing from HFs logs related to UF connections then I propose that it's time for tcpdump to check if there is any traffic towards HF.

r. Ismo

Re: HF doesn't accept traffic from UF

slipinski — Tue, 16 Nov 2021 11:46:10 GMT

Yes, there is. However, HF replys to UF with [RST,ACK] packets. This generally means that port is closed, but in reality it isn't. As I mentioned before, I can telnet to HF on port 8089.

Re: HF doesn't accept traffic from UF

gcusello — Tue, 16 Nov 2021 11:57:49 GMT

Hi @slipinski,

please try telnet on port 9997.

Ciao.

Giuseppe

Re: HF doesn't accept traffic from UF

PickleRick — Tue, 16 Nov 2021 12:33:35 GMT

Exactly. That's the first step with any connection problems. Dump the traffic on the appropriate interface and see whether any connection tries even take place.

tcpdump/wireshark is your greatest friend with network/connection troubleshooting.