Why are peer nodes not receiving data from Universal forwarder?

[petops147]

Hello, I recently setup a test environment(clustered deployment) on  AWS  to monitor and get data into the peer nodes.

My environment include: cluster master(hosting license), 3 Indexers, 1 Deployer, 3 Search heads, 1 Deployment server and 2 Universal forwarders.

 I configured the deployment server to push configuration to the forwarders, and all seem working fine; the forwarders are phoning home, their is sync between the DS and the UFs. But the peer nodes are not receiving the data. even though, I set up the listening port (9997).

I did troubleshoot on the UF to see if they are pushing, excerpt of the output from the UFs:

05-05-2022 11:02:11.118 +0000 INFO HttpPubSubConnection [3276 HttpClientPollingThread_71E59550-AD46-4814-8460-DB66C1DD0BAD] - Running phone uri=/services/broker/phonehome/connection_172.31.28.182_8089_ip-172-31-28-182.ec2.internal_uf01_71E59550-AD46-4814-8460-DB66C1DD0BAD
05-05-2022 11:02:25.767 +0000 INFO TailReader [3323 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
05-05-2022 11:02:31.076 +0000 INFO AutoLoadBalancedConnectionStrategy [3316 TcpOutEloop] - Connected to idx=172.31.21.254:9997, pset=0, reuse=0. using ACK.
05-05-2022 11:02:55.767 +0000 INFO TailReader [3323 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
05-05-2022 11:03:01.006 +0000 INFO AutoLoadBalancedConnectionStrategy [3316 TcpOutEloop] - Connected to idx=172.31.22.208:9997, pset=0, reuse=0. using ACK.
05-05-2022 11:03:11.118 +0000 INFO HttpPubSubConnection [3276 HttpClientPollingThread_71E59550-AD46-4814-8460-DB66C1DD0BAD] - Running phone uri=/services/broker/phonehome/connection_172.31.28.182_8089_ip-172-31-28-182.ec2.internal_uf01_71E59550-AD46-4814-8460-DB66C1DD0BAD
05-05-2022 11:03:25.597 +0000 INFO TailReader [3323 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
05-05-2022 11:03:30.890 +0000 INFO AutoLoadBalancedConnectionStrategy [3316 TcpOutEloop] - After randomization, current is first in the list. Swapping with last item
05-05-2022 11:03:30.891 +0000 INFO AutoLoadBalancedConnectionStrategy [3316 TcpOutEloop] - Connected to idx=172.31.21.254:9997, pset=0, reuse=1.
05-05-2022 11:03:55.596 +0000 INFO TailReader [3323 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
05-05-2022 11:04:00.813 +0000 INFO AutoLoadBalancedConnectionStrategy [3316 TcpOutEloop] - Connected to idx=172.31.18.160:9997, pset=0, reuse=0. using ACK.
05-05-2022 11:04:11.124 +0000 INFO HttpPubSubConnection [3276 HttpClientPollingThread_71E59550-AD46-4814-8460-DB66C1DD0BAD] - Running phone uri=/services/broker/phonehome/connection_172.31.28.182_8089_ip-172-31-28-182.ec2.internal_uf01_71E59550-AD46-4814-8460-DB66C1DD0BAD
05-05-2022 11:04:25.596 +0000 INFO TailReader [3323 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
05-05-2022 11:04:30.704 +0000 INFO AutoLoadBalancedConnectionStrategy [3316 TcpOutEloop] - Connected to idx=172.31.22.208:9997, pset=0, reuse=0. using ACK.
05-05-2022 11:04:55.596 +0000 INFO TailReader [3323 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
05-05-2022 11:05:00.613 +0000 INFO AutoLoadBalancedConnectionStrategy [3316 TcpOutEloop] - Connected to idx=172.31.18.160:9997, pset=0, reuse=1.
05-05-2022 11:05:11.129 +0000 INFO HttpPubSubConnection [3276 HttpClientPollingThread_71E59550-AD46-4814-8460-DB66C1DD0BAD] - Running phone 

Any idea on solution to this ?

[richgalloway]

Look for TcpOutputProc messages on the UF to check connectivity to indexers.

Confirm the instances do not have firewalls or the firewalls are allowing connections.

Double-check the outputs.conf settings on the UFs to make sure the right values are used in the server setting(s).

[petops147]

Hello rich,

Can you check this output to see if you can make any meaning out it:

05-07-2022 05:40:15.060 +0000 WARN TcpOutputFd [3311 TcpOutEloop] - Connect to 172.31.18.160:9997 failed. Connection refused
05-07-2022 05:40:15.060 +0000 ERROR TcpOutputFd [3311 TcpOutEloop] - Connection to host=172.31.18.160:9997 failed
05-07-2022 05:40:15.061 +0000 WARN TcpOutputFd [3311 TcpOutEloop] - Connect to 172.31.22.208:9997 failed. Connection refused
05-07-2022 05:40:15.061 +0000 ERROR TcpOutputFd [3311 TcpOutEloop] - Connection to host=172.31.22.208:9997 failed
05-07-2022 05:40:15.062 +0000 WARN TcpOutputFd [3311 TcpOutEloop] - Connect to 172.31.21.254:9997 failed. Connection refused
05-07-2022 05:40:15.062 +0000 ERROR TcpOutputFd [3311 TcpOutEloop] - Connection to host=172.31.21.254:9997 failed
05-07-2022 05:40:15.063 +0000 WARN TcpOutputFd [3311 TcpOutEloop] - Connect to 172.31.21.254:9997 failed. Connection refused
05-07-2022 05:40:15.063 +0000 ERROR TcpOutputFd [3311 TcpOutEloop] - Connection to host=172.31.21.254:9997 failed
05-07-2022 05:40:15.063 +0000 WARN TcpOutputFd [3311 TcpOutEloop] - Connect to 172.31.18.160:9997 failed. Connection refused
05-07-2022 05:40:15.063 +0000 ERROR TcpOutputFd [3311 TcpOutEloop] - Connection to host=172.31.18.160:9997 failed
05-07-2022 05:40:15.063 +0000 WARN TcpOutputFd [3311 TcpOutEloop] - Connect to 172.31.22.208:9997 failed. Connection refused
05-07-2022 05:40:15.064 +0000 ERROR TcpOutputFd [3311 TcpOutEloop] - Connection to host=172.31.22.208:9997 failed
05-07-2022 06:44:57.434 +0000 INFO loader [3846 MainThread] - SAML cert db registration with KVStore failed
05-07-2022 06:44:57.434 +0000 INFO loader [3846 MainThread] - Auth cert db registration with KVStore failed
05-07-2022 06:44:57.434 +0000 INFO loader [3846 MainThread] - JsonWebToken Manager registration with KVStore failed.

[petops147]

Thank you rich!

However, I'm still stuck.

I issue this command but no output messages; it was blank:

[splunkforwarder@uf01 splunk]$
[splunkforwarder@uf01 splunk]$ tail -f splunkd.log | egrep 'TcpOutputProc|TcpOutputFd'

no messages came up.

I have checked the security to see if any port is not allowed; everything seem alright

Note: I am pretty new to splunk 

[isoutamo]

Hi

As you are using DS server you probably have some apps to do the base configurations like connection to DS and output to indexer peers? Are you using indexer discovery (probably better option in AWS as default behaviour is that indexers' IP will change) or fixed IP/fqdn names on outputs.conf?

Or how you have configured UF to IDXc output connection?

r. Ismo

[petops147]

Thank you for the response.

Though I am new to splunk; however, here is how i configured the inputs.conf and outputs.conf in the Deployment-app subdirectory. the clients are phoning home /polling the DS and I can see the distributed Apps in the UF /etc/apps/ only that the  data I am monitoring are not getting into the  peer nodes or search peers. I have 3 Search Head Cluster members, 3 Peer nodes , 1 Deployer, 1 Cluster master/master node, 2 Universal forwarder, 1 DS in my AWS splunk environment.

1, I created a sub directory(fwd_receivers) in deployment-apps directory  :

 deployment-apps>fwd_receivers>default>output.conf  

2. I created another subdirectory(mon_input) in the deployment-apps directory for monitoring inputs:

deployment-apps>mon_input>default>inputs.conf

* I am wondering if I am using the right path "default" instead of  "local"? splunk best practices is to only modify in the local directory and not in the default. I did my config based on some video tutorials I watched.

[isoutamo]

Those apps sounds to be ok. And you have probably added correct serverclasess to bind those to correct server as you can see show on UF's disk.

When you are creating those apps by yourself then default is the correct place for those configurations. Only passwords etc. which you want to crypt on target UF must be on local folder.

Can you see those internal events on your splunk server or only in UF's filesystem?

index=_internal host=<your UF name/ip>

r. Ismo

[petops147]

index=_internal host=<your UF name/ip>

Yes I can see both UFs. And can be seen as the defined names uf01 & uf02. 

I can practically see all Splunk component when issue the command : index = _internal .