Splunk Forwarder - Deployment Apps not being downloaded

[ahmadgul21]

Hi,

The issue is that some servers with universal forwarder agent deployed on them are not being able to successfully download the apps from the deployment server. 

Environment Details:

Server: Linux RHEL 7.9 (3.x Kernel)
Deployment Server: Splunk Enterprise 8.x
Splunk Universal Forwarder: 8.2.2 for Linux

The agent is successfully installed and connected to the deployment server using the below command

./splunk set deploy-poll depoloyment-server:8089

And it is showing up successfully on the deployment server as well however when I push apps to the server via the deployment server they aren't successfully downloaded. 

From the universal forwarder splunkd.log, 

ERROR HttpClientRequest *** - HTTP client error=Connection closed by peer while accessing server=*** for request=***

From the deployment server splunkd.log,

What can be the possible reason for this behavior? Since the communication seems fine (we've opened uni-directional communication from server to deployment-server on port 8089). 

Kind regards

[ahmadgul21]

An update,

This issue was fixed I believe by using newer version (8.1.1) of the Splunk Agent and the Apps were successfully downloaded from the Splunk DS. 

[isoutamo]

One comment about push vs pull. All traffic are initiated by client never by DS! The DS just listened and respond to clients initiated request.
r. Ismo

[ephemeric]

Seems like your connection is broken somehow. You said that "some" of the UFs are failing. I think you meant apps are pulled from the DS, not pushed from the DS to UFs.

Can you check your firewall log for any errors?

Please try from your failing UF:

echo '\n' | openssl s_client -connect <deployment-server>:8089

[ahmadgul21]

Well, It is my understanding that the apps are pushed from the deployment server to the universal forwarders when they phone home. For example, a newly deployed universal forwarder once it phones home and the deployment server has assigned it to a server class then all the apps of that server class are pushed to the new universal forwarder, is that right?

Also, I tried manually configuring the outputs.conf and inputs.conf in /opt/splunkforwarder/etc/system/local and have verified that I can successfully receive the logs being monitored. 

The only major issue right now is that the universal forwarder is still not able to download the apps from the deployment server even though both telnet and the command that you shared earlier show that connectivity is successful.

The below command results in output of "CONNECTED" .... "Verify Return Code: 19 (Self-Signed Certificate in Certificate Chain)" .... "DONE"

echo '\n' | openssl s_client -connect <deployment-server>:8089

Can there be any issue due to the universal forwarder being version 8.2.2 and deployment server being version 8.1.1 ? 

[ephemeric]

I can only find:

https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwar...

WRT compatibility.

Can anyone confirm compatibility between UFs and DSs?