Getting Data In

Splunk Forwarder - Deployment Apps not being downloaded

ahmadgul21
Explorer

Hi,

The issue is that some servers with universal forwarder agent deployed on them are not being able to successfully download the apps from the deployment server. 

Environment Details:

Server: Linux RHEL 7.9 (3.x Kernel)
Deployment Server: Splunk Enterprise 8.x
Splunk Universal Forwarder: 8.2.2 for Linux

The agent is successfully installed and connected to the deployment server using the below command

./splunk set deploy-poll depoloyment-server:8089

And it is showing up successfully on the deployment server as well however when I push apps to the server via the deployment server they aren't successfully downloaded. 

From the universal forwarder splunkd.log, 

ERROR HttpClientRequest *** - HTTP client error=Connection closed by peer while accessing server=*** for request=***

From the deployment server splunkd.log,

ahmadgul21_0-1629811363334.png

What can be the possible reason for this behavior? Since the communication seems fine (we've opened uni-directional communication from server to deployment-server on port 8089). 

Kind regards

Labels (2)
Tags (2)
0 Karma

ahmadgul21
Explorer

An update,

This issue was fixed I believe by using newer version (8.1.1) of the Splunk Agent and the Apps were successfully downloaded from the Splunk DS. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
One comment about push vs pull. All traffic are initiated by client never by DS! The DS just listened and respond to clients initiated request.
r. Ismo
0 Karma

ephemeric
Contributor

Seems like your connection is broken somehow. You said that "some" of the UFs are failing. I think you meant apps are pulled from the DS, not pushed from the DS to UFs.

Can you check your firewall log for any errors?

Please try from your failing UF:

echo '\n' | openssl s_client -connect <deployment-server>:8089

 

0 Karma

ahmadgul21
Explorer

Well, It is my understanding that the apps are pushed from the deployment server to the universal forwarders when they phone home. For example, a newly deployed universal forwarder once it phones home and the deployment server has assigned it to a server class then all the apps of that server class are pushed to the new universal forwarder, is that right?

Also, I tried manually configuring the outputs.conf and inputs.conf in /opt/splunkforwarder/etc/system/local and have verified that I can successfully receive the logs being monitored. 

The only major issue right now is that the universal forwarder is still not able to download the apps from the deployment server even though both telnet and the command that you shared earlier show that connectivity is successful.

The below command results in output of "CONNECTED" .... "Verify Return Code: 19 (Self-Signed Certificate in Certificate Chain)" .... "DONE"

echo '\n' | openssl s_client -connect <deployment-server>:8089

Can there be any issue due to the universal forwarder being version 8.2.2 and deployment server being version 8.1.1 ? 

0 Karma

ephemeric
Contributor

I can only find:

https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwar...

WRT compatibility.

Can anyone confirm compatibility between UFs and DSs?

0 Karma
Get Updates on the Splunk Community!

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...

Thank You for Celebrating CX Day with Splunk!

Yesterday the entire team at Splunk &#43; Cisco joined the global celebration of CX Day - celebrating our ...