Deployment Architecture

How to fix this Search head Clustering bundle Network-layer error: Read Timeout while applying bundle from deployer?

kchaitanya
Explorer

We have setup a Searchhead cluster for Enterprise Security (3 SHs) .. and receive the below error most of the times we push the bundle from deployer after making changes to the App under /opt/splunk/etc/shcluster/apps/xxxx/

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while deploying apps to first member: Error while updating app=SA-EndpointProtection on t

arget=https://xxxxxxx:8089: Network-layer error: Read Timeout

I have checked the connection between the deployer and SHs over 8089 which is good ... also provided the correct mgmt_uri of the captain ... I see that the changes are getting pushed to the SHs but the deployer does not provide the "bundle has been pushed successfully" message and waits on and then provide the ERROR message

The push worked at times .. i can say 2 out of 10 times and mostly give this error

Labels (1)
0 Karma

jkat54
SplunkTrust
SplunkTrust

you should be able to solve timeout problems by first insuring the ports are open.

If the ports are closed from your host you're executing this command on, then that will cause a "timeout" or "connection refused".

You can verify with openssl:

openssl s_client -connect https://yourSH:8089

if that opens a connection and reads a bunch of details about cryptography to you, you're good... if it fails with timeout or connection refused, the ports are blocked / or you cant otherwise route to "yourSH:8089".


If you're in a resource constrained environment and you absolutely must increase this timeout setting, then you do so by editing the following configuration item in web.conf:

so we need to edit web.conf:
[settings]
...
splunkdConnectionTimeout = <integer>
* The amount of time, in seconds, to wait before timing out when communicating with
  splunkd.
* Must be at least 30.
* Values smaller than 30 will be ignored, resulting in the use of the
  default value
* Default: 30
...

...and dont forget to restart!

I hope this helps!

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

Hi

it looks like the apps you are pushing take a little while to deploy
In order to wait for more time, try using
splunk apply sgcluster-bundle -target xxxxx:8089 -timeout 600
with the appropriate timeout value for your env

Tags (1)
0 Karma

kchaitanya
Explorer

looks like timeout is not a valid paremeter for the shcluster bundle command

kittu1991
New Member

Thank you for the reply.. i have already tried to increase the timeout from default to few mins(2 minutes)... but will try with the command including timeout ... hope this is not related to any other issue than time out

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...