Deployment Architecture

How to fix this Search head Clustering bundle Network-layer error: Read Timeout while applying bundle from deployer?

kchaitanya
Explorer

We have setup a Searchhead cluster for Enterprise Security (3 SHs) .. and receive the below error most of the times we push the bundle from deployer after making changes to the App under /opt/splunk/etc/shcluster/apps/xxxx/

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while deploying apps to first member: Error while updating app=SA-EndpointProtection on t

arget=https://xxxxxxx:8089: Network-layer error: Read Timeout

I have checked the connection between the deployer and SHs over 8089 which is good ... also provided the correct mgmt_uri of the captain ... I see that the changes are getting pushed to the SHs but the deployer does not provide the "bundle has been pushed successfully" message and waits on and then provide the ERROR message

The push worked at times .. i can say 2 out of 10 times and mostly give this error

Labels (1)
0 Karma

jkat54
SplunkTrust
SplunkTrust

you should be able to solve timeout problems by first insuring the ports are open.

If the ports are closed from your host you're executing this command on, then that will cause a "timeout" or "connection refused".

You can verify with openssl:

openssl s_client -connect https://yourSH:8089

if that opens a connection and reads a bunch of details about cryptography to you, you're good... if it fails with timeout or connection refused, the ports are blocked / or you cant otherwise route to "yourSH:8089".


If you're in a resource constrained environment and you absolutely must increase this timeout setting, then you do so by editing the following configuration item in web.conf:

so we need to edit web.conf:
[settings]
...
splunkdConnectionTimeout = <integer>
* The amount of time, in seconds, to wait before timing out when communicating with
  splunkd.
* Must be at least 30.
* Values smaller than 30 will be ignored, resulting in the use of the
  default value
* Default: 30
...

...and dont forget to restart!

I hope this helps!

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

Hi

it looks like the apps you are pushing take a little while to deploy
In order to wait for more time, try using
splunk apply sgcluster-bundle -target xxxxx:8089 -timeout 600
with the appropriate timeout value for your env

Tags (1)
0 Karma

kchaitanya
Explorer

looks like timeout is not a valid paremeter for the shcluster bundle command

kittu1991
New Member

Thank you for the reply.. i have already tried to increase the timeout from default to few mins(2 minutes)... but will try with the command including timeout ... hope this is not related to any other issue than time out

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...