Deployment Architecture

Slow/Unresponsive Deployment Server. Deployment Clients timing out.

hrawat_splunk
Splunk Employee
Splunk Employee

There are 503 errors and DC can't connect messages in splunkd.log on deployment client.

Labels (2)
0 Karma
1 Solution

hrawat_splunk
Splunk Employee
Splunk Employee

That is because with default DS configuration, DS has reached a saturation point of handling too many phone home checks. Here are the recommendations to increase DS scalability.

On the DS server.conf

[sslConfig]

sslServerSessionTimeout = 7200

[httpServer]

dedicatedIoThreads = 10

 

Set following config to 512.

In $SPLUNK_HOME/etc/splunk-launch.conf on DS.

SPLUNK_LISTEN_BACKLOG = 512

Note: Make sure linux net.core.somaxconn setting is more than SPLUNK_LISTEN_BACKLOG.

 

On DC side  server.conf

[sslConfig]

useSslClientSessionCache=true

Upgrade all DC to 7.1.3 and above to have configurable and higher default DC timeouts. Before 7.1.3 these are hardcoded 5 sec.

connect_timeout = <positive integer>

* Default: 60

send_timeout = <positive integer>

* Default: 60

recv_timeout = <positive integer>

* Default: 60

View solution in original post

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

That is because with default DS configuration, DS has reached a saturation point of handling too many phone home checks. Here are the recommendations to increase DS scalability.

On the DS server.conf

[sslConfig]

sslServerSessionTimeout = 7200

[httpServer]

dedicatedIoThreads = 10

 

Set following config to 512.

In $SPLUNK_HOME/etc/splunk-launch.conf on DS.

SPLUNK_LISTEN_BACKLOG = 512

Note: Make sure linux net.core.somaxconn setting is more than SPLUNK_LISTEN_BACKLOG.

 

On DC side  server.conf

[sslConfig]

useSslClientSessionCache=true

Upgrade all DC to 7.1.3 and above to have configurable and higher default DC timeouts. Before 7.1.3 these are hardcoded 5 sec.

connect_timeout = <positive integer>

* Default: 60

send_timeout = <positive integer>

* Default: 60

recv_timeout = <positive integer>

* Default: 60

View solution in original post

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!