When applying a change to search head cluster bundle, why am I receiving "Error in JSON response: Unexpected EOF"?

[606866581]

Hi all,
I've had a search head cluster running for a few months but haven't needed to use the deployer until recently. I have made a minor configuration change on the deployer in .../shcluster/apps/.../local/limits.conf and then tried to apply the bundle with:

splunk apply shcluster-bundle -target https://:8089 -preserve-lookups true

It runs for exactly 6 minutes before coming back with one of the below error messages. I suspect it is exceeding a timeout because it fails at such a round number:

• Error while deploying apps to target=https://:8089 with members=4: Error while updating app=$USERS on target=https://:8089: Error in JSON response: Unexpected EOF
• Error while creating deployable apps: Error clearing old_staging_area="/opt/splunk/var/run/splunk/deploy.3fb473d4b630bdf2.tmp": No such file or directory

If anyone has any suggestions for how to troubleshoot this issue please let me know. Thanks.

[horsefez]

Hi "random-number-guy",

do you specifiy a search-head cluster-member as the target?

example:

splunk apply shcluster-bundle -target https://splunksearchhead1:8089

[606866581]

It appears we meet all of these criteria. We have replication factor 2, they are all separate machines running Splunk 6.4 on the same OS. The deployer is an indexer, indexer cluster master node, and our licence server, but it is not a search head itself.

[606866581]

4 Members in total:

Captain:
dynamic_captain : 1
elected_captain : Wed Oct 12 09:49:16 2016
id : 2D8B0803-3A46-4B1C-80FB-3A883E4FFD8D
initialized_flag : 1
label : SHT45
maintenance_mode : 0
mgmt_uri : https://SH45:8089
min_peers_joined_flag : 1
rolling_restart_flag : 0
service_ready_flag : 1

Members:
SHR45
label : SHR45
mgmt_uri : https://SHR45.com:8089
mgmt_uri_alias : https://111.111.111.45:8089
status : Up
SHT45
label : SHT45
mgmt_uri : https://SHT45.com:8089
mgmt_uri_alias : https://111.111.112.45:8089
status : Up
SHR44
label : SHR44
mgmt_uri : https://SHR44.com:8089
mgmt_uri_alias : https://111.111.111.44:8089
status : Up
SHT45
label : SHT44
mgmt_uri : https://SHT44.com:8089
mgmt_uri_alias : https://111.111.112.44:8089
status : Up

[horsefez]

I don't get why you decided to only have 4 SH's. Not entirely sure this works!

Let's check the key requirements.

Summary of key requirements

These are the main issues to note regarding provisioning of cluster members:

Each member must run on its own machine or virtual machine, and all machines must run the same operating system.
All members must run on the same version of Splunk Enterprise.
All members must be connected over a high-speed network.
You must deploy at least as many members as either the replication factor or three, whichever is greater.

In addition to the cluster members, you need a deployer to distribute updates to the members. The deployer must run on a non-member instance. In some cases, it can run on the same instance as a deployment server or an indexer cluster master node.

[606866581]

I created a new file: /opt/splunk/etc/shcluster/apps/my-app/local/limits.conf with the stanza:
[default]
max_mem_usage_mb = 500

[606866581]

I do supply the full url of a SH member, yes

[horsefez]

Hi,
would you share the config change you did to the limits.conf?