Deployment Architecture
Highlighted

Why is the deployment server unable to push apps to search head clusters?

Communicator

Hello !!

I have created multiple apps under $SPLUNKHOME/etc/shcluster/apps/ on the deployment server and checked the permissions on the directories and files under the apps. I have placed the files under local subdirectory in the SPLUNKHOME/etc/shcluster/apps/{apps}/local/. Also checked the pass4Symmkey in the server.conf under /opt/splunk/etc/system/local/ on the search head deploy and on the search head cluster.

When executed the command that is /splunk apply shcluster-bundle -target https://{search_head_cluster_captain_ip_address}:port -auth {username}:{password}

Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://{search_head_cluster_captain_ip_address}:{port}: Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}.

I can see the apps staged under $SPLUNK_HOME/var/run/splunk/deploy/ but I'm unable to resolve it.

I have restarted the Splunk service and also the instance too but no luck. Every time I try to execute the command it always gives the above error.

When checked in the documentation they referred to this error because of the mismatch of Pass4Symmkey between the search head deploy and search head cluster.

Let me know where I can start debugging this issue. I have checked in the logs too on the search head deploy but it doesn't help since it only says about the above error.

Thanks

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Champion

Is this a new cluster and the first time you're trying to deploy apps to it? Or was this a working cluster that started throwing this error?

And to be that guy for a moment: be careful about using "deployment server" and "deployer" names interchangeably - they are two different things and may lead to confusion if you say one and mean the other.

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Communicator

Yes, this is a new cluster and deploying apps for the first time. I'm trying to replicate design of the production splunk environment such that development splunk environment is identical.

Deployment Server
I have pushed apps through deployment-server which has a functionality of forwarder management feature where you can track all the apps, clients and serverclasses. Apps got deployed on the deployment clients too after reloading the deployment server.

Search Head Deploy
I'm having issues with search head deploy which is responsible to deploy apps to the cluster search heads. The deployment of the apps to the cluster search heads is not possible through deployment server, it has to be done using search head deploy server.

In spite of identical Pass4Symmkey in the general stanza and also in shclustering stanza in the server.conf file doesn't fix the issue.

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Champion

So did you make it through the first 5 steps of the process ok and are now on 6d (or the equivalent of a previous version)?

http://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/SHCdeploymentoverview

did you specify the same key when configuring the deployer and the cluster members? Or after? If after, did you restart splunk after changing it?

Any more details in the internal logs on the deployer or captain?

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Communicator

Yes, earlier configuration had different pass4Symmkey between search head deploy and search head cluster and I was aware about this issue when I applied the shcluster bundle from the search head deploy. Later, I have changed the pass4Symmkey on the search cluster head and rebooted the cluster head also splunk service too.

I have ensured the pass4Symmkey used for the searh head deploy and search head cluster are same in the general stanza as well as in the shclustering stanza too.

In the splunkd.log on the search head deploy says about the same error which I have posted in the beginning.

On the cluster search head in splunkd.log

SHPMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/members/{guid} captain={searchheadclustercaptainhostname}:{port} rc=0 actualresponsecode=502 expectedresponsecode=200 status_line=Error connecting: Connection refused error="Connection refused"

06-29-2017 19:54:01.879 +0000 ERROR SHPSlave - event=SHPSlave::handleHeartbeatDone heartbeat failure (reason: failed method=POST path=/services/shcluster/captain/members/{guid} captain={searchheadclustercaptainhostname}:{port} rc=0 actualresponsecode=502 expectedresponsecode=200 status_line=Error connecting: Connection refused error="Connection refused")

ERROR LMTracker - failed to send rows, reason='WARN: path=/masterlm/usage: invalid signature on request from ip={searchheadclustercaptainip_address}

These were the errors reported.

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Ultra Champion
0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Communicator

Drrillic, the error is different from mine. I have also tried using http instead of https still no progress. The error is about the fetching the apps baseline on the search head cluster captain.

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Communicator

You've verified that you are pushing to the nominated/dynamic "captain"?

#Linux
$SPLUNK_HOME/bin/splunk show shcluster-status

#Windows
%SPLUNK_HOME%/bin/splunk show shcluster-status

The nominated "captain" will be at the top.

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Communicator

In dev environment, we have only one cluster search head so obviously captain would be the existing one. I executed the command and status is Up for the captain and the hostname and mgmt_uri looks good.

0 Karma
Highlighted

Re: Why is the deployment server unable to push apps to search head clusters?

Champion

wait, one clustered search head? If you are saying that you are trying to create a search head cluster with one search head, pretty sure that's not allowed. I believe you need 3 for a cluster.

0 Karma