Why is the deployment server unable to push apps to search head clusters?

[vgollapudi]

Hello !!

I have created multiple apps under $SPLUNK_HOME/etc/shcluster/apps/ on the deployment server and checked the permissions on the directories and files under the apps. I have placed the files under local subdirectory in the SPLUNK_HOME/etc/shcluster/apps/{apps}/local/. Also checked the pass4Symmkey in the server.conf under /opt/splunk/etc/system/local/ on the search head deploy and on the search head cluster.

When executed the command that is /splunk apply shcluster-bundle -target https://{search_head_cluster_captain_ip_address}:port -auth {username}:{password}

Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://{search_head_cluster_captain_ip_address}:{port}: Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}.

I can see the apps staged under $SPLUNK_HOME/var/run/splunk/deploy/ but I'm unable to resolve it.

I have restarted the Splunk service and also the instance too but no luck. Every time I try to execute the command it always gives the above error.

When checked in the documentation they referred to this error because of the mismatch of Pass4Symmkey between the search head deploy and search head cluster.

Let me know where I can start debugging this issue. I have checked in the logs too on the search head deploy but it doesn't help since it only says about the above error.

Thanks

[joesrepsolc]

Having this same issue now on a brand new Splunk setup. Search head cluster is (3), and (1) deployer . Got everything dialed in but this command keeps generating the same message. I've tried against the captain, and not a captain. Same result.

Running command on the Deployer:
splunk apply shcluster-bundle -target https://SHCaptainName:8089 -auth admin:secretkey

Response:
Error while deploying apps to first member: Error while fetching apps baseline on target=https://SHCaptainName:8089: Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Tried creating a new folder in /opt/splunk/etc/shcluster/apps/testing/local/outputs.conf
Tried installing an app in /opt/splunk/etc/shcluster/apps/datagovernance

... same results/error/

splunk show shcluster-status shows all the cluster members are good, and "up". Can't push an app through the Deployer.

Stuck. Help?

Joe

[vgollapudi · Accepted solution]

I have found the solution for the issue.
The secret key is not same across the search head deploy and search head cluster member. So I have created new key which will be in plaintext format and I have copied under /opt/splunk/etc/system/local/server.conf in shclustering stanza on both search head deploy and cluster search head.

[shclustering]
pass4SymmKey = plaintext

Once I have made the above change I have restarted splunkd service on both of them. The plain text got converted to encrypted text to know the change is effective.

I tried to push the bundle through the search head deploy once the splunk service is up then I was able to push the apps successfully and the apps got copied under /opt/splunk/etc/apps on the cluster search head.

I hope this helps.

Links:

https://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/Setsecretkey

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/SHCconfigurationoverview#Configuration_...

[aaraneta_splunk]

@vgollapudi - Glad you were able to find the solution to your question. Please don't forget to click "Accept" to resolve your question so that others can easily find it if they run into the same issue. Thanks!

[vnguyen46]

@vgollapudi - excellent trick. I'd like to follow up with a question if you can help too. After first bundle applied, I need to install a new app or update existing apps. Does this task need to be complete on the deployer and how I can apply only the new/updated apps to the three SH cluster members I have? And is there a way I can push the changes to all three SH members with one command?
I am using this command from the deployer:
cd bin
./splunk apply shcluster-bundle -target https://SH1:8089 -push-default-apps true

Thank you,

[vgollapudi]

Thanks aaraneta

[adamsaul]

You've verified that you are pushing to the nominated/dynamic "captain"?

#Linux
$SPLUNK_HOME/bin/splunk show shcluster-status

#Windows
%SPLUNK_HOME%/bin/splunk show shcluster-status

The nominated "captain" will be at the top.

[vgollapudi]

In dev environment, we have only one cluster search head so obviously captain would be the existing one. I executed the command and status is Up for the captain and the hostname and mgmt_uri looks good.

[maciep]

wait, one clustered search head? If you are saying that you are trying to create a search head cluster with one search head, pretty sure that's not allowed. I believe you need 3 for a cluster.

[vgollapudi]

Maciep, I'm not creating search head cluster as of now, justing apps through shdeploy to search head cluster instance. At present, only one cluster search head is available in Development environment. I didn't research into the topic of downsides of having only one cluster search head instead of 3. If that's the case won't the shdeploy raise an exception about the instances available in the search head cluster since in this scenario only one search cluster head is present.

[vgollapudi]

You can have only search head according this documentation link.

http://docs.splunk.com/Documentation/Splunk/6.6.1/Deploy/Searchheadwithindexers

[maciep]

Ok, now I'm totally confused 🙂 if you're not creating a search head cluster, why do you keep referring to "cluster search heads" and "search head cluster instance"?

If you're just building a standalone search head that will query a set of indexers (like in the doc you linked to above), then push apps to it from the deployment server, i.e. etc/deployment-apps on your deployment server. A standalone search head can be managed like a forwarder. Or just manually install them on the search head.

If you're not building a search head cluster, don't use the deployer (etc/shcluster/apps). Unless something has changed from 6.5 to 6.6, that role is only for pushing config to search head cluster members....which you won't have.

[vgollapudi]

Sorry for the confusion. I'm not trying to build the environment, I'm just configuring the environment with the existing instances launched previously. I'm trying to understand the importances of pushing the apps through the deploy(etc/shcluster/apps) to the search head clusters which will query set of indexers. I will probably skip the pushing of the apps to the cluster search head.

[ddrillic]

Identical issue a year ago at Getting "Error while fetching apps baseline on target=http://:8089: Network-layer error: Connection ...

[vgollapudi]

Drrillic, the error is different from mine. I have also tried using http instead of https still no progress. The error is about the fetching the apps baseline on the search head cluster captain.

[maciep]

Is this a new cluster and the first time you're trying to deploy apps to it? Or was this a working cluster that started throwing this error?

And to be that guy for a moment: be careful about using "deployment server" and "deployer" names interchangeably - they are two different things and may lead to confusion if you say one and mean the other.

[vgollapudi]

Yes, this is a new cluster and deploying apps for the first time. I'm trying to replicate design of the production splunk environment such that development splunk environment is identical.

Deployment Server
I have pushed apps through deployment-server which has a functionality of forwarder management feature where you can track all the apps, clients and serverclasses. Apps got deployed on the deployment clients too after reloading the deployment server.

Search Head Deploy
I'm having issues with search head deploy which is responsible to deploy apps to the cluster search heads. The deployment of the apps to the cluster search heads is not possible through deployment server, it has to be done using search head deploy server.

In spite of identical Pass4Symmkey in the general stanza and also in shclustering stanza in the server.conf file doesn't fix the issue.

[maciep]

So did you make it through the first 5 steps of the process ok and are now on 6d (or the equivalent of a previous version)?

http://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/SHCdeploymentoverview

did you specify the same key when configuring the deployer and the cluster members? Or after? If after, did you restart splunk after changing it?

Any more details in the internal logs on the deployer or captain?

[vgollapudi]

Yes, earlier configuration had different pass4Symmkey between search head deploy and search head cluster and I was aware about this issue when I applied the shcluster bundle from the search head deploy. Later, I have changed the pass4Symmkey on the search cluster head and rebooted the cluster head also splunk service too.

I have ensured the pass4Symmkey used for the searh head deploy and search head cluster are same in the general stanza as well as in the shclustering stanza too.

In the splunkd.log on the search head deploy says about the same error which I have posted in the beginning.

On the cluster search head in splunkd.log

SHPMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/members/{guid} captain={search_head_cluster_captain_hostname}:{port} rc=0 actual_response_code=502 expected_response_code=200 status_line=Error connecting: Connection refused error="Connection refused"

06-29-2017 19:54:01.879 +0000 ERROR SHPSlave - event=SHPSlave::handleHeartbeatDone heartbeat failure (reason: failed method=POST path=/services/shcluster/captain/members/{guid} captain={search_head_cluster_captain_hostname}:{port} rc=0 actual_response_code=502 expected_response_code=200 status_line=Error connecting: Connection refused error="Connection refused")

ERROR LMTracker - failed to send rows, reason='WARN: path=/masterlm/usage: invalid signature on request from ip={search_head_cluster_captain_ip_address}

These were the errors reported.