Splunk apply shcluster-bundle fail

[shandman]

Having a heck of a time implementing an application. (In this case the app=dnslookup).

Here is my command and error message.

root@cmgr1.splunk (Linux) $ ./splunk apply shcluster-bundle -target https://fe1.splunk.hq1.xxx.com:8089 -auth admin:xxxxxxxx
Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while deploying apps to first member: ConfDeploymentException: Error while updating app=dnslookup on target=https://10.99.106.31:8089: Non-200/201 status_code=404; {"messages":[{"type":"ERROR","text":"Application does not exist: "}]}

I really, really do not want to work with customer support. Please help!

1. I have verified iptables is not running
2. I have verified that there is no firewall blocking traffic.
3. I have verified that permissions are all correct

[yahuja_splunk]

i fixed it by changing the max_content_length in server.conf.

As stated, the file exceeds max_content_length in server.conf of 800 MB. This can be increased by adding the following to $SPLUNK_HOME/etc/system/local/server.conf.

[httpServer]
max_content_length = 1600000000

my bundle had a huge lookups and that was causing this error.

[rbal_splunk · Accepted solution]

I just tested by deploying this on my Search Head Cluster and below is my experience .

1) Before deploying the app - I installed dnslookup on the deployer server.
2) After installing this App on the deployer -when I restarted it it throws the following message

$SPLUNk_HOME/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
..                                                         [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.

Splunk> See your world.  Maybe wish you hadn't.

Checking prerequisites...
        Checking http port [24400]: open
        Checking mgmt port [24401]: open
        Checking appserver port [127.0.0.1:24405]: open
        Checking kvstore port [24406]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _internal _introspection _thefishbucket history main msad perfmon summary winevents
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
                Invalid key in stanza [install] in /opt/splunk/etc/apps/dnslookup/default/app.conf, line 10: author  (value:  Travis Freeland).
                Invalid key in stanza [install] in //opt/splunk/etc/apps/dnslookup/default/app.conf, line 11: description  (value:  dnslookup <forward|reverse> <input field> <outputfield>, servicelookup <input field> <output field> <optional services file path>).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunk-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]

3) To resolve the above issue before deploying app - fixed the $SPLUNK_HOME_DEPLOYER/etc/apps/dnslookup/default/apps.conf and comment out the two lines shows below in bold-text. (i.e attribute author and description under stanza [install])

-----------$SPLUNK_HOME_DEPLOYER/etc/apps/dnslookup/default/apps.conf---------

[package]
id = dnslookup

[install]
is_configured = 0
**#author = Travis Freeland

description = dnslookup , servicelookup **

build = 1
install_source_checksum = 5224a17b098abd2d668cb74836b9c1b394c4f373

[ui]
is_visible = 0
label = dnslookup

[launcher]
author = Travis Freeland
description = dnslookup , servicelookup

version = 0.1

NOTE: Since this is not Splunk provided app , but has been devloped by Travis Freeland I have provided feedback for above issue to be resolved..

4) Once the above issue was fixed copied the app from $SPLUNk_HOME_DEPLOYER/etc/apps/dnslookup to $SPLUNk_HOME_DEPLOYER/etc/shcluster/apps/dnslookup and deployed the bundle using command

$SPLUNK_HOME_DEPLOYER/bin/splunk apply shcluster-bundle -target https://:8089

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Your session is invalid. Please login.
Splunk username: admin
Password:
Bundle has been pushed successfully to all the cluster members

5) The App got deployed without any issue.

6) In your case have you been able to deploy other apps besides this app.

[jcrabb_splunk]

Ran into the same error where there was a folder with the name "-" (dash) in $SPLUNK_HOME/etc/shcluster/apps/. Removing that directory resolved the issue.

[sjohnson_splunk]

You should crank up the logging level on the deployer log channel: ConfDeployment to debug via the the Server Settings | Server Logging UI. Then run the apply command again and review the details in the splunkd log.

You may also need to look at the splunkd log on the target server as well and look for error messages.

[butzowj]

I am having the same issue.

First we try to run the deployer command to push the app to the cluster members, but we get the error above. I then created the app locally on each search head cluster member, but we still see the same issue:

[splunk@ bin]$ ./splunk apply shcluster-bundle -target https://ClusterMember3.labcorp.com:808 9 -auth admin:labcorp1
Warning: Depending on the configuration changes being pushed, this command might initiate a rollin g restart of the cluster members. Please refer to the documentation for the details. Do you wish t o continue? [y/n]: y
Error while deploying apps to target=https://ClusterMember3.labcorp.com:8089 with members=3: ConfDeploy mentException: Error while updating app=admin_app on target=https://10.111.1.192:8089: Non-200/201 status_code=404; {"messages":[{"type":"ERROR","text":"Application does not exist: "}]}
Error while deploying apps to target=https://ClusterMember3.labcorp.com:8089 with members=3: ConfDeploy mentException: Error while updating app=admin_app on target=https://10.111.1.193:8089: Non-200/201 status_code=404; {"messages":[{"type":"ERROR","text":"Application does not exist: "}]}

Thanks,
JB

[dbhagi_splunk]

Client is trying to update an app (dnslookup) that is supposedly exist on target server (indicated as “target=https://fe1.splunk.hq1.xxx.com:8089” in the error message).
Status 404 indicates that the client was able to establish the connection to the server but the server doesn’t have the requested resource (which is dnslookup app) and that’s what the error message is indicating.

Is the app existing on the target at an expected location?