topic Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer? in Deployment Architecture

Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung — Tue, 29 Sep 2020 08:03:48 GMT

OS Linux
Splunk Version 6.3.1

After executing the following command on my deployer instance, I am seeing this error.

-bash-3.2$ ./splunk/bin/splunk apply shcluster-bundle -target https://myhost@domain.com:8089

 Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while creating deployable apps: Error copying src="/opt/splunk/etc/shcluster" to tmp_staging_area="/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp": No such file or directory

The temp_staging_area does exist and permissions are okay.

-bash-3.2$ ls -ltr
total 376
drwx------    2 splunkd splunkd   4096 Oct 30 04:45 upload
drwx--x---    4 splunkd splunkd   4096 Oct 30 04:45 appserver
drwx------    4 splunkd splunkd   4096 Oct 30 04:46 scheduler
drwx------    2 splunkd splunkd   4096 Nov  5 21:48 typeahead
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:39 deploy.fe8c0cae1fa6cd18.tmp
drwx------    2 splunkd splunkd   4096 Dec  1 00:47 merged
-rw-------    1 splunkd splunkd      6 Dec  1 00:47 conf-mutator.pid
-rw-------    1 splunkd splunkd  10937 Dec  1 00:47 composite.xml
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:48 deploy.c2659c1df562d295.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:49 deploy.39dbf1c7b7c94f85.tmp
drwx------    4 splunkd splunkd   4096 Dec  2 20:50 deploy
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:53 deploy.41b88613e6ab63d0.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:56 deploy.8e9ed66fdbd0ac3d.tmp
drwx--x--x 1309 splunkd splunkd 311296 Dec  2 20:58 dispatch
drwx------    2 splunkd splunkd   4096 Dec  2 20:58 srtemp
-rw-r-----    1 splunkd splunkd     30 Dec  2 21:00 splunkd.pid

I also have verified that the apps and users directory existing in the deploy.8e9ed66fdbd0ac3d.tmp folder.

-bash-3.2$ cd deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ pwd
/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ ls -ltr
total 88
drwx--x--- 1963 splunkd splunkd 81920 Dec  2 20:56 users
drwxr-x---   32 splunkd splunkd  4096 Dec  2 20:56 apps

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung — Thu, 03 Dec 2015 16:48:14 GMT

Support Case 293707

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung — Thu, 03 Dec 2015 17:51:40 GMT

I was told SPL-106728 was created to have more logging for the deployer instance.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ckurtz — Thu, 03 Dec 2015 18:46:58 GMT

Try doing a "chown -R splunkd:splunkd" on the entire Splunk install tree on the deployer. I had a similar error (and the SPL is probably my complaint) about vague error messages that was a caused by a single bad file in a completely unassociated directory.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung — Thu, 03 Dec 2015 19:05:24 GMT

It was not successful. To add a little more context about the shcluster/apps directory, I have symlinked custom apps that relate to a local git repo that I have keeping the apps up to date. Splunk dev confirmed that symlinks would cause this issue.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung — Thu, 03 Dec 2015 21:30:03 GMT

I had apps symlinked in shcluster/apps...

Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ckurtz — Thu, 03 Dec 2015 21:32:28 GMT

Ok, at least we have confirmation of an issue and a bugID.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

jibanes — Tue, 31 May 2016 13:30:27 GMT

Symlinks are no issue here, but you happen to have one "pointing" to a non-existing directory (dereferenced) from shcluster/apps to apps and/or the symlink itself doesn't have the right permissions.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung — Fri, 10 Jun 2016 17:18:23 GMT

I wonder if this is the same behavior for deployment server? Would you experience issues if you symlinked the deployment_apps?

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

dkoops — Fri, 03 Feb 2017 10:42:17 GMT

I had the same problem and this fixed it for me (no symlinks involved);

Doing "chown -R" on the ../etc/shcluster/apps folder did the trick.

Thx!

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

georgen_splunk — Tue, 06 Feb 2018 21:59:31 GMT

just had the same problem with our Deployer. Evidently, an app had multiple files exceeding 256 characters. Errors reported in splunkd.log regarding tar: Path/name too long

To view your name + path limits:

$getconf NAME_MAX /opt/splunk
255

$ getconf PATH_MAX /opt/splunk
4096

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

mkolkebeck — Thu, 18 Oct 2018 21:21:42 GMT

Knowing this is an old issue, I had a similar problem on Splunk 7.0.x with the users directory going missing within the shcluster path.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "

wallacd2018 — Mon, 23 Aug 2021 18:32:38 GMT

it worked for me the same way chown -R the shcluster directory.

Re: Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "

mlf — Wed, 04 Jan 2023 21:26:47 GMT

Stray emacs temp files can apparently cause this sort of error as well. (as of 8.2.x). E.g.

.../default/data/ui/views/.#example.xml