Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

[ben_leung]

OS Linux
Splunk Version 6.3.1

After executing the following command on my deployer instance, I am seeing this error.

-bash-3.2$ ./splunk/bin/splunk apply shcluster-bundle -target https://myhost@domain.com:8089

 Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while creating deployable apps: Error copying src="/opt/splunk/etc/shcluster" to tmp_staging_area="/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp": No such file or directory

The temp_staging_area does exist and permissions are okay.

-bash-3.2$ ls -ltr
total 376
drwx------    2 splunkd splunkd   4096 Oct 30 04:45 upload
drwx--x---    4 splunkd splunkd   4096 Oct 30 04:45 appserver
drwx------    4 splunkd splunkd   4096 Oct 30 04:46 scheduler
drwx------    2 splunkd splunkd   4096 Nov  5 21:48 typeahead
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:39 deploy.fe8c0cae1fa6cd18.tmp
drwx------    2 splunkd splunkd   4096 Dec  1 00:47 merged
-rw-------    1 splunkd splunkd      6 Dec  1 00:47 conf-mutator.pid
-rw-------    1 splunkd splunkd  10937 Dec  1 00:47 composite.xml
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:48 deploy.c2659c1df562d295.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:49 deploy.39dbf1c7b7c94f85.tmp
drwx------    4 splunkd splunkd   4096 Dec  2 20:50 deploy
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:53 deploy.41b88613e6ab63d0.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:56 deploy.8e9ed66fdbd0ac3d.tmp
drwx--x--x 1309 splunkd splunkd 311296 Dec  2 20:58 dispatch
drwx------    2 splunkd splunkd   4096 Dec  2 20:58 srtemp
-rw-r-----    1 splunkd splunkd     30 Dec  2 21:00 splunkd.pid

I also have verified that the apps and users directory existing in the deploy.8e9ed66fdbd0ac3d.tmp folder.

-bash-3.2$ cd deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ pwd
/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ ls -ltr
total 88
drwx--x--- 1963 splunkd splunkd 81920 Dec  2 20:56 users
drwxr-x---   32 splunkd splunkd  4096 Dec  2 20:56 apps

[mlf]

Stray emacs temp files can apparently cause this sort of error as well.  (as of 8.2.x). E.g.

.../default/data/ui/views/.#example.xml

[georgen_splunk]

just had the same problem with our Deployer. Evidently, an app had multiple files exceeding 256 characters. Errors reported in splunkd.log regarding tar: Path/name too long

To view your name + path limits:

$getconf NAME_MAX /opt/splunk
255

$ getconf PATH_MAX /opt/splunk
4096

[ben_leung]

I wonder if this is the same behavior for deployment server? Would you experience issues if you symlinked the deployment_apps?

[jibanes]

Symlinks are no issue here, but you happen to have one "pointing" to a non-existing directory (dereferenced) from shcluster/apps to apps and/or the symlink itself doesn't have the right permissions.

[ben_leung · Accepted solution]

I had apps symlinked in shcluster/apps...

Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.

[ckurtz]

Try doing a "chown -R splunkd:splunkd" on the entire Splunk install tree on the deployer. I had a similar error (and the SPL is probably my complaint) about vague error messages that was a caused by a single bad file in a completely unassociated directory.

[dkoops]

I had the same problem and this fixed it for me (no symlinks involved);

Doing "chown -R" on the ../etc/shcluster/apps folder did the trick.

Thx!

[wallacd2018]

it worked for me the same way chown -R the shcluster directory. 

[mkolkebeck]

Knowing this is an old issue, I had a similar problem on Splunk 7.0.x with the users directory going missing within the shcluster path.

[ben_leung]

It was not successful. To add a little more context about the shcluster/apps directory, I have symlinked custom apps that relate to a local git repo that I have keeping the apps up to date. Splunk dev confirmed that symlinks would cause this issue.

[ckurtz]

Ok, at least we have confirmation of an issue and a bugID.

[ben_leung]

Support Case 293707

[ben_leung]

I was told SPL-106728 was created to have more logging for the deployer instance.