Deployment Architecture

Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung
Builder

OS Linux
Splunk Version 6.3.1

After executing the following command on my deployer instance, I am seeing this error.

-bash-3.2$ ./splunk/bin/splunk apply shcluster-bundle -target https://myhost@domain.com:8089

 Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while creating deployable apps: Error copying src="/opt/splunk/etc/shcluster" to tmp_staging_area="/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp": No such file or directory

The temp_staging_area does exist and permissions are okay.

-bash-3.2$ ls -ltr
total 376
drwx------    2 splunkd splunkd   4096 Oct 30 04:45 upload
drwx--x---    4 splunkd splunkd   4096 Oct 30 04:45 appserver
drwx------    4 splunkd splunkd   4096 Oct 30 04:46 scheduler
drwx------    2 splunkd splunkd   4096 Nov  5 21:48 typeahead
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:39 deploy.fe8c0cae1fa6cd18.tmp
drwx------    2 splunkd splunkd   4096 Dec  1 00:47 merged
-rw-------    1 splunkd splunkd      6 Dec  1 00:47 conf-mutator.pid
-rw-------    1 splunkd splunkd  10937 Dec  1 00:47 composite.xml
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:48 deploy.c2659c1df562d295.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:49 deploy.39dbf1c7b7c94f85.tmp
drwx------    4 splunkd splunkd   4096 Dec  2 20:50 deploy
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:53 deploy.41b88613e6ab63d0.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:56 deploy.8e9ed66fdbd0ac3d.tmp
drwx--x--x 1309 splunkd splunkd 311296 Dec  2 20:58 dispatch
drwx------    2 splunkd splunkd   4096 Dec  2 20:58 srtemp
-rw-r-----    1 splunkd splunkd     30 Dec  2 21:00 splunkd.pid

I also have verified that the apps and users directory existing in the deploy.8e9ed66fdbd0ac3d.tmp folder.

-bash-3.2$ cd deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ pwd
/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ ls -ltr
total 88
drwx--x--- 1963 splunkd splunkd 81920 Dec  2 20:56 users
drwxr-x---   32 splunkd splunkd  4096 Dec  2 20:56 apps
1 Solution

ben_leung
Builder

I had apps symlinked in shcluster/apps...

Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.

View solution in original post

mlf
Path Finder

Stray emacs temp files can apparently cause this sort of error as well.  (as of 8.2.x). E.g.

.../default/data/ui/views/.#example.xml

 

0 Karma

georgen_splunk
Splunk Employee
Splunk Employee

just had the same problem with our Deployer. Evidently, an app had multiple files exceeding 256 characters. Errors reported in splunkd.log regarding tar: Path/name too long

To view your name + path limits:

$getconf NAME_MAX /opt/splunk
255

$ getconf PATH_MAX /opt/splunk
4096

ben_leung
Builder

I wonder if this is the same behavior for deployment server? Would you experience issues if you symlinked the deployment_apps?

0 Karma

jibanes
Path Finder

Symlinks are no issue here, but you happen to have one "pointing" to a non-existing directory (dereferenced) from shcluster/apps to apps and/or the symlink itself doesn't have the right permissions.

0 Karma

ben_leung
Builder

I had apps symlinked in shcluster/apps...

Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.

ckurtz
Path Finder

Try doing a "chown -R splunkd:splunkd" on the entire Splunk install tree on the deployer. I had a similar error (and the SPL is probably my complaint) about vague error messages that was a caused by a single bad file in a completely unassociated directory.

dkoops
Path Finder

I had the same problem and this fixed it for me (no symlinks involved);

Doing "chown -R" on the ../etc/shcluster/apps folder did the trick.

Thx!

wallacd2018
Engager

it worked for me the same way chown -R the shcluster directory. 

0 Karma

mkolkebeck
Path Finder

Knowing this is an old issue, I had a similar problem on Splunk 7.0.x with the users directory going missing within the shcluster path.

0 Karma

ben_leung
Builder

It was not successful. To add a little more context about the shcluster/apps directory, I have symlinked custom apps that relate to a local git repo that I have keeping the apps up to date. Splunk dev confirmed that symlinks would cause this issue.

0 Karma

ckurtz
Path Finder

Ok, at least we have confirmation of an issue and a bugID.

0 Karma

ben_leung
Builder

Support Case 293707

0 Karma

ben_leung
Builder

I was told SPL-106728 was created to have more logging for the deployer instance.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...