Deployment Architecture

Why am I getting "Error while creating deployable apps...No such file or directory" trying to to run "apply shcluster-bundle" on our search head cluster deployer?

ben_leung
Builder

OS Linux
Splunk Version 6.3.1

After executing the following command on my deployer instance, I am seeing this error.

-bash-3.2$ ./splunk/bin/splunk apply shcluster-bundle -target https://myhost@domain.com:8089

 Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while creating deployable apps: Error copying src="/opt/splunk/etc/shcluster" to tmp_staging_area="/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp": No such file or directory

The temp_staging_area does exist and permissions are okay.

-bash-3.2$ ls -ltr
total 376
drwx------    2 splunkd splunkd   4096 Oct 30 04:45 upload
drwx--x---    4 splunkd splunkd   4096 Oct 30 04:45 appserver
drwx------    4 splunkd splunkd   4096 Oct 30 04:46 scheduler
drwx------    2 splunkd splunkd   4096 Nov  5 21:48 typeahead
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:39 deploy.fe8c0cae1fa6cd18.tmp
drwx------    2 splunkd splunkd   4096 Dec  1 00:47 merged
-rw-------    1 splunkd splunkd      6 Dec  1 00:47 conf-mutator.pid
-rw-------    1 splunkd splunkd  10937 Dec  1 00:47 composite.xml
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:48 deploy.c2659c1df562d295.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  1 00:49 deploy.39dbf1c7b7c94f85.tmp
drwx------    4 splunkd splunkd   4096 Dec  2 20:50 deploy
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:53 deploy.41b88613e6ab63d0.tmp
drwxr-x---    4 splunkd splunkd   4096 Dec  2 20:56 deploy.8e9ed66fdbd0ac3d.tmp
drwx--x--x 1309 splunkd splunkd 311296 Dec  2 20:58 dispatch
drwx------    2 splunkd splunkd   4096 Dec  2 20:58 srtemp
-rw-r-----    1 splunkd splunkd     30 Dec  2 21:00 splunkd.pid

I also have verified that the apps and users directory existing in the deploy.8e9ed66fdbd0ac3d.tmp folder.

-bash-3.2$ cd deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ pwd
/opt/splunk/var/run/splunk/deploy.8e9ed66fdbd0ac3d.tmp
-bash-3.2$ ls -ltr
total 88
drwx--x--- 1963 splunkd splunkd 81920 Dec  2 20:56 users
drwxr-x---   32 splunkd splunkd  4096 Dec  2 20:56 apps
1 Solution

ben_leung
Builder

I had apps symlinked in shcluster/apps...

Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.

View solution in original post

mlf
Path Finder

Stray emacs temp files can apparently cause this sort of error as well.  (as of 8.2.x). E.g.

.../default/data/ui/views/.#example.xml

 

0 Karma

georgen_splunk
Splunk Employee
Splunk Employee

just had the same problem with our Deployer. Evidently, an app had multiple files exceeding 256 characters. Errors reported in splunkd.log regarding tar: Path/name too long

To view your name + path limits:

$getconf NAME_MAX /opt/splunk
255

$ getconf PATH_MAX /opt/splunk
4096

ben_leung
Builder

I wonder if this is the same behavior for deployment server? Would you experience issues if you symlinked the deployment_apps?

0 Karma

jibanes
Path Finder

Symlinks are no issue here, but you happen to have one "pointing" to a non-existing directory (dereferenced) from shcluster/apps to apps and/or the symlink itself doesn't have the right permissions.

0 Karma

ben_leung
Builder

I had apps symlinked in shcluster/apps...

Splunk support with confirmation of dev says the symlinks would cause issues, so I have removed them and now works fine.

ckurtz
Path Finder

Try doing a "chown -R splunkd:splunkd" on the entire Splunk install tree on the deployer. I had a similar error (and the SPL is probably my complaint) about vague error messages that was a caused by a single bad file in a completely unassociated directory.

dkoops
Path Finder

I had the same problem and this fixed it for me (no symlinks involved);

Doing "chown -R" on the ../etc/shcluster/apps folder did the trick.

Thx!

wallacd2018
Engager

it worked for me the same way chown -R the shcluster directory. 

0 Karma

mkolkebeck
Path Finder

Knowing this is an old issue, I had a similar problem on Splunk 7.0.x with the users directory going missing within the shcluster path.

0 Karma

ben_leung
Builder

It was not successful. To add a little more context about the shcluster/apps directory, I have symlinked custom apps that relate to a local git repo that I have keeping the apps up to date. Splunk dev confirmed that symlinks would cause this issue.

0 Karma

ckurtz
Path Finder

Ok, at least we have confirmation of an issue and a bugID.

0 Karma

ben_leung
Builder

Support Case 293707

0 Karma

ben_leung
Builder

I was told SPL-106728 was created to have more logging for the deployer instance.

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...