Installation
Highlighted

After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Path Finder

After upgrading Splunk Enterprise from 6.3.3 to 6.4.0, I see this message:

[root@splunk bin]# $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.


Perform migration and upgrade without previewing configuration changes? [y/n] y

-- Migration information is being logged to '/opt/splunk/var/log/splunk/migration.log.2016-04-07.10-40-46' --

Migrating to:
VERSION=6.4.0
BUILD=f2c836328108
PRODUCT=splunk
PLATFORM=Linux-x86_64

Can't create directory "": No such file or directory


An error occurred: Could not create audit keys (returned 4).
[root@splunk bin]#

As a result Splunk does not start.
Please help me to resolve this issue!
Maybe you have ideas which directions to look.

Thank You!

Labels (1)
0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Communicator

I might be stating the obvious here but, does the user that runs the start command have write access on the Splunk directory and on /var/log (for the migration log file)?

0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Path Finder

My splunk always works from user root. I install and run it in Linux console also from root.
As an experiment I attempted to change access to files and folders in 777.
Command: chmod -R 777 /opt/splunk
The result is the same. What do you think?

Thank you!

0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Path Finder

Migration log file is created after each trial run, but it contains very short information.

[root@splunk bin]# cat /opt/splunk/var/log/splunk/migration.log.2016-04-08.11-52-31

Migrating to:
VERSION=6.4.0
BUILD=f2c836328108
PRODUCT=splunk
PLATFORM=Linux-x86_64

[root@splunk bin]#

0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Path Finder

I think what problem is with audit key, but old keys is available in destination folder. I don't understand why Splunk doesn't can to rebuild their.

[root@splunk audit]# ls -la /opt/splunk/etc/auth/audit/
total 16
drwxrwxrwx. 2 splunk splunk 4096 Apr 6 19:31 .
drwxrwxrwx. 6 splunk splunk 4096 Apr 6 19:43 ..
-rwxrwxrwx 1 splunk splunk 891 Mar 18 2014 private.pem
-rwxrwxrwx 1 splunk splunk 272 Mar 18 2014 public.pem
[root@splunk audit]#

0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Communicator

Were you able to resolve this yet?

0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Path Finder

No. This problem is actual.

0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Engager

Same error happened.
I was able to overcome this by creating file /opt/splunk/etc/system/local/audit.conf with the following content:

[auditTrail]
privateKey = /opt/splunk/etc/auth/audit/private.pem
publicKey = /opt/splunk/etc/auth/audit/public.pem

View solution in original post

Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

Path Finder

The issue was resolved in accordance with your recommendations.
Thank you!

0 Karma
Highlighted

Re: After upgrading Splunk from 6.3.3 to 6.4.0, why does Splunk not start with error "Can't create directory "": No such file or directory"?

I've had the same. In my case it was caused by the following configuration, which is part of the Enterprise Security (version 3.1.1) App:

$SPLUNK_HOME/etc/apps/SA-AuditAndDataProtection/default/audit.conf:

[filterSpec:event_whitelist:stashWhitelist]
sourcetype=stash

[filterSpec:event_blacklist:nothingElse]
all=True

[eventHashing]
filters=stashWhitelist,nothingElse

SOLNESS-2268: Disabling auditTrail signing by default

To enable, copy the following stanza to SA-AuditAndDataProtection/local/audit.conf

and swap the empty private/public key values for the populated ones

[auditTrail]
privateKey =

privateKey = $SPLUNK_HOME/etc/auth/audit/private.pem

publicKey =

publicKey = $SPLUNK_HOME/etc/auth/audit/public.pem

I also applied the proposed fix for the upgrade successfully, then reverted back.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.